A high-resolution hyperspectral video fusion imaging device and method

Abstract

The present invention discloses a high-resolution hyperspectral video fusion imaging device and method. The high-resolution hyperspectral video fusion imaging device of the present invention comprises a lens, a spectroscopic device and a plurality of imaging chips, wherein the plurality of imaging chips comprises a full-color CCD chip and at least one pixel-level mosaic-coated CCD chip. The lens captures the reflected light of the target and then divides it into multiple beams through the spectroscopic device, and receives and converts them into electrical signals through a plurality of imaging chips respectively. The imaging method of the present invention comprises converting the problem of fusion of a high-resolution full-color image Y with a low-resolution hyperspectral image X into the problem of estimating a spectral subspace D and its subspace coefficient matrix C to realize fusion imaging of a high-resolution hyperspectral image. The present invention aims to solve the problems of difficulty in realizing high-resolution hyperspectral images and high costs, reduce the cost of acquiring hyperspectral images, and improve acquisition efficiency.

Description

Technical Field

[0001] The present invention relates to a high-resolution high-spectral video fusion imaging device and method. Background Art

[0002] Hyperspectral imaging technology combines two-dimensional spatial data information with spectral data information. It has a wider spectral band, higher spectral resolution, and richer spectral data information. It can simultaneously record image information, spatial information, peak and trough information in the spectral graph, and reflection intensity within the corresponding range of the target to be measured within the selected band. Hyperspectral imaging technology has great advantages in image recognition and detection, and plays an important role in remote sensing detection, agricultural land cover surveys, environmental monitoring and other fields. At present, the main hyperspectral imagers on the market are spectral scanning, swing scanning, and push scanning. Due to the limitations of optical imaging hardware facilities, it is difficult to quickly obtain high-resolution hyperspectral data at low cost. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a high-resolution hyperspectral video fusion imaging device and method are provided. The present invention aims to solve the problems of difficulty in realizing high-resolution hyperspectral images and high costs, reduce the cost of acquiring hyperspectral images, and improve the acquisition efficiency.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A high-resolution hyperspectral video fusion imaging device comprises a lens, a spectroscopic device and a plurality of imaging chips, wherein the plurality of imaging chips comprise a full-color CCD chip and at least one pixel-level mosaic-coated CCD chip. After the lens captures the reflected light of the target, the reflected light is divided into multiple beams by the spectroscopic device and respectively received and converted into electrical signals by the plurality of imaging chips.

[0006] Optionally, the pixel-level mosaic-coated CCD chips all include a CCD chip and a pixel-level mosaic coating provided on the light-incident side of the CCD chip.

[0007] Optionally, the pixel-level mosaic coating is composed of a plurality of pixel-level mosaic units arranged repeatedly, and the pixel-level mosaic unit is composed of a plurality of pixel-level mosaics, and different pixel-level mosaics in the pixel-level mosaic unit have different filtering bands.

[0008] In addition, the present invention also provides a high-resolution hyperspectral video fusion imaging method based on the high-resolution hyperspectral video fusion imaging device, comprising the following steps:

[0009] S101, obtaining a cropped aligned image corresponding to the maximum overlapping field of view in the original image by acquiring three imaging chips;

[0010] S102, combining the cropped and aligned images of multiple pixel-level mosaic-coated CCD chips into a single-band image and stacking them to obtain a low-resolution hyperspectral image, and interpolating them to the size of the cropped and aligned images corresponding to the maximum overlapping field of view to obtain an interpolated low-resolution hyperspectral image X;

[0011] S103: The cropped aligned image of the full-color CCD chip is used as a high-resolution full-color image Y, and is fused with the low-resolution hyperspectral image X to obtain a high-resolution hyperspectral image Z.

[0012] Optionally, obtaining the cropped aligned images corresponding to the maximum overlapping field of view in the original image obtained by the three imaging chips in step S101 includes: dedistorting the original images obtained by the three imaging chips and obtaining the maximum overlapping field of view of the three imaging chips according to calibration parameters obtained by pre-calibrating the high-resolution hyperspectral video fusion imaging device, and then cropping the cropped aligned images corresponding to the maximum overlapping field of view from the original images.

[0013] Optionally, in step S102, when the cropped and aligned images of multiple pixel-level mosaic coated CCD chips are combined into a single-band image and stacked to obtain a low-resolution high-spectral image, the process includes: The cropped and aligned images of pixel-level mosaic coated CCD chips form a single-band image, and the total number of bands is c 1 , composed of any The image size of each band is , and any pixel coordinate The pixel value corresponding to the position The calculation function expression is:

[0014] ,

[0015] In the above formula, For a / c 2 Round up to make F 1 or F 2 , F 1 and F 2 These are the cropped and aligned images of the first pixel-level mosaic-coated CCD chip and the second pixel-level mosaic-coated CCD chip. is the floor of (a-1) / n, where a is the band number and c is the 2 is the total number of bands c 1 half.

[0016] Optionally, when the interpolated low-resolution hyperspectral image X is obtained by interpolating the size of the cropped aligned image corresponding to the maximum overlapping field of view in step S102, the interpolated low-resolution hyperspectral image X includes: Estimate pixel values ​​and complete pixel values ​​for each target point to be interpolated Estimated pixel values ​​include:

[0017] S201, calculate the target point to be interpolated and The Euclidean distance between adjacent pixels;

[0018] S202, substitute the calculated Euclidean distance into the one-dimensional Gaussian function of the following formula to obtain The weight of neighboring pixels :

[0019] ,

[0020] In the above formula, represents the variance, represents the scale factor, The interpolation target point and Among the neighboring pixels The Euclidean distance between adjacent pixels, and The sum of the weights of adjacent pixels is 1;

[0021] S203, calculate the target point to be interpolated according to the following formula Estimated pixel value of:

[0022]

[0023] In the above formula, The target point to be interpolated The estimated pixel value of is the target point to be interpolated in the low-resolution hyperspectral image of Among the neighboring pixels The pixel values ​​of adjacent pixels.

[0024] Optionally, in step S103, taking the cropped aligned image of the panchromatic CCD chip as the high-resolution panchromatic image Y and fusing it with the low-resolution hyperspectral image X to obtain the high-resolution hyperspectral image Z includes:

[0025] S301, establishing an imaging model of a low-resolution hyperspectral image X and a high-resolution panchromatic image Y:

[0026] ,

[0027] ,

[0028] In the above formula, is the two-dimensional matrix of the low-resolution hyperspectral image X expanded along the image band dimension, is the two-dimensional matrix of the high-resolution hyperspectral image Z expanded along the image band dimension, B is the spatial blur operation matrix, and S is the spatial downsampling operation matrix; is the two-dimensional matrix of the high-resolution panchromatic image Y expanded along the image band dimension, and R is the spectral downsampling operation matrix;

[0029] Establish a low-rank subspace representation of the high-resolution hyperspectral image Z:

[0030]

[0031] In the above formula, D represents the spectral subspace, is a two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension; through the imaging model of the low-resolution hyperspectral image X and the high-resolution panchromatic image Y and the low-rank subspace representation of the high-resolution hyperspectral image Z, the problem of fusing the high-resolution panchromatic image Y with the low-resolution hyperspectral image X is converted into the problem of estimating the spectral subspace D and its subspace coefficient matrix C;

[0032] S302, solving the problem of estimating the spectral subspace D and its subspace coefficient matrix C;

[0033] The problem of solving the estimation of spectral subspace D includes:

[0034] 1) Perform singular value decomposition on the low-resolution hyperspectral image X:

[0035] ,

[0036] In the above formula, and is the chief matrix, is a diagonal matrix;

[0037] 2) Keep the diagonal matrix Specified quantity The largest singular value, we get the following low-dimensional decomposition:

[0038] ,

[0039] In the above formula, is the low-dimensional part of the low-resolution hyperspectral image X, Chief Matrix The low-dimensional part of is a diagonal matrix The low-dimensional part of Chief Matrix The low-dimensional part of , and:

[0040] , , ;

[0041] 3) The Chief Matrix The low-dimensional part of is used as the estimated spectral subspace D;

[0042] The problem of solving the subspace coefficient matrix C includes:

[0043] 1) Establish an optimization problem model for solving the subspace coefficient matrix C:

[0044] ,

[0045] In the above formula, Represents the prior constraints on the subspace coefficient matrix C;

[0046] 2) Divide the high-resolution full-color image Y into several spatial sizes Then cluster these image blocks into Class image block:

[0047] ,

[0048] In the above formula, For the Class image block, For the The j-th image patch of class, For the The number of similar image blocks in the same image block, N is the number of clusters;

[0049] 3) Based on the fact that the high-resolution hyperspectral image Z and the subspace coefficient matrix C have the same Frobenius norm, the image block clustering structure learned from the high-resolution panchromatic image Y is applied to the subspace coefficient matrix C to obtain:

[0050] ,

[0051] In the above formula, For the Class coefficient block, For the The j-th coefficient block of the class, and Spatial location and The jth image patch of class Same, each Class coefficient block Composing tensors , tensor The three dimensions represent the number of similarity coefficient blocks in the class, the spectral subspace dimension, and the number of pixels in the image block;

[0052] 4) For tensors By imposing a tensor low-rank prior constraint, the optimization problem model for solving the subspace coefficient matrix C is transformed into solving the following optimization problem:

[0053] ,

[0054] In the above formula, is the regularization parameter, is the regularization term of the tensor rank, is the two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension; an additional variable required for solving the optimization problem by an alternating direction multiplication method is introduced , and the augmented Lagrangian function is obtained:

[0055]

[0056] In the above formula, is the augmented Lagrangian function, is the penalty factor, is the Lagrange multiplier, For tensors The representation of in the form of variable V;

[0057] 5) Convert the minimization optimization of the augmented Lagrangian function into iterative solutions to the following sub-problems:

[0058] Subproblem (a): Estimate the two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension :

[0059] ,

[0060] In the above formula, and They are tensors and The three-dimensional expansion matrix of

[0061] Subproblem (b): Estimating variables :

[0062] ,

[0063] Subproblem (c): Estimating variables :

[0064] ,

[0065] 6) Use the alternating direction multiplier method to iterate and update the variables for the sub-problems in turn. , as well as , when the number of iterations reaches the preset maximum number of iterations When , the final subspace coefficient matrix C is expanded according to the channel dimension to obtain a two-dimensional matrix ;

[0066] S303, a two-dimensional matrix is ​​obtained by expanding the estimated spectral subspace D and the final subspace coefficient matrix C according to the channel dimension ,according to Get the two-dimensional matrix of the high-resolution hyperspectral image Z after it is expanded along the image band dimension , as the high-resolution panchromatic image Y is fused with the low-resolution hyperspectral image X to obtain the high-resolution hyperspectral image Z.

[0067] In addition, the present invention also provides a high-resolution hyperspectral video fusion imaging system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the high-resolution hyperspectral video fusion imaging method.

[0068] In addition, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by a microprocessor to execute the high-resolution hyperspectral video fusion imaging method.

[0069] Compared with the prior art, the present invention mainly has the following advantages: the high-resolution hyperspectral video fusion imaging device of the present invention comprises a lens, a spectrometer and three imaging chips, the three imaging chips comprise a first pixel-level mosaic coated CCD chip, a second pixel-level mosaic coated CCD chip and a full-color CCD chip, the lens captures the reflected light of the target and then divides it into three beams through the spectrometer, and receives and converts them into electrical signals through the three imaging chips respectively, on the one hand, by simultaneously imaging through the three imaging chips, the efficiency of generating the required data for the high-resolution hyperspectral image can be improved; on the other hand, by using the pixel-level mosaic coating, the three imaging chips of the first pixel-level mosaic coated CCD chip, the second pixel-level mosaic coated CCD chip and the full-color CCD chip are used to obtain the required data for generating different high-resolution hyperspectral images, which can provide the required basic data for the high-quality generation of high-resolution hyperspectral images, solve the problems of difficulty in realizing high-resolution hyperspectral images and high costs, reduce the cost of acquiring hyperspectral images, and improve the acquisition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Schematic diagram of the principle of a high-resolution hyperspectral video fusion imaging device in an embodiment of the present invention.

[0071] Figure 2 Schematic diagram of the local structure of pixel-level mosaic coating in an embodiment of the present invention.

[0072] Figure 3 Schematic diagram of the principle of interpolation pixel completion in an embodiment of the present invention.

[0073] Figure 4 It is the imaging model of the high-resolution hyperspectral image Z in the embodiment of the present invention. DETAILED DESCRIPTION

[0074] like Figure 1 As shown, the high-resolution hyperspectral video fusion imaging device of this embodiment includes a lens 1, a spectrometer 2 and three imaging chips, wherein the three imaging chips include a first pixel-level mosaic-coated CCD chip 3, a second pixel-level mosaic-coated CCD chip 4 and a full-color CCD chip 5. The lens 1 captures the reflected light of the target and then divides it into three beams through the spectrometer 2, and the three imaging chips receive and convert them into electrical signals respectively. The lens 1 captures the reflected light of the target, and the spectrometer 2 divides the captured light into three beams; two of the beams are received and converted into electrical signals by the first pixel-level mosaic-coated CCD chip 3 and the second pixel-level mosaic-coated CCD chip 4, and one beam is received and converted into electrical signals by the full-color CCD chip 5, thereby obtaining a low-resolution hyperspectral image and a high-resolution full-color image, and a high-resolution hyperspectral image can be obtained by using a high-resolution hyperspectral video fusion imaging method. It should be noted that, in this embodiment, two pixel-level mosaic-coated CCD chips, namely a first pixel-level mosaic-coated CCD chip 3 and a second pixel-level mosaic-coated CCD chip 4, are used to collect spectral information of more bands. In addition, a single pixel-level mosaic-coated CCD chip or more pixel-level mosaic-coated CCD chips may be used as needed.

[0075] like Figure 1 As shown, in this embodiment, the spectroscopic device 2 includes three lenses arranged in sequence. The reflected light of the target is vertically incident on the first lens arranged in the vertical direction, and is transmitted through the first lens to the second lens arranged obliquely. Then it is divided into two paths on the second lens: the first path is reflected by the second lens to the back of the first lens, and then reflected through the back of the first lens to enter the second pixel-level mosaic coated CCD chip 4, and the second path is transmitted through the second lens to enter the third lens arranged obliquely. Then it is divided into two paths on the third lens: the first path is reflected by the third lens to the back of the second lens, and then reflected through the back of the second lens to enter the first pixel-level mosaic coated CCD chip 3, and the second path is transmitted through the third lens to enter the full-color CCD chip 5.

[0076] In this embodiment, the first pixel-level mosaic coating CCD chip 3 and the second pixel-level mosaic coating CCD chip 4 both include a CCD chip and a pixel-level mosaic coating disposed on the light-incident side of the CCD chip. Figure 2 As shown, in this embodiment, the pixel-level mosaic coating is composed of a plurality of pixel-level mosaic units arranged repeatedly, and the pixel-level mosaic unit is composed of a plurality of pixel-level mosaics, and the filtering bands of different pixel-level mosaics in the pixel-level mosaic unit are different. Specifically, in this embodiment, the pixel-level mosaic unit is composed of 3×3 pixel-level mosaics to obtain spectral information of 9 bands.

[0077] It should be noted that, according to the low-resolution hyperspectral images of different bands obtained by the first pixel-level mosaic-coated CCD chip 3 and the second pixel-level mosaic-coated CCD chip 4, and the high-resolution full-color image obtained by the full-color CCD chip 5, the required hyperspectral image fusion imaging method can be used as needed to generate a high-resolution hyperspectral image. For example, as an optional implementation, this embodiment also provides a high-resolution hyperspectral video fusion imaging method based on the high-resolution hyperspectral video fusion imaging device, including the following steps:

[0078] S101, obtaining a cropped aligned image corresponding to the maximum overlapping field of view in the original image by acquiring three imaging chips;

[0079] S102, the cropped and aligned images of the first pixel-level mosaic coated CCD chip 3 and the second pixel-level mosaic coated CCD chip 4 are combined into a single-band image and stacked to obtain a low-resolution hyperspectral image (if a single pixel-level mosaic coated CCD chip is used, the image output by the pixel-level mosaic coated CCD chip can be directly used as the low-resolution hyperspectral image), and interpolated to the size of the cropped and aligned images corresponding to the maximum overlapping field of view to obtain an interpolated low-resolution hyperspectral image X;

[0080] S103 , the cropped aligned image of the full-color CCD chip 5 is used as a high-resolution full-color image Y, and is fused with the low-resolution hyperspectral image X to obtain a high-resolution hyperspectral image Z.

[0081] In step S101 of this embodiment, obtaining the cropped aligned image corresponding to the maximum overlapping field of view in the original image obtained by the three imaging chips includes: dedistorting the original image obtained by the three imaging chips and obtaining the maximum overlapping field of view of the three imaging chips according to the calibration parameters obtained by pre-calibration of the high-resolution hyperspectral video fusion imaging device, and then cropping the cropped aligned images corresponding to the maximum overlapping field of view from the original image respectively. The calibration parameters obtained by pre-calibration can adopt the required calibration method as needed, for example, as an optional implementation, in this embodiment, the calibration parameters are specifically obtained by calibration using the Zhang Zhengyou calibration method.

[0082] The size of the cropped aligned image of the first pixel-level mosaic-coated CCD chip 3 is , pixel coordinates The pixel value corresponding to the position is The size of the cropped aligned image of the second pixel-level mosaic coated CCD chip 4 is , pixel coordinates The pixel value corresponding to the position is In this embodiment, the size of the pixel-level mosaic unit is The pixel size can be obtained as follows: Low-resolution hyperspectral images In step S102 of this embodiment, when the cropped and aligned images of multiple pixel-level mosaic coated CCD chips are combined into a single-band image and stacked to obtain a low-resolution high-spectral image, the step includes: The cropped and aligned images of pixel-level mosaic coated CCD chips form a single-band image, and the total number of bands is c 1 , composed of any The image size of each band is , and any pixel coordinate The pixel value corresponding to the position The calculation function expression is:

[0083] ,

[0084] In the above formula, For a / c 2 Round up to make F 1 or F 2 , F 1 and F 2 They are respectively the cropped and aligned images of the first pixel-level mosaic coated CCD chip 3 and the second pixel-level mosaic coated CCD chip 4, is the floor of (a-1) / n, where a is the band number and c is the 2 is the total number of bands c 1 half.

[0085] Specifically, the total number of bands c in this embodiment is 1 is 18, c 2 is 9, n is 3, then any pixel coordinate The pixel value corresponding to the position The calculation function expression is:

[0086] ,

[0087] In the above formula, is the upper integer of a / 9, so that F 1 or F 2 .

[0088] In step S102 of this embodiment, when the size of the cropped aligned image corresponding to the maximum overlapping field of view is interpolated to obtain the interpolated low-resolution hyperspectral image X, the interpolated low-resolution hyperspectral image X includes: for each target point to be interpolated in the interpolated low-resolution hyperspectral image X Estimate pixel values ​​and complete pixel values. Figure 3 As shown in the figure, the grids with numbers 1 to 9 are the pixels in the cropped aligned image, and the grids without numbers are the target points to be interpolated. All pixel values ​​need to be filled, and the interpolation multiple is:

[0089]

[0090] In the above formula, H and W are the height and width of the cropped aligned image, respectively, and h and w are the height and width of the stacked low-resolution hyperspectral image.

[0091] In this embodiment, for each target point to be interpolated Estimated pixel values ​​include:

[0092] S201, calculate the target point to be interpolated and The Euclidean distance between adjacent pixels;

[0093] S202, substitute the calculated Euclidean distance into the one-dimensional Gaussian function of the following formula to obtain The weight of neighboring pixels :

[0094] ,

[0095] In the above formula, represents the variance, represents the scale factor, The interpolation target point and Among the neighboring pixels The Euclidean distance between adjacent pixels, and The sum of the weights of adjacent pixels is 1;

[0096] S203, calculate the target point to be interpolated according to the following formula Estimated pixel value of:

[0097]

[0098] In the above formula, The target point to be interpolated The estimated pixel value of is the target point to be interpolated in the low-resolution hyperspectral image of Among the neighboring pixels The pixel values ​​of adjacent pixels.

[0099] In step S103, the cropped aligned image of the full-color CCD chip 5 needs to be used as a high-resolution full-color image. , compared with low-resolution hyperspectral images The fusion resolution is And the number of bands is High-resolution hyperspectral images In step S103 of this embodiment, the cropped aligned image of the full-color CCD chip 5 is used as the high-resolution full-color image Y, and fused with the low-resolution hyperspectral image X to obtain the high-resolution hyperspectral image Z, which includes:

[0100] S301, low-resolution hyperspectral image It can be regarded as Spatially downsampled, full-color image It can be seen as a high-resolution hyperspectral image The two-dimensional tensor obtained by spectral downsampling operation can establish the imaging model of low-resolution hyperspectral image X and high-resolution panchromatic image Y:

[0101] ,

[0102] ,

[0103] In the above formula, is the two-dimensional matrix of the low-resolution hyperspectral image X expanded along the image band dimension, is the two-dimensional matrix of the high-resolution hyperspectral image Z expanded along the image band dimension, B is the spatial blur operation matrix, and S is the spatial downsampling operation matrix; is a two-dimensional matrix after the high-resolution full-color image Y is expanded along the image band dimension, and R is a spectral downsampling operation matrix; in this embodiment, and , , , , the spatial blur operation matrix B is a known parameter determined by the point spread function of the camera itself, and the spectral downsampling operation matrix R is a known parameter determined by the spectral response function of the panchromatic camera.

[0104] Since the bands of hyperspectral images have a large correlation, a low-rank subspace representation of the high-resolution hyperspectral image Z can be established:

[0105]

[0106] In the above formula, D represents the spectral subspace, is a two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension; through the imaging model of the low-resolution hyperspectral image X and the high-resolution panchromatic image Y and the low-rank subspace representation of the high-resolution hyperspectral image Z, the problem of fusing the high-resolution panchromatic image Y with the low-resolution hyperspectral image X is converted into the problem of estimating the spectral subspace D and its subspace coefficient matrix C, as shown in Figure 4 As shown; In this embodiment, , ;

[0107] S302, solving the problem of estimating the spectral subspace D and its subspace coefficient matrix C;

[0108] The problem of solving the estimation of spectral subspace D includes:

[0109] 1) Perform singular value decomposition on the low-resolution hyperspectral image X:

[0110] ,

[0111] In the above formula, and is the chief matrix, is a diagonal matrix;

[0112] 2) Keep the diagonal matrix Specified quantity The largest singular value, we get the following low-dimensional decomposition:

[0113] ,

[0114] In the above formula, is the low-dimensional part of the low-resolution hyperspectral image X, Chief Matrix The low-dimensional part of is a diagonal matrix The low-dimensional part of Chief Matrix The low-dimensional part of , and:

[0115] , , ;

[0116] 3) The Chief Matrix The low-dimensional part of is used as the estimated spectral subspace D;

[0117] The problem of solving the subspace coefficient matrix C includes:

[0118] 1) Establish an optimization problem model for solving the subspace coefficient matrix C:

[0119] ,

[0120] In the above formula, Represents the prior constraints on the subspace coefficient matrix C; since the subspace coefficient It covers the main spatial structure information of the hyperspectral image, so we use a low-rank tensor representation prior model to estimate the subspace coefficients;

[0121] 2) Divide the high-resolution full-color image Y into several spatial sizes Then cluster these image blocks into Class image block:

[0122] ,

[0123] In the above formula, For the Class image block, For the The j-th image patch of class, For the The number of similar image blocks in the same image block, N is the number of clusters;

[0124] 3) Based on the fact that the high-resolution hyperspectral image Z and the subspace coefficient matrix C have the same Frobenius norm, the image block clustering structure learned from the high-resolution panchromatic image Y is applied to the subspace coefficient matrix C to obtain:

[0125] ,

[0126] In the above formula, For the Class coefficient block, For the The j-th coefficient block of the class, and Spatial location and The jth image patch of class Same, each Class coefficient block Composing tensors , tensor The three dimensions represent the number of similarity coefficient blocks in the class, the spectral subspace dimension, and the number of pixels in the image block;

[0127] 4) For tensors By imposing a tensor low-rank prior constraint, the optimization problem model for solving the subspace coefficient matrix C is transformed into solving the following optimization problem:

[0128] ,

[0129] In the above formula, is the regularization parameter, is the regularization term of the tensor rank, is the two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension; since the optimization problem with regularization terms is difficult to solve directly, we use the alternating direction multiplier method to decompose the problem into several more easily solved sub-problems for iterative solution, so it is necessary to introduce an additional variable required for the alternating direction multiplier method to solve the optimization problem , and the augmented Lagrangian function is obtained:

[0130]

[0131] In the above formula, is the augmented Lagrangian function, is the penalty factor, is the Lagrange multiplier, For tensors The representation of in the form of variable V;

[0132] 5) Convert the minimization optimization of the augmented Lagrangian function into iterative solutions to the following sub-problems:

[0133] Subproblem (a): Estimate the two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension :

[0134] ,

[0135] In the above formula, and They are tensors and The three-dimensional expansion matrix of

[0136] Subproblem (b): Estimating variables :

[0137] ,

[0138] Subproblem (c): Estimating variables :

[0139] ,

[0140] 6) Use the alternating direction multiplier method to iterate and update the variables for the sub-problems in turn. , as well as , when the number of iterations reaches the preset maximum number of iterations When , the final subspace coefficient matrix C is expanded according to the channel dimension to obtain a two-dimensional matrix ;

[0141] S303, a two-dimensional matrix is ​​obtained by expanding the estimated spectral subspace D and the final subspace coefficient matrix C according to the channel dimension ,according to Get the two-dimensional matrix of the high-resolution hyperspectral image Z after it is expanded along the image band dimension , as the high-resolution panchromatic image Y is fused with the low-resolution hyperspectral image X to obtain the high-resolution hyperspectral image Z.

[0142] The method of this embodiment converts the problem of fusing the high-resolution panchromatic image Y with the low-resolution hyperspectral image X into the problem of estimating the spectral subspace D and its subspace coefficient matrix C through the imaging model of the low-resolution hyperspectral image X and the high-resolution panchromatic image Y and the low-rank subspace representation of the high-resolution hyperspectral image Z. It can quickly and accurately generate high-quality high-resolution hyperspectral images based on the high-image data obtained by the three imaging chips, thereby providing a complete solution for high-resolution hyperspectral video fusion imaging from the software level.

[0143] In summary, the high-resolution hyperspectral video fusion imaging device of the present embodiment can generate the image data required for high-resolution hyperspectral video fusion imaging at low cost and high efficiency. The high-resolution hyperspectral video fusion imaging method of the present embodiment can fuse based on the required image data, quickly and accurately generate high-quality high-resolution hyperspectral images. From hardware to software, it provides an overall solution for high-resolution hyperspectral video fusion imaging, which can effectively reduce the data acquisition and generation costs of high-resolution hyperspectral images, ensure the generation quality of high-resolution hyperspectral images, solve the problems of difficult realization and high cost of high-resolution hyperspectral images, reduce the acquisition cost of hyperspectral images, and improve the acquisition efficiency.

[0144] In addition, this embodiment also provides a high-resolution hyperspectral video fusion imaging system, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the high-resolution hyperspectral video fusion imaging method. This embodiment also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program is used to be programmed or configured by the microprocessor to execute the high-resolution hyperspectral video fusion imaging method.

[0145] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0146] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A high-resolution hyperspectral video fusion imaging method based on a high-resolution hyperspectral video fusion imaging device, characterized in that: The high-resolution hyperspectral video fusion imaging device comprises a lens (1), a spectroscopic device (2) and a plurality of imaging chips, wherein the plurality of imaging chips comprise a full-color CCD chip (5) and at least one pixel-level mosaic-coated CCD chip. After the lens (1) captures the reflected light of the target, the reflected light is divided into a plurality of beams through the spectroscopic device (2), and the beams are received and converted into electrical signals respectively through the plurality of imaging chips. The high-resolution hyperspectral video fusion imaging method comprises the following steps: S101, obtaining a cropped aligned image corresponding to the maximum overlapping field of view in the original image by acquiring three imaging chips; S102, combining the cropped and aligned images of multiple pixel-level mosaic-coated CCD chips into a single-band image and stacking them to obtain a low-resolution hyperspectral image, and interpolating them to the size of the cropped and aligned images corresponding to the maximum overlapping field of view to obtain an interpolated low-resolution hyperspectral image X; S103, taking the cropped aligned image of the full-color CCD chip (5) as the high-resolution full-color image Y, and fusing it with the low-resolution hyperspectral image X to obtain the high-resolution hyperspectral image Z; In step S102, when the cropped and aligned images of multiple pixel-level mosaic coated CCD chips are combined into a single band image and stacked to obtain a low-resolution high-spectral image, the process includes: The cropped and aligned images of pixel-level mosaic coated CCD chips form a single-band image, with a total number of bands c1. The image size of each band is , and any pixel coordinate The pixel value corresponding to the position The calculation function expression is: ， In the above formula, is the upper integer of a / c2, so that is F1 or F2, F1 and F2 are respectively the cropped and aligned images of the first pixel-level mosaic coated CCD chip (3) and the second pixel-level mosaic coated CCD chip (4), It is the floor value of (a-1) / n, where a is the serial number of the band and c2 is half of the total number of bands c1.

2. The high-resolution hyperspectral video fusion imaging method according to claim 1, characterized in that: Obtaining the cropped aligned images corresponding to the maximum overlapping field of view in the original image obtained by the three imaging chips in step S101 includes: dedistorting the original images obtained by the three imaging chips and obtaining the maximum overlapping field of view of the three imaging chips according to the calibration parameters obtained by pre-calibrating the high-resolution hyperspectral video fusion imaging device, and then cropping the cropped aligned images corresponding to the maximum overlapping field of view from the original images.

3. The high-resolution hyperspectral video fusion imaging method according to claim 1, characterized in that: When the interpolated low-resolution hyperspectral image X is obtained by interpolating the size of the cropped aligned image corresponding to the maximum overlapping field of view in step S102, the interpolated low-resolution hyperspectral image X includes each target point to be interpolated in the interpolated low-resolution hyperspectral image X. Estimate pixel values ​​and complete pixel values ​​for each target point to be interpolated Estimated pixel values ​​include: S201, calculate the target point to be interpolated and The Euclidean distance between adjacent pixels; S202, substitute the calculated Euclidean distance into the one-dimensional Gaussian function of the following formula to obtain The weight of neighboring pixels : ， In the above formula, represents the variance, represents the scale factor, The interpolation target point and Among the neighboring pixels The Euclidean distance between adjacent pixels, and The sum of the weights of adjacent pixels is 1; S203, calculate the target point to be interpolated according to the following formula Estimated pixel value of: In the above formula, The target point to be interpolated The estimated pixel value of is the target point to be interpolated in the low-resolution hyperspectral image of Among the neighboring pixels The pixel values ​​of adjacent pixels.

4. The high-resolution hyperspectral video fusion imaging method according to claim 1, characterized in that: In step S103, the cropped aligned image of the full-color CCD chip (5) is used as the high-resolution full-color image Y, and fused with the low-resolution hyperspectral image X to obtain the high-resolution hyperspectral image Z, which includes: S301, establishing an imaging model of a low-resolution hyperspectral image X and a high-resolution panchromatic image Y: ， ， In the above formula, is the two-dimensional matrix of the low-resolution hyperspectral image X expanded along the image band dimension, is the two-dimensional matrix of the high-resolution hyperspectral image Z expanded along the image band dimension, B is the spatial blur operation matrix, and S is the spatial downsampling operation matrix; is the two-dimensional matrix of the high-resolution panchromatic image Y expanded along the image band dimension, and R is the spectral downsampling operation matrix; Establish a low-rank subspace representation of the high-resolution hyperspectral image Z: In the above formula, D represents the spectral subspace, is a two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension; through the imaging model of the low-resolution hyperspectral image X and the high-resolution panchromatic image Y and the low-rank subspace representation of the high-resolution hyperspectral image Z, the problem of fusing the high-resolution panchromatic image Y with the low-resolution hyperspectral image X is converted into the problem of estimating the spectral subspace D and its subspace coefficient matrix C; S302, solving the problem of estimating the spectral subspace D and its subspace coefficient matrix C; The problem of solving the estimation of spectral subspace D includes: 1) Perform singular value decomposition on the low-resolution hyperspectral image X: ， In the above formula, and is the chief matrix, is a diagonal matrix; 2) Keep the diagonal matrix Specified quantity The largest singular value, we get the following low-dimensional decomposition: ， In the above formula, is the low-dimensional part of the low-resolution hyperspectral image X, Chief Matrix The low-dimensional part of is a diagonal matrix The low-dimensional part of Chief Matrix The low-dimensional part of , and: ， ， ； 3) The Chief Matrix The low-dimensional part of is used as the estimated spectral subspace D; The problem of solving the subspace coefficient matrix C includes: 1) Establish an optimization problem model for solving the subspace coefficient matrix C: ， In the above formula, Represents the prior constraints on the subspace coefficient matrix C; 2) Divide the high-resolution full-color image Y into several spatial sizes Then cluster these image blocks into Class image block: ， In the above formula, For the Class image block, For the The j-th image patch of class, For the The number of similar image blocks in the same image block, N is the number of clusters; 3) Based on the fact that the high-resolution hyperspectral image Z and the subspace coefficient matrix C have the same Frobenius norm, the image block clustering structure learned from the high-resolution panchromatic image Y is applied to the subspace coefficient matrix C to obtain: ， In the above formula, For the Class coefficient block, For the The j-th coefficient block of the class, and Spatial location and The jth image patch of class Same, each Class coefficient block Composing tensors , tensor The three dimensions represent the number of similarity coefficient blocks in the class, the spectral subspace dimension, and the number of pixels in the image block; 4) For tensors By imposing a tensor low-rank prior constraint, the optimization problem model for solving the subspace coefficient matrix C is transformed into solving the following optimization problem: ， In the above formula, is the regularization parameter, is the regularization term of the tensor rank, is the two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension; an additional variable required for solving the optimization problem by an alternating direction multiplication method is introduced , and the augmented Lagrangian function is obtained: In the above formula, is the augmented Lagrangian function, is the penalty factor, is the Lagrange multiplier, For tensors The representation of in the form of variable V; 5) Convert the minimization optimization of the augmented Lagrangian function into iterative solutions to the following sub-problems: Subproblem (a): Estimate the two-dimensional matrix obtained by expanding the subspace coefficient matrix C according to the channel dimension : ， In the above formula, and They are tensors and The three-dimensional expansion matrix of Subproblem (b): Estimating variables : ， Subproblem (c): Estimating variables : ， 6) Use the alternating direction multiplier method to iterate and update the variables for the above sub-problems in turn , as well as , when the number of iterations reaches the preset maximum number of iterations When , the final subspace coefficient matrix C is expanded according to the channel dimension to obtain a two-dimensional matrix ; S303, a two-dimensional matrix is ​​obtained by expanding the estimated spectral subspace D and the final subspace coefficient matrix C according to the channel dimension ,according to Get the two-dimensional matrix of the high-resolution hyperspectral image Z after it is expanded along the image band dimension , as the high-resolution panchromatic image Y is fused with the low-resolution hyperspectral image X to obtain the high-resolution hyperspectral image Z.

5. The high-resolution hyperspectral video fusion imaging method according to claim 1, characterized in that: The pixel-level mosaic coating CCD chips all include a CCD chip and a pixel-level mosaic coating arranged on the light incident side of the CCD chip.

6. The high-resolution hyperspectral video fusion imaging method according to claim 2, characterized in that: The pixel-level mosaic coating is composed of a plurality of pixel-level mosaic units arranged repeatedly, and the pixel-level mosaic unit is composed of a plurality of pixel-level mosaics, and different pixel-level mosaics in the pixel-level mosaic unit have different filtering bands.

7. A high-resolution hyperspectral video fusion imaging system, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the high-resolution hyperspectral video fusion imaging method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored therein, characterized in that: The computer program is used to be programmed or configured by a microprocessor to execute the high-resolution hyperspectral video fusion imaging method according to any one of claims 1 to 4.

Images