Method and system for generating hyperspectral images based on hyperspectral and multispectral image fusion
By fusing singular value decomposition and deep generative networks to generate hyperspectral images, the problem of difficulty in acquiring hyperspectral video data is solved, and fast and low-cost hyperspectral image generation is achieved.
Patent Information
- Application Number
- CN202211118069.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-14
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2042-09-14
AI Technical Summary
Existing technologies face difficulties in acquiring and processing hyperspectral video data, resulting in high costs and low efficiency.
By extracting the left low-rank vector U from the hyperspectral image through singular value decomposition and generating the right low-rank vector V using a trained deep generative network, and then combining the hyperspectral image with multispectral image fusion, the cost is reduced and the efficiency is improved.
It enables rapid acquisition of hyperspectral images, reduces costs and improves processing efficiency, has a wide range of applications, and strong noise resistance.
Smart Images
Figure CN115456923B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to hyperspectral image generation technology, specifically to a method and system for generating hyperspectral images based on the fusion of hyperspectral and multispectral images. Background Technology
[0002] Compared to traditional panchromatic or RGB images, hyperspectral images have more spectral bands, covering the visible light spectrum to the short-wave infrared spectrum. Since different materials have different reflectivities, the rich spectral information in hyperspectral images is more beneficial for analyzing the physicochemical properties of objects. Therefore, hyperspectral images are widely used in remote sensing, medical imaging, geological exploration, and facial recognition. Currently, the main types of hyperspectral imagers on the market are spectral scan, swing scan, and push-broom types. Due to limitations in optical imaging hardware, it is difficult to quickly acquire high-resolution hyperspectral video data. Furthermore, the acquired hyperspectral video data needs to be stored and transmitted to other devices for processing, resulting in a long processing cycle. Summary of the Invention
[0003] The technical problem to be solved by this invention is to address the difficulties in acquiring and processing hyperspectral video data in a timely manner. This invention provides a method and system for generating hyperspectral images based on the fusion of hyperspectral and multispectral images. This invention can generate hyperspectral images based on the fusion of hyperspectral and multispectral images, thereby reducing the cost and improving the efficiency of hyperspectral image acquisition.
[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0005] A method for generating hyperspectral images based on the fusion of hyperspectral and multispectral images includes:
[0006] 1) Input hyperspectral image Y Subspace decomposition is performed to extract the left low-rank vector U of the hyperspectral image to be generated; the input multispectral image is then processed. Z The right low-rank vector V of the hyperspectral image to be generated is generated by the trained deep generative network.
[0007] 2) The left low-rank vector of the hyperspectral image to be generated U and right low-rank vector V Fusion to generate hyperspectral images X .
[0008] Optionally, the deep generative network in step 1) includes forward computation units and inverse computation units. Both forward and inverse computation units consist of multiple cascaded two-dimensional convolutional layers. The scales of the multiple two-dimensional convolutional layers in the forward computation unit are arranged from largest to smallest, while the scales of the multiple two-dimensional convolutional layers in the inverse computation unit are arranged from smallest to largest. Two-dimensional convolutional layers of the same scale between the forward and inverse computation units are connected in a skip connection manner. Adjacent two-dimensional convolutional layers within the same unit are connected through corresponding upsampling and downsampling operations. (Multispectral image) Z The input is a right low-rank vector from the largest two-dimensional convolutional layer in the forward operation unit, and the output is a right low-rank vector from the largest two-dimensional convolutional layer in the inverse operation unit. V .
[0009] Optionally, each two-dimensional convolutional layer contains, in sequence, one two-dimensional convolution, one two-dimensional batch normalization, one linear rectification, and one downsampling.
[0010] Optionally, the loss function used for training the deep generative network in step 1) is the root mean square error (RMSE).
[0011] Optionally, in step 1), the input hyperspectral image Y Subspace decomposition specifically refers to decomposing the input hyperspectral image into subspaces. Y Perform singular value decomposition (SVD), and use the left singular vector obtained from SVD as the left low-rank vector U of the hyperspectral image to be generated; and use the input hyperspectral image... Y The function expression for performing singular value decomposition is:
[0012] Y = U 1 Σ 1 V 1
[0013] In the above formula, U 1 is a matrix composed of left singular vectors. Σ 1 is a diagonal matrix composed of singular values. V 1 is the transpose of the matrix composed of right singular vectors.
[0014] Optionally, the hyperspectral image in step 1). Y The hyperspectral image is obtained from image frames in a hyperspectral video, and is based on the original image frames in the hyperspectral video. Y This includes: performing image distortion correction on the original image frames in the hyperspectral video using the distortion coefficient matrix of the imaging sensor obtained by Zhang Zhengyou's calibration method; cropping images of each spectral band, aligning them, and stacking them in band order to obtain the hyperspectral image. Y .
[0015] Optionally, the multispectral image in step 1).Z The images are derived from image frames in a multispectral video, and multispectral images are obtained based on the original image frames in the multispectral video. Z This includes: performing image distortion correction on the original image frames in the multispectral video using the distortion coefficient matrix of the imaging sensor obtained by Zhang Zhengyou's calibration method, and comparing it with the hyperspectral image... Y The correspondence was cropped out to match the hyperspectral image. Y The same region to obtain multispectral images Z .
[0016] Furthermore, the present invention also provides the aforementioned system for generating hyperspectral images based on the fusion of hyperspectral and multispectral images, comprising a primary mirror, an aperture, a beam splitter, a collimating lens, a filter array and a microlens array, a hyperspectral image sensor, a reflector, an imaging lens, and a multispectral image sensor. The light captured by the primary mirror is split into two paths after passing through the aperture and the beam splitter. One path of light passes sequentially through the collimating lens, the filter array, and the microlens array before being imaged on the hyperspectral image sensor to obtain the original image frame in the hyperspectral video. The other path of light passes sequentially through the reflector and the imaging lens before being imaged on the multispectral image sensor to obtain the original image frame in the multispectral video.
[0017] Furthermore, the present invention also provides a system for generating hyperspectral images based on the fusion of hyperspectral and multispectral images, comprising a processing computing unit and a memory interconnected thereto, wherein the processing computing unit is programmed or configured to perform the steps of the method for generating hyperspectral images based on the fusion of hyperspectral and multispectral images.
[0018] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program for being programmed or configured by a processing computing unit to perform the steps of the method for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images.
[0019] Compared with the prior art, the present invention has the following advantages: the present invention inputs a hyperspectral image Y Perform singular value decomposition to extract the left singular vector, which is then used as the left low-rank vector U of the hyperspectral image to be generated; the input multispectral image is then... Z The trained deep generative network generates the right low-rank vector V of the hyperspectral image to be generated; the left low-rank vector of the hyperspectral image to be generated is then used. U and right low-rank vector V Fusion to generate hyperspectral images X This enables the generation of hyperspectral images based on the fusion of hyperspectral and multispectral images, reducing the cost and efficiency of hyperspectral image acquisition. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.
[0021] Figure 2 This is a schematic diagram of the structure of a deep generative network in an embodiment of the present invention.
[0022] Figure 3 This is a structural diagram of the system in an embodiment of the present invention.
[0023] Figure 4 This is a schematic diagram of the optical path of the system in an embodiment of the present invention.
[0024] Figure 5 This is a flowchart of the hardware system operation in an embodiment of the present invention. Detailed Implementation
[0025] like Figure 1 As shown, the method for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images in this embodiment includes:
[0026] 1) Input hyperspectral image Y Perform singular value decomposition to extract the left singular vector, which is then used as the left low-rank vector U of the hyperspectral image to be generated; the input multispectral image is then... Z The right low-rank vector V of the hyperspectral image to be generated is generated by the trained deep generative network.
[0027] 2) The left low-rank vector of the hyperspectral image to be generated U and right low-rank vector V Fusion to generate hyperspectral images X .
[0028] like Figure 2 As shown, the deep generative network in step 1) includes forward computation units and inverse computation units. Both forward and inverse computation units consist of multiple cascaded two-dimensional convolutional layers. The scales of the convolutional layers in the forward computation unit are arranged from largest to smallest, while the scales of the convolutional layers in the inverse computation unit are arranged from smallest to largest. Two-dimensional convolutional layers of the same scale between the forward and inverse computation units are connected in a skip connection manner. Adjacent two-dimensional convolutional layers are connected through corresponding upsampling and downsampling operations. (Multispectral image) Z The input is a right low-rank vector from the largest two-dimensional convolutional layer in the forward operation unit, and the output is a right low-rank vector from the largest two-dimensional convolutional layer in the inverse operation unit. V See also Figure 2 Multispectral images ZThe scale is HW*S, where H is the height, W is the width, and S is the number of multispectral image bands. In this embodiment, both the forward and reverse operation units are composed of four cascaded two-dimensional convolutional layers, with the scales of the four two-dimensional convolutional layers being HW / 2*S / 2, HW / 4*S / 4, HW / 8*S / 8, and HW / 16*S / 16, respectively.
[0029] In this embodiment, each two-dimensional convolutional layer contains, in sequence, one two-dimensional convolution, one two-dimensional batch normalization, one linear rectification, and one downsampling.
[0030] In this embodiment, the loss function used for training the deep generative network in step 1) is the root mean square error (RMSE). Furthermore, this embodiment employs a self-supervised method to train the deep generative network. Using a self-supervised method for network training reduces the data requirement, solves the problem of scarce training data, and is easy to implement. Since the self-supervised method for training the deep generative network is a known training method, its implementation details will not be elaborated here.
[0031] In this embodiment, step 1) involves inputting a hyperspectral image. Y Subspace decomposition specifically refers to decomposing the input hyperspectral image into subspaces. Y Perform singular value decomposition (SVD), and use the left singular vector obtained from SVD as the left low-rank vector U of the hyperspectral image to be generated; and use the input hyperspectral image... Y The function expression for performing singular value decomposition is:
[0032] Y = U 1 Σ 1 V 1
[0033] In the above formula, U 1 is a matrix composed of left singular vectors. Σ 1 is a diagonal matrix composed of singular values. V 1 represents the transpose of the matrix composed of right singular vectors. The input hyperspectral image... Y When performing singular value decomposition, take a specified number of... L The largest singular value can be obtained by performing singular value decomposition according to the above formula. Since the singular value decomposition method is an existing method, its details will not be elaborated here.
[0034] In this embodiment, the hyperspectral image in step 1) Y The hyperspectral image is obtained from image frames in a hyperspectral video, and is based on the original image frames in the hyperspectral video. YThis includes: performing image distortion correction on the original image frames in the hyperspectral video using the distortion coefficient matrix of the imaging sensor obtained by Zhang Zhengyou's calibration method; cropping images of each spectral band, aligning them, and stacking them in band order to obtain the hyperspectral image. Y .
[0035] In this embodiment, the multispectral image in step 1) Z The images are derived from image frames in a multispectral video, and multispectral images are obtained based on the original image frames in the multispectral video. Z This includes: performing image distortion correction on the original image frames in the multispectral video using the distortion coefficient matrix of the imaging sensor obtained by Zhang Zhengyou's calibration method, and comparing it with the hyperspectral image... Y The correspondence was cropped out to match the hyperspectral image. Y The same region to obtain multispectral images Z .
[0036] In this embodiment, step 2) involves using the left low-rank vector of the hyperspectral image to be generated. U and right low-rank vector V Fusion to generate hyperspectral images X It can be represented as: X = U × V.
[0037] like Figure 3 and Figure 4As shown, this embodiment also provides a system for generating hyperspectral images based on the fusion of hyperspectral and multispectral images as described above, including a primary mirror 1, an aperture 2, a beam splitter 3, a collimating lens 4, a filter array and a microlens array 5, a hyperspectral image sensor 6, a reflector 7, an imaging lens 8, and a multispectral image sensor 9. The light captured by the primary mirror 1 is split into two paths after passing through the aperture 2 and the beam splitter 3. One path of light passes sequentially through the collimating lens 4, the filter array, and the microlens array 5 and is then imaged on the hyperspectral image sensor 6 to obtain the original image frame in the hyperspectral video; the other path of light passes sequentially through the reflector 7 and the imaging lens 8 and is then imaged on the multispectral image sensor 9 to obtain the original image frame in the multispectral video. In this specific implementation, the primary mirror 1 is a variable-focus optical lens used to capture the light emitted by the target and perform the first imaging at the aperture stop 2; the aperture stop 2 is a physical entity that limits the beam and restricts the imaging range of the primary mirror 1; the beam splitter 3 is a beam splitter prism used to split the beam limited by the aperture stop M into two identical and mutually perpendicular beams, one beam directed towards the collimating lens 4 and the other towards the reflecting mirror 7; the collimating lens 4 is used to maintain the collimation of the beam directed towards the filter array and the microlens array; the filter array and the microlens array 5 include: the filter array: the filters can be used to select the desired radiation band, wherein the filter array consists of 63 selected radiation bands of 400~1000 Å. An array of filters with different apertures (m) is used; the microlens array consists of 63 lenses with apertures and relief depths in the micrometer range, each capable of independent imaging, with the imaging surface located on the photosensitive surface of the hyperspectral image sensor 6; the hyperspectral image sensor, a panchromatic camera, receives light signals and converts them into electrical signals for acquiring hyperspectral video data; the reflector 7 reflects another beam of light split by the beam splitter 3, changing the direction of the light path so that it is directed towards the imaging lens 8; the imaging lens 8 focuses the light for imaging, with the imaging surface located on the photosensitive surface of the multispectral image sensor 9; the multispectral image sensor 9, an RGB camera, receives light signals and converts them into electrical signals for acquiring multispectral video data. The hyperspectral image sensor 6 and the multispectral image sensor 9 are connected to the processing and computing unit via a data acquisition module, allowing the processing and computing unit to execute the steps of the aforementioned method for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images.
[0038] Furthermore, this embodiment also provides a system for generating hyperspectral images based on the fusion of hyperspectral and multispectral images, including interconnected processing and computing units and memory. The processing and computing units are programmed or configured to execute the steps of the aforementioned method for generating hyperspectral images based on the fusion of hyperspectral and multispectral images. It should be noted that the processing and computing unit here can be a single microprocessor or a combination of a microprocessor and its accelerator processor. For example, the system in this embodiment also includes a small artificial intelligence computing motherboard, employing a CPU + GPU combination: the CPU is a Quad-core ARMA57 with a clock speed of 1.43GHz; the GPU uses the NVIDIA Maxwell architecture, with 128 NVIDIA CUDA cores, providing application programming interfaces (APIs) for AI and computer vision. In addition, the system's memory includes 4GB of 64-bit LPDDR4, and it provides one power interface, one HDMI interface, one DP interface, four USB 3.0 Type A interfaces, one Ethernet interface, one Micro USB B interface, two camera connection ports, and several interfaces for development use. Its mechanical dimensions are 100mm × 80mm × 29mm. Figure 5As shown, after the target emits light, it is captured by the primary mirror and then processed by the system described above for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images. Hyperspectral and multispectral imaging is then performed. When the imaging sensors (hyperspectral image sensor 6 and multispectral image sensor 9) receive the control command to start acquisition from the processing unit, they enter the working mode, converting the light signal on the photosensitive surface into an electrical signal and transmitting the hyperspectral and multispectral video data to the processing unit via a data link. The processing unit uses the aforementioned method for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images (referred to as spatial-spectral super-resolution intelligent fusion) to process the data, obtaining a high-resolution hyperspectral image and further acquiring video data. If further processing is required, the processing unit performs intelligent processing according to the deployed algorithm and outputs the results; otherwise, it directly outputs high-resolution hyperspectral video image data. After receiving the control command to stop acquisition from the processing unit, the imaging sensors (hyperspectral image sensor 6 and multispectral image sensor 9) stop working and enter standby mode, waiting for the next control command; otherwise, they maintain the working mode. Furthermore, a wireless communication module can be added to the processing unit as needed to enable wireless communication with the host computer, transmit processing results, or utilize the host computer for image fusion processing to obtain high-resolution hyperspectral images. The system provided in this embodiment, which generates hyperspectral images based on the fusion of hyperspectral and multispectral images, integrates hyperspectral video acquisition and intelligent processing functions, and has a wide range of applications. In summary, this embodiment provides a system that integrates hyperspectral video acquisition and intelligent processing functions, realizing hyperspectral video acquisition and spatial-spectral super-resolution intelligent real-time processing, with advantages such as fast acquisition speed, strong noise resistance, and wide applicability.
[0039] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by a processing computing unit to perform the steps of the aforementioned method for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images.
[0040] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing computing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing computing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0041] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A method for generating hyperspectral images based on the fusion of hyperspectral and multispectral images, characterized in that, include: 1) Input hyperspectral image Y Subspace decomposition is performed to extract the left low-rank vector U of the hyperspectral image to be generated; the input multispectral image is then processed. Z The right low-rank vector V of the hyperspectral image to be generated is generated by the trained deep generative network. 2) The left low-rank vector of the hyperspectral image to be generated U and right low-rank vector V Fusion to generate hyperspectral images X ; The deep generative network in step 1) includes forward computation units and inverse computation units. Both forward and inverse computation units consist of multiple cascaded two-dimensional convolutional layers. The scales of the convolutional layers in the forward computation unit are arranged from largest to smallest, while the scales of the convolutional layers in the inverse computation unit are arranged from smallest to largest. Convolutional layers of the same scale between the forward and inverse computation units are connected in a skip connection manner. Adjacent convolutional layers within the same unit are connected through corresponding upsampling and downsampling operations. (Multispectral image) Z The input is a right low-rank vector from the largest two-dimensional convolutional layer in the forward operation unit, and the output is a right low-rank vector from the largest two-dimensional convolutional layer in the inverse operation unit. V ; Step 1) involves inputting the hyperspectral image. Y Subspace decomposition specifically refers to decomposing the input hyperspectral image into subspaces. Y Perform singular value decomposition (SVD), and use the left singular vector obtained from SVD as the left low-rank vector U of the hyperspectral image to be generated; and use the input hyperspectral image... Y The function expression for performing singular value decomposition is: Y = U 1 Σ 1 V 1 In the above formula, U 1 is a matrix composed of left singular vectors. Σ 1 is a diagonal matrix composed of singular values. V 1 is the transpose of the matrix composed of right singular vectors.
2. The method for generating hyperspectral images based on the fusion of hyperspectral and multispectral images according to claim 1, characterized in that, Each two-dimensional convolutional layer contains, in sequence, one two-dimensional convolution, one two-dimensional batch normalization, one linear rectification, and one downsampling.
3. The method for generating hyperspectral images based on the fusion of hyperspectral and multispectral images according to claim 2, characterized in that, The loss function used for training the deep generative network in step 1) is the root mean square error (RMSE).
4. The method for generating hyperspectral images based on the fusion of hyperspectral and multispectral images according to claim 1, characterized in that, Hyperspectral image in step 1) Y The hyperspectral image is obtained from image frames in a hyperspectral video, and is based on the original image frames in the hyperspectral video. Y This includes: performing image distortion correction on the original image frames in the hyperspectral video using the distortion coefficient matrix of the imaging sensor obtained by Zhang Zhengyou's calibration method; cropping images of each spectral band, aligning them, and stacking them in band order to obtain the hyperspectral image. Y .
5. The method for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images according to claim 1, characterized in that, Multispectral images in step 1) Z The images are derived from image frames in a multispectral video, and multispectral images are obtained based on the original image frames in the multispectral video. Z This includes: performing image distortion correction on the original image frames in the multispectral video using the distortion coefficient matrix of the imaging sensor obtained by Zhang Zhengyou's calibration method, and comparing it with the hyperspectral image... Y The correspondence was cropped out to match the hyperspectral image. Y The same region to obtain multispectral images Z .
6. A system for generating hyperspectral images based on the fusion of hyperspectral and multispectral images as described in any one of claims 1 to 5, characterized in that, The system includes a primary mirror (1), an aperture (2), a beam splitter (3), a collimating lens (4), a filter array and a microlens array (5), a hyperspectral image sensor (6), a reflector (7), an imaging lens (8), and a multispectral image sensor (9). The light captured by the primary mirror (1) is split into two paths after passing through the aperture (2) and the beam splitter (3). One path of light passes through the collimating lens (4), the filter array and the microlens array (5) in sequence and is then imaged on the hyperspectral image sensor (6) to obtain the original image frame in the hyperspectral video. The other path of light passes through the reflector (7) and the imaging lens (8) in sequence and is then imaged on the multispectral image sensor (9) to obtain the original image frame in the multispectral video.
7. A system for generating hyperspectral images based on the fusion of hyperspectral and multispectral images, comprising interconnected processing and computing units and a memory, characterized in that, The processing unit is programmed or configured to perform the steps of the method for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to program or configure the processed computing unit to perform the steps of the method for generating a hyperspectral image based on the fusion of hyperspectral and multispectral images as described in any one of claims 1 to 5.