A UAV RGB Image Spectral Reconstruction Method Based on Band Differential

Abstract

The present invention discloses a method for spectral reconstruction of UAV RGB images based on band difference. First, collect UAV aerial RGB images and hyperspectral auxiliary images to construct a dataset with RGB and hyperspectral images. Secondly, use band difference and superpixel segmentation to generate training data. Then, establish a spectral mapping model based on a deep network and train to obtain the spectral mapping relationship from RGB images to band difference sequences. Further, use the spectral mapping relationship obtained by training the model to perform spectral reconstruction on any RGB image collected by the UAV to obtain the difference sequence of the target image. Finally, perform inverse difference processing on the difference sequence to obtain the reconstructed target image. By introducing the difference idea to define the relationship between adjacent bands of the hyperspectral auxiliary image, the present invention can represent both the intra-band information and the inter-band correlation information of the hyperspectral auxiliary image, and can effectively improve the quality of spectral reconstruction of RGB images.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and particularly relates to a method for spectral reconstruction of UAV RGB images based on band difference. Background Technique

[0002] Using unmanned aerial vehicles to carry out operations in environmental areas where it is difficult to reach by force and where the terrain conditions are complex can obtain the target distribution and its accurate positioning in the shortest time, providing on-site data support for command and decision-making. Compared with the image acquisition platform based on manned aircraft, the UAV image acquisition technology has the advantages of strong autonomy, simple operation, more comprehensive monitoring, lower cost and consumption, etc., and has become a research hotspot in the field of information technology.

[0003] In practical applications, an RGB color camera is often mounted on a UAV to obtain target images, aiming to mimic the three-color perception of humans and make the images more in line with human visual habits. However, the RGB images collected by UAVs lack spectral information and cannot reflect the spectral curves and differences of different objects. Spectral reconstruction of UAV RGB images to recover the spectral information of RGB images can effectively improve the ability to identify and distinguish target categories in the scene, and has important theoretical significance and application value. The most basic problem faced by UAV RGB image spectral reconstruction is information loss, that is, recovering a high-light image from a small amount of spectral information is an ill-posed problem and there is serious loss itself. A commonly used method is the method based on image fusion, which guides the spectral reconstruction of RGB images by introducing an auxiliary low-resolution hyperspectral image containing rich spectral information, as Figure 1 shown. Therefore, it is necessary to additionally mount a hyperspectral camera on the UAV to introduce information outside the RGB image data to participate in the spectral reconstruction of RGB images.

[0004] In fact, the hyperspectral auxiliary image not only contains a large amount of information within the spectral bands, but also contains rich correlation information between spectral bands, as Figure 2 shown. It should be noted that when using the hyperspectral auxiliary image to guide the spectral reconstruction of RGB images, existing reconstruction methods often only focus on the information within the spectral bands of the auxiliary image and ignore the correlation information between bands. If as much correlation information between spectral bands as possible can be fully utilized, the overall reconstruction performance will surely be further improved, and it is expected to further enhance the ability of UAVs to perceive target scene information.

[0005] Inspired by the above, in the present invention, an attempt is made to establish a perception mechanism for the correlation information between appropriate spectral frequency bands, and further make full use of the spectral information of the hyperspectral auxiliary image to participate in the spectral reconstruction of the UAV RGB image. In the field of electrical engineering, people often use the voltage difference between two signals (i.e., differential signal) as the signal input of the circuit, which not only contains the information of the two signals, but also effectively reflects the relationship between the two signals, so as to improve the application efficiency of the signal. Summary of the Invention

[0006] The present invention aims to design a method for spectral reconstruction of UAV RGB images, which can simultaneously perceive and utilize the intra-band information and inter-band correlation information of the hyperspectral auxiliary image, improve the overall reconstruction effect of the RGB image, and further enhance the ability of the UAV to perceive the target scene information.

[0007] To achieve the above object, the present invention adopts the following technical solutions: A method for spectral reconstruction of UAV RGB images based on band difference, characterized by comprising the following steps:

[0008] Step 1, collect RGB images and hyperspectral auxiliary images to construct a dataset with RGB and hyperspectral auxiliary images. Assume that the constructed dataset contains N RGB images and hyperspectral auxiliary images, that is, N pairs of images, a total of 2N images, where each RGB image is represented as X n , and each hyperspectral image is represented as H n , where 1 ≤ n ≤ N.

[0009] Step 2, use band difference and superpixel segmentation to form training data, specifically:

[0010] Segment each RGB image in the dataset into Q colored irregular image patches;

[0011] Segment each hyperspectral auxiliary image in the dataset into Q band-difference irregular image patches;

[0012] Step 3, establish a spectral mapping model based on a deep network. Use the colored irregular image patches of the RGB image and the band-difference irregular image patches of its hyperspectral auxiliary image as training data, and train the spectral mapping model to obtain the spectral mapping relationship T from the RGB image to the band-difference sequence, that is:

[0013]

[0014] Among them, represents the loss function of the spectral mapping model, is the parameter of the network, β is the penalty parameter, represents the jth colored irregular image patch, Denote the j-th band differential irregular image block, where 1 ≤ j ≤ NQ; here, N represents the number of RGB and hyperspectral auxiliary image samples, Q is the number of irregular blocks corresponding to each sample, and NQ is the number of generated color irregular image blocks and band differential irregular image blocks in the dataset. Since the differential restricts the information of adjacent bands of the training data (the irregular image blocks corresponding to the hyperspectral auxiliary image ), the spectral mapping relationship T optimized through the above deep network model training method not only reflects the information within each band of the hyperspectral auxiliary image but also contains the correlation information between bands.

[0015] Step 4: For the RGB image to be spectrally reconstructed Utilize the spectral mapping relationship T to obtain the differential sequence y of the target image, which is expressed as follows:

[0016]

[0017] Step 5: Perform inverse difference processing on the differential sequence y of the target image to obtain the target image after RGB image reconstruction.

[0018] Furthermore, in Step 2, each RGB image in the dataset is segmented into Q color irregular image blocks, specifically:

[0019] Perform sparse principal component analysis (SPCA) dimensionality reduction on each RGB image to obtain the principal component of the RGB image, and then use the superpixel segmentation algorithm to calculate the superpixel labels of the principal component to obtain Q color irregular image blocks;

[0020] Each hyperspectral auxiliary image in the dataset is segmented into Q band differential irregular image blocks, specifically: Perform differential limitation on each hyperspectral auxiliary image, calculate the difference between adjacent bands to obtain the band differential sequence of the auxiliary hyperspectral image, upsample the band differential sequence to obtain a band differential sequence with the same spatial resolution as the RGB image; segment the upsampled band differential sequence using the superpixel segmentation algorithm to obtain Q band differential irregular image blocks.

[0021] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0022] The method for spectral reconstruction of UAV RGB images based on band differential proposed by the present invention restricts the relationship between adjacent bands of the hyperspectral auxiliary image by introducing the differential idea, obtains the band differential sequence of the hyperspectral auxiliary image, can simultaneously describe the information within the spectral band and the correlation information between bands, and combines a deep neural network to obtain the spectral mapping relationship from the RGB image to the target image, overcoming the problem that traditional reconstruction methods ignore the correlation information between bands. Description of the Drawings

[0023] Figure 1 is a spectral reconstruction diagram of an RGB image based on image fusion;

[0024] Figure 2 is a schematic diagram of information within each spectral band and correlation information between spectral bands in a hyperspectral image;

[0025] Figure 3 is a flowchart of the steps of the present invention;

[0026] Figure 4 is a schematic diagram of generating training data based on band difference and superpixels. Detailed Embodiment

[0027] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0028] This embodiment relates to a method for spectral reconstruction of UAV RGB images based on band difference. By introducing the difference idea to define the relationship between adjacent bands of a hyperspectral auxiliary image, a band difference sequence of the hyperspectral auxiliary image is obtained, which can simultaneously describe the information within the spectral band and the correlation information between bands, overcoming the problem that traditional reconstruction methods ignore the correlation information between bands. The overall flowchart of the present invention is as Figure 3 shown, and the main steps involved are as follows:

[0029] Step 1: Collect UAV aerial RGB images and hyperspectral auxiliary images to construct a dataset with RGB and hyperspectral images. Assume that the constructed dataset contains N RGB images and hyperspectral images (i.e., N pairs of images, a total of 2N images). Among them, each RGB image is represented as X n , and each hyperspectral image is represented as H n , where 1 ≤ n ≤ N.

[0030] Step 2: Use band difference and superpixel segmentation to establish a training data generation method, as Figure 4 shown. Specifically:

[0031] 1) In order to fully consider the characteristics of each region in the image, use the superpixel segmentation algorithm to perform superpixel segmentation on each RGB image sample to obtain multiple colored irregular image blocks, which are used as one of the inputs of the deep network spectral mapping model. Suppose there are N RGB image samples in total, and each RGB image can be segmented into Q colored irregular image blocks through superpixel segmentation, then the model input has a total of NQ colored irregular image blocks.

[0032] 2) To simultaneously describe the information within the frequency bands and the correlation information between the frequency bands in the hyperspectral auxiliary images, each auxiliary image in the sample is differentially defined, that is, the difference between adjacent frequency bands is calculated to obtain the frequency band difference sequence of the auxiliary image. The sequence is upsampled to obtain a frequency band difference sequence with the same spatial resolution as the RGB image, and through superpixel segmentation operation, a series of irregular frequency band difference image patches are obtained as the second input to the deep network model. The colored irregular image patches and the irregular frequency band difference image patches correspond one by one;

[0033] Here, it should be noted that by performing dimensionality reduction on each RGB image through sparse principal component analysis (SPCA), its principal component is obtained, and then the superpixel labels of this principal component are calculated using the superpixel segmentation algorithm. Using the labels for the RGB image and the frequency band difference sequence after upsampling, the corresponding colored irregular image patches and irregular frequency band difference image patches can be generated.

[0034] Step 3: Establish a spectral mapping model based on a deep network, and train to obtain the spectral mapping relationship from the RGB image to the frequency band difference sequence. The spectral mapping model can be any deep learning network framework; assume that the colored irregular image patches and irregular frequency band difference image patches generated above as training data are respectively represented as and where N represents the number of RGB and hyperspectral auxiliary image samples, Q is the number of irregular patches corresponding to each sample, and NQ is the number of colored irregular image patches and irregular frequency band difference image patches generated in the dataset. Based on this, a spectral mapping model based on a deep network is established, and the spectral mapping relationship T from the RGB image to the frequency band difference sequence is trained, that is:

[0035]

[0036] where, represents the loss function of the spectral mapping model, is the parameter of the network, β is the penalty parameter, and penalty(T) is the penalty term. Since the difference limits the information of adjacent frequency bands of the training data (the irregular image patches corresponding to the hyperspectral auxiliary images ), the spectral mapping relationship T optimized through the above deep network model training method not only reflects the information within each frequency band of the hyperspectral auxiliary image, but also contains the correlation information between the frequency bands.

[0037] Step 4: Use the spectral mapping relationship T obtained by training the model (1) to perform spectral reconstruction on any RGB image collected by the drone to obtain the difference sequence of the target image. Assume that a certain RGB image to be reconstructed is Then the difference sequence y of the target image (i.e., the reconstructed image) can be obtained by the following formula:

[0038]

[0039] Step Five: Perform inverse difference processing on the difference sequence y to obtain the reconstructed target image.

[0040] Compared with most traditional RGB image spectral reconstruction methods, the present invention can simultaneously represent the information within the frequency bands and the inter-band correlation information of the hyperspectral auxiliary image, and use them for RGB image spectral reconstruction, which can effectively improve the reconstruction quality.

[0041] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A method for RGB image spectrum reconstruction, characterized in that: The steps include: Step 1, collect RGB images and their hyperspectral auxiliary images, and construct a dataset of RGB images and hyperspectral auxiliary images, including N RGB images and N hyperspectral auxiliary images; Among them, each RGB image is represented as X n , each hyperspectral auxiliary image is represented as H n , 1≤n≤N; Step 2, dividing each RGB image in the data set into Q color irregular image blocks; Each hyperspectral auxiliary image in the data set is divided into Q frequency band difference irregular image blocks, specifically: Each hyperspectral auxiliary image is differentially limited, the difference between adjacent frequency bands is calculated to obtain a frequency band differential sequence of the auxiliary hyperspectral image, the frequency band differential sequence is upsampled to obtain a frequency band differential sequence with the same spatial resolution as the RGB image, and the upsampled frequency band differential sequence is segmented using a superpixel segmentation algorithm to obtain Q frequency band differential irregular image blocks; Step 3: Establish a spectral mapping model based on a deep network and train the spectral mapping relationship T from the RGB image to the frequency band difference sequence, that is: in, represents the loss function of the spectral mapping model, is the network parameter, β is the penalty parameter, penalty(T) is the penalty term, represents the jth color irregular image block, represents the j-th frequency band differential irregular image block, 1≤j≤NQ; Step 4: For the RGB image to be spectrally reconstructed Using the spectral mapping relationship T, the differential sequence y of the target image is obtained, which is expressed as follows: Step 5: Perform reverse difference processing on the differential sequence y of the target image to obtain the target image after RGB image reconstruction.

2. The RGB image spectrum reconstruction method according to claim 1, characterized in that: In step 2, each RGB image in the data set is divided into Q color irregular image blocks, specifically: Sparse principal component analysis (SPCA) is performed on each RGB image to reduce its dimension, and the principal component of the RGB image is obtained. The superpixel label of the principal component is calculated using a superpixel segmentation algorithm to obtain Q color irregular image blocks.

Images