Unmanned aerial vehicle RGB image spectrum reconstruction method based on frequency band difference

Abstract

The invention discloses an unmanned aerial vehicle RGB image spectrum reconstruction method based on frequency band difference, and the method comprises the steps: firstly collecting an unmanned aerial vehicle aerial RGB image and a hyperspectral auxiliary image, and constructing a data set with the RGB image and the hyperspectral image; secondly, generating training data by using frequency band difference and super-pixel segmentation; then, establishing a spectral mapping model based on a deep network, and training to obtain a spectral mapping relation from the RGB image to the frequency band difference sequence; furthermore, spectrum reconstruction is performed on any RGB image acquired by the unmanned aerial vehicle by using a spectrum mapping relation obtained through the training model, and a difference sequence of a target image is obtained. And finally, performing reverse differential processing on the differential sequence to obtain a reconstructed target image. According to the method, the relation between the adjacent frequency bands of the hyperspectral auxiliary image is limited by introducing the difference idea, the information in the frequency bands and the correlation information between the frequency bands of the hyperspectral auxiliary image can be expressed at the same time, and the RGB image spectrum reconstruction quality can be effectively improved.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a method for reconstructing RGB images of unmanned aerial vehicles based on frequency band difference. Background Art

[0002] Using unmanned cameras to carry out operations in areas that are difficult to reach and have complex terrain conditions can obtain target distribution and accurate positioning in the shortest time to provide on-site data support for command decisions. Compared with image acquisition platforms based on manned aircraft, drone image acquisition technology has the advantages of strong autonomy, simple operation, more comprehensive monitoring, and lower cost and consumption, and has become a research hotspot in the field of information technology.

[0003] In practical applications, RGB color cameras are often installed on drones to acquire target images. The purpose is to imitate human three-color perception and make the images more in line with human visual habits. However, the RGB images collected by drones lack spectral information and cannot reflect the spectral curves and differences of different objects. Spectrally reconstructing drone RGB images and restoring the spectral information of RGB images can effectively improve the recognition and distinction capabilities of target categories in the scene, and has important theoretical significance and application value. The most basic problem faced by spectral reconstruction of drone RGB images is information loss, that is, restoring a high-light image from a small amount of spectral information is an ill-posed problem, which itself has serious deficiencies. A commonly used method is based on image fusion, which guides the spectral reconstruction of RGB images by introducing an auxiliary low-resolution high-spectral image containing rich spectral information, such as Figure 1 Therefore, it is necessary to carry a hyperspectral camera on the drone to introduce information other than RGB image data to participate in the RGB image spectrum reconstruction.

[0004] In fact, the hyperspectral auxiliary image contains not only a large amount of information within the spectral band, but also rich correlation information between spectral bands, such as Figure 2 As shown. It is worth noting that when using the hyperspectral auxiliary image to guide the RGB image for spectral reconstruction, the existing reconstruction methods often only focus on the information within the spectral band of the auxiliary image, ignoring the correlation information between the bands. If the correlation information between the spectral bands can be fully utilized as much as possible, the overall reconstruction performance is bound to be further improved, and it is expected to further enhance the ability of drones to perceive target scene information.

[0005] Inspired by the above, in the present invention, an attempt is made to establish a suitable perception mechanism for correlation information between spectral frequency bands, and further make full use of the spectral information of the hyperspectral auxiliary image to participate in the spectral reconstruction of the drone RGB image. In the field of electrical engineering, people often use the voltage difference between two signals (i.e., differential signal) as the signal input of the circuit, which not only contains the information of the two signals, but also effectively reflects the relationship between the two signals, thereby improving the application efficiency of the signal. Summary of the invention

[0006] The present invention aims to design a spectral reconstruction method for UAV RGB images, which can simultaneously perceive and utilize the intra-band information and inter-band correlation information of hyperspectral auxiliary images, improve the overall reconstruction effect of RGB images, and thus further enhance the ability of UAVs to perceive target scene information.

[0007] In order to achieve the above object, the present invention adopts the following technical solution: a method for spectral reconstruction of UAV RGB image based on frequency band difference, characterized in that it comprises the following steps:

[0008] Step 1: Collect RGB images and hyperspectral auxiliary images to construct a dataset with RGB and hyperspectral auxiliary images. Assume that the constructed dataset contains N RGB images and hyperspectral auxiliary images, that is, N pairs of images, a total of 2N images, where each RGB image is represented by X n , each hyperspectral image is represented as H n , where 1≤n≤N.

[0009] Step 2: Use frequency band difference and superpixel segmentation to form training data, specifically:

[0010] Divide each RGB image in the data set into Q color irregular image blocks;

[0011] Segment each hyperspectral auxiliary image in the data set into Q frequency band difference irregular image blocks;

[0012] Step 3: Establish a spectral mapping model based on a deep network. Use the color irregular image blocks of the RGB image and the frequency band difference irregular image blocks of its hyperspectral auxiliary image as training data to train the spectral mapping model. The spectral mapping relationship T from the RGB image to the frequency band difference sequence is obtained through training, that is:

[0013]

[0014] in, represents the loss function of the spectral mapping model, is the network parameter, β is the penalty parameter, represents the jth color irregular image block, represents the j-th frequency band difference irregular image block, 1≤j≤NQ; where N represents the number of RGB and hyperspectral auxiliary image samples, Q is the number of irregular blocks corresponding to each sample, and NQ is the number of color irregular image blocks and frequency band difference irregular image blocks generated in the data set; due to the difference, the model (1) training data (irregular image blocks corresponding to the hyperspectral auxiliary image) ), so the spectral mapping relationship T optimized by the above-mentioned deep network model training method not only reflects the information in each frequency band of the hyperspectral auxiliary image, but also includes the correlation information between frequency bands.

[0015] Step 4: For the RGB image to be spectrally reconstructed Using the spectral mapping relationship T, the differential sequence y of the target image is obtained, which is expressed as follows:

[0016]

[0017] Step 5: Perform reverse difference processing on the differential sequence y of the target image to obtain the target image after RGB image reconstruction.

[0018] Furthermore, in step 2, each RGB image in the data set is divided into Q color irregular image blocks, specifically:

[0019] Perform sparse principal component analysis (SPCA) on each RGB image to reduce its dimension, obtain the principal component of the RGB image, and then use the superpixel segmentation algorithm to calculate the superpixel label of the principal component to obtain Q color irregular image blocks;

[0020] Each hyperspectral auxiliary image in the data set is divided into Q frequency band differential irregular image blocks, specifically: each hyperspectral auxiliary image is differentially limited, the difference between adjacent frequency bands is calculated to obtain a frequency band differential sequence of the auxiliary hyperspectral image, the frequency band differential sequence is upsampled to obtain a frequency band differential sequence with the same spatial resolution as the RGB image; the upsampled frequency band differential sequence is segmented using a superpixel segmentation algorithm to obtain Q frequency band differential irregular image blocks.

[0021] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0022] The spectral reconstruction method of UAV RGB image based on frequency band difference proposed in the present invention limits the relationship between adjacent frequency bands of hyperspectral auxiliary images by introducing the difference idea, obtains the frequency band difference sequence of hyperspectral auxiliary images, can simultaneously describe the information within the spectral band and the correlation information between frequency bands, and combines with deep neural network to obtain the spectral mapping relationship from RGB image to target image, thus overcoming the problem that traditional reconstruction methods ignore the correlation information between frequency bands. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a spectrum reconstruction image of RGB image based on image fusion;

[0024] Figure 2 It is a schematic diagram of the information within each spectral band and the correlation information between spectral bands in the hyperspectral image;

[0025] Figure 3 It is a flow chart of the steps of the present invention;

[0026] Figure 4 This is a schematic diagram of training data generation based on frequency band differences and superpixels. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] This embodiment relates to a spectrum reconstruction method for UAV RGB images based on frequency band difference. By introducing the difference concept to limit the relationship between adjacent frequency bands of the hyperspectral auxiliary image, the frequency band difference sequence of the hyperspectral auxiliary image is obtained, which can simultaneously describe the information within the spectral frequency band and the correlation information between frequency bands, thus overcoming the problem that the traditional reconstruction method ignores the correlation information between frequency bands. The overall flow chart of the present invention is as follows: Figure 3 The main steps involved are as follows:

[0029] Step 1: Collect UAV aerial RGB images and hyperspectral auxiliary images to construct a dataset with RGB and hyperspectral images. Assume that the constructed dataset contains N RGB images and hyperspectral images (i.e., N pairs of images, a total of 2N images), where each RGB image is represented by X n , each hyperspectral image is represented as H n , where 1≤n≤N.

[0030] Step 2: Use frequency band difference and superpixel segmentation to establish a training data generation method, such as Figure 4 shown. Specifically:

[0031] 1) In order to fully consider the characteristics of each region in the image, the superpixel segmentation algorithm is used to perform superpixel segmentation on each RGB image sample to obtain multiple color irregular image blocks as one of the inputs of the deep network spectrum mapping model. If there are N RGB image samples in total, each RGB image can obtain Q color irregular image blocks through superpixel segmentation, then the model input has a total of NQ color irregular image blocks.

[0032] 2) In order to simultaneously describe the intra-band information and the correlation information between frequency bands in the hyperspectral auxiliary image, each auxiliary image in the sample is differentially limited, that is, the difference between adjacent frequency bands is calculated to obtain the frequency band differential sequence of the auxiliary image. The sequence is upsampled to obtain a frequency band differential sequence with the same spatial resolution as the RGB image, and after superpixel segmentation operation, a series of frequency band differential irregular image blocks are obtained as the second input of the deep network model. The color irregular image blocks correspond one to one to the frequency band differential irregular image blocks;

[0033] It should be noted here that by performing sparse principal component analysis (SPCA) on each RGB image to reduce its dimension, its principal component is obtained, and then the superpixel label of the principal component is calculated using the superpixel segmentation algorithm. The label is applied to the RGB image and the frequency band difference sequence after upsampling to generate the corresponding color irregular image block and frequency band difference irregular image block.

[0034] Step 3: Establish a spectral mapping model based on a deep network, and train to obtain the spectral mapping relationship from the RGB image to the frequency band difference sequence. The spectral mapping model can be any deep learning network framework; assume that the color irregular image blocks and frequency band difference irregular image blocks generated above as training data are represented by and Where N represents the number of RGB and hyperspectral auxiliary image samples, Q is the number of irregular blocks corresponding to each sample, and NQ is the number of color irregular image blocks and frequency band difference irregular image blocks generated in the dataset. Based on this, a spectral mapping model based on a deep network is established, and the spectral mapping relationship T from RGB images to frequency band difference sequences is obtained through training, that is:

[0035]

[0036] in, represents the loss function of the spectral mapping model, is the network parameter, β is the penalty parameter, and penalty(T) is the penalty term. Since the difference limits the model (1) training data (irregular image blocks corresponding to the hyperspectral auxiliary image ), so the spectral mapping relationship T optimized by the above-mentioned deep network model training method not only reflects the information in each frequency band of the hyperspectral auxiliary image, but also includes the correlation information between frequency bands.

[0037] Step 4: Use the spectral mapping relationship T obtained by training the model (1) to perform spectral reconstruction on any RGB image collected by the drone to obtain the differential sequence of the target image. Assume that a RGB image to be reconstructed is Then the differential sequence y of the target image (i.e. the reconstructed image) can be obtained by the following formula:

[0038]

[0039] Step 5: Perform reverse difference processing on the difference sequence y to obtain the reconstructed target image.

[0040] Compared with most traditional RGB image spectrum reconstruction methods, the present invention can simultaneously represent the intra-band information and inter-band correlation information of the hyperspectral auxiliary image, and use it for RGB image spectrum reconstruction, which can effectively improve the reconstruction quality.

[0041] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A method for RGB image spectrum reconstruction, characterized in that: The steps include: Step 1, collect RGB images and their hyperspectral auxiliary images, and construct a dataset of RGB images and hyperspectral auxiliary images, including N RGB images and N hyperspectral auxiliary images; Among them, each RGB image is represented as X n , each hyperspectral auxiliary image is represented as H n , 1≤n≤N; Step 2, dividing each RGB image in the data set into Q color irregular image blocks; Segment each hyperspectral auxiliary image in the data set into Q frequency band difference irregular image blocks; Step 3: Establish a spectral mapping model based on a deep network and train the spectral mapping relationship T from the RGB image to the frequency band difference sequence, that is: in, represents the loss function of the spectral mapping model, is the network parameter, β is the penalty parameter, represents the jth color irregular image block, represents the j-th frequency band differential irregular image block, 1≤j≤NQ; Step 4: For the RGB image to be spectrally reconstructed Using the spectral mapping relationship T, the differential sequence y of the target image is obtained, which is expressed as follows: Step 5: Perform reverse difference processing on the differential sequence y of the target image to obtain the target image after RGB image reconstruction.

2. The RGB image spectrum reconstruction method according to claim 1, characterized in that: In step 2, each RGB image in the data set is divided into Q color irregular image blocks, specifically: Perform sparse principal component analysis (SPCA) on each RGB image to reduce its dimension, obtain the principal component of the RGB image, and then use the superpixel segmentation algorithm to calculate the superpixel label of the principal component to obtain Q color irregular image blocks; Each hyperspectral auxiliary image in the data set is divided into Q frequency band differential irregular image blocks, specifically: Each hyperspectral auxiliary image is differentially limited, and the difference between adjacent frequency bands is calculated to obtain the frequency band difference sequence of the auxiliary hyperspectral image. The frequency band difference sequence is upsampled to obtain a frequency band difference sequence with the same spatial resolution as the RGB image. The upsampled frequency band difference sequence is segmented using a superpixel segmentation algorithm to obtain Q frequency band difference irregular image blocks.

Images