A multi-frame fusion mosaic video spectrometer spectral information reconstruction method

Abstract

The present invention provides a method for reconstructing the spectrum information of a mosaic video spectrometer using multi-frame fusion in order to solve the technical problems that the current multi-spectral mosaic image restoration method based on the color image restoration algorithm estimates the pixel information that is not sampled in each spectrum segment by means of spatial interpolation, resulting in inaccurate pixel information at the edge of the target and unsatisfactory estimation of the area where the material information inside the target changes slowly. The present invention reconstructs the spectrum cube of a single frame using multi-frame information to minimize the error caused by interpolation. For a mosaic image sequence of continuous frames, a sliding time window method can be used for sequential reconstruction to obtain multi-spectral video data with complete spectrum information. The spectrum of one frame is reconstructed using multi-frame information, and the reconstruction accuracy is much higher than that of the traditional single-frame reconstruction method, which can achieve the recovery of multi-spectral video information.

Description

Technical Field

[0001] The present invention relates to the field of spectral information reconstruction, and in particular to a method for reconstructing spectral information of a mosaic video spectrometer by multi-frame fusion. Background Art

[0002] Mosaic video spectrometer is a kind of spectrometer that uses pixel-level coating or pixel-level filter on the detector to make multiple pixels in each image unit obtain different spectral information, so as to obtain multi-spectral data in one exposure. The image obtained in this way is composed of several pixel matrix blocks, such as Figure 1 shown.

[0003] Similar to the imaging method of color digital images, different pixels in the pixel matrix block respond to different spectral bands, and each pixel in the image only responds to information of a certain spectral band. The output multispectral image is similar to a digital mosaic image. This imaging method can ensure that the spatial resolution and spectral resolution of the output image remain unchanged, and greatly reduce information redundancy, transmit image data in real time, and can also perform multispectral image video shooting.

[0004] However, since one image unit is used to obtain multiple spectral bands, each single spectral band obtained is actually a sparse sampling of the complete scene. Such data is subject to many limitations in practical applications. When the target imaging size is relatively small and cannot completely cover a mosaic unit, the spectral information of the target cannot be fully obtained. This disadvantage is especially obvious when there are a large number of spectral bands. Therefore, it is necessary to restore the two-dimensional multispectral image data so that the spatial resolution of the single spectral band image of the multispectral image is significantly enhanced, and all spectral information of each pixel is reconstructed to restore the complete multispectral image.

[0005] In the prior art, there is a class of methods that extend the color image restoration algorithm to a multispectral mosaic image restoration method. The general idea is to estimate the pixel information that is not sampled in each spectrum segment by spatial interpolation. However, such methods usually have difficulty in obtaining accurate spectral estimation at the edge of the target, and their interpolation estimation results are usually not ideal for areas where the material information inside the target changes slowly. Summary of the invention

[0006] The purpose of the present invention is to solve the technical problems that the current multispectral mosaic image restoration method based on the color image restoration algorithm estimates the pixel information that has not been sampled in each spectral segment by spatial interpolation, and the pixel information at the edge of the target is inaccurate, and the estimation of the area where the material information inside the target changes slowly is not ideal, and to provide a multi-frame fusion mosaic video spectrometer spectrum information reconstruction method.

[0007] The overall concept of the present invention is that when the mosaic spectral imager shakes, each target point in the scene will be imaged on multiple adjacent pixels. Since the detector imaging frame rate is usually high, reaching more than 100fps, the spectral information of the scene in multiple consecutive frames can be considered constant. In this case, a single-frame atlas cube can be reconstructed using multiple frame information to minimize the error caused by interpolation. For a mosaic image sequence of consecutive frames, a sliding time window method can be used for sequential reconstruction to obtain multi-spectral video data with complete atlas information.

[0008] In order to achieve the above invention purpose and complete the above invention concept, the technical solution adopted by the present invention is:

[0009] A multi-frame fusion mosaic video spectrometer spectrum information reconstruction method is special in that it includes the following steps:

[0010] Step 1, obtaining the original multispectral mosaic image of the mosaic video spectrometer;

[0011] Step 2: randomly select M consecutive frames of images as a time window, take the first frame as a reference, perform demosaicing on each frame of the image in the time window, and obtain a demosaiced image; M ≥ 2 and is an integer;

[0012] Step 3, respectively calculating the optical flow between two adjacent frames in the forward and reverse directions in the time window of the demosaiced image, and obtaining the optical flow maps of the previous and next frames in the forward and reverse directions;

[0013] Step 4: Generate the motion trajectory of the pixel points on each frame image in the time window according to the optical flow information in the obtained optical flow map;

[0014] Step 5: Use the mosaic template to extract multiple single-spectral band images in the first frame of the original mosaic multi-spectral image within the spectrum of the spectrometer to obtain the corresponding atlas data cube, obtain the motion trajectory of the pixel points according to step 4, determine the band position of each pixel point in the first frame in the subsequent M-1 frame image and the spectral information at the corresponding band position, and fill the spectral information of the determined band position into the pixel position corresponding to the single-spectral band image in sequence;

[0015] Step 6: Use interpolation to supplement the missing pixel information in all single-spectrum images;

[0016] Step 7, slide the time window and follow the methods of steps 2 to 6 to solve the spectral data cube of the remaining M-1 frames and realize the sequential reconstruction of the spectral information of the mosaic video spectrometer.

[0017] Furthermore, step 2 is specifically as follows:

[0018] 2.1. Select M consecutive frames of images as a time window, eliminate the grayscale changes in the multispectral mosaic image within the time window, and make the grayscale levels of all pixels in each frame of the image in the same horizontal range;

[0019] 2.2. Further filter out the high-frequency signals in the horizontal and vertical directions of the image to obtain a demosaiced image.

[0020] Furthermore, in step 2.1, a grayscale equalization method is used to eliminate sudden grayscale changes in the multispectral mosaic image.

[0021] Furthermore, in step 2.2, the wavelet transform method is used to filter out high-frequency signals in the horizontal and vertical directions of the image.

[0022] Furthermore, step 4 is specifically as follows:

[0023] 4.1. Project the reverse optical flow between two adjacent frames onto the previous frame image in sequence, and perform interpolation estimation on each pixel point of the previous frame image;

[0024] 4.2. Add the projected optical flow of each pixel on each frame of the image to the forward optical flow and calculate the modulus to obtain the optical flow deviation value. Pixels whose optical flow deviation values ​​are less than the set threshold t are retained. Pixels whose optical flow deviation values ​​are greater than the set threshold t are considered unreliable and the pixel is discarded.

[0025] 4.3. Connect the optical flow of each pixel on each frame image to obtain the motion trajectory of each pixel on each frame image.

[0026] Furthermore, in step 4.2, the threshold value t=0.2 is set.

[0027] Furthermore, in step 2, M=10.

[0028] Further, in step 7, the sliding time window refers to sliding backward one frame.

[0029] Furthermore, in step 7, when solving the atlas data cube of the remaining M-1 frames, the optical flow information and motion trajectory of the newly added frames after the sliding time window are sequentially filled based on the optical flow information and motion trajectory obtained in steps 3 and 4 when solving the atlas data cube of the previous frame.

[0030] Compared with the prior art, the present invention has the following beneficial technical effects:

[0031] The multi-frame fusion mosaic video spectrometer spectrum information reconstruction method provided by the present invention takes into account the natural shaking of the mosaic video spectrometer under handheld, airborne and other shooting conditions, which causes the image scene movement, and utilizes the characteristics of continuous acquisition of mosaic video spectrum information, and combines the spectrum information of time and space information to achieve continuous reconstruction of the spectrum information of the mosaic video spectrometer. The spectrum of one frame is reconstructed by using multiple frames of information, and the reconstruction accuracy is much higher than the traditional single-frame reconstruction method, which can achieve the recovery of multi-spectral video information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is the original multispectral mosaic image;

[0033] Figure 2 It is a flow chart of the method for reconstructing the spectral information of the mosaic video spectrometer by multi-frame fusion of the present invention;

[0034] Figure 3 is an image after grayscale equalization in an embodiment of the present invention;

[0035] Figure 4 is an image after wavelet transformation in an embodiment of the present invention;

[0036] Figure 5 The optical flow images of the front and back frames in the embodiment of the present invention;

[0037] Figure 6 This is a flow chart of information aggregation based on optical flow in an embodiment of the present invention;

[0038] Figure 7 is a single spectrum image restored by the embodiment of the present invention;

[0039] Figure 8 for Figure 7 Spectral curve of the marked point. DETAILED DESCRIPTION

[0040] In order to make the purpose, advantages and features of the present invention clearer, the following is a further detailed description of a multi-frame fusion mosaic video spectrometer spectrum information reconstruction method proposed by the present invention in combination with the accompanying drawings and specific embodiments. It should be understood by those skilled in the art that these implementations are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0041] like Figure 2 As shown, the present invention provides a multi-frame fusion mosaic video spectrometer spectrum information reconstruction method, which specifically includes the following steps:

[0042] Step 1: Obtain the original multispectral mosaic image of the mosaic video spectrometer.

[0043] Step 2: randomly select M consecutive frames of images as a time window, take the first frame as a reference, perform demosaicing on each frame of the image in the time window, and obtain a demosaiced image; M ≥ 2 and is an integer;

[0044] The larger the value of M is, the better the reconstruction effect of the mosaic video atlas information is. However, if the value of M is too large, it will cause an increase in the amount of data and reduce the efficiency. In this embodiment, the value of M is 10.

[0045] 2.1. Select 10 consecutive frames of images as a time window, and use the grayscale equalization method to eliminate the grayscale changes in the multispectral mosaic image within the time window, so that the grayscale levels of all pixels in each frame are in the same horizontal range. The processed image is as follows: Figure 3 As shown;

[0046] 2.2. After grayscale equalization, there are still some bright horizontal and vertical stripes in the original image due to spectral mixing. Then, wavelet transform is used to remove the high-frequency signals in the horizontal and vertical directions of the image. The image after wavelet transform is as follows: Figure 4 shown.

[0047] Step 3: Calculate the optical flow between two adjacent frames in the time window of the demosaiced image in the forward and reverse directions, and obtain the optical flow graphs of the previous and next frames in the forward and reverse directions, such as Figure 5 shown.

[0048] Step 4: Generate the motion trajectory of the pixel points on each frame image in the time window according to the obtained optical flow information;

[0049] 4.1. Project the reverse optical flow between two adjacent frames onto the previous frame image in sequence, and perform interpolation estimation on each pixel point of the previous frame image;

[0050] 4.2. Add the projected optical flow of each pixel on each frame of the image to the forward optical flow and calculate the modulus to obtain the optical flow deviation value. Pixels whose optical flow deviation values ​​are less than the set threshold t (t can be 0.2) are retained. Pixels whose optical flow deviation values ​​are greater than the set threshold t are considered unreliable and are discarded.

[0051] 4.3. Connect the optical flow of each pixel on each frame image to obtain the motion trajectory of each pixel on each frame image; the motion trajectory of the pixel may be interrupted in the middle, so the length of each trajectory may be different.

[0052] Step 5. First, use the mosaic template to extract multiple single-spectral images in the first frame of the original mosaic multi-spectral image within the spectrum of the spectrometer to obtain the corresponding atlas data cube, and then obtain the motion trajectory of the pixel points according to step 4. The band position of each pixel point in the first frame in the subsequent nine frames of images and the spectral information at the corresponding band position can be determined, and the spectral information of the determined band position is filled into the pixel position corresponding to the single-spectral image.

[0053] The spectrometer used in this embodiment has 25 spectral bands, so 25 sparse single-spectral band images are extracted. Many pixel information of these 25 single-spectral band images is missing, so the atlas data cube is seriously sparse. Figure 6 However, according to the motion trajectory, it is impossible for the target object to traverse 25 bands. Therefore, in the single-spectrum image reconstructed by the complementary information of pixels between images, some pixel information must be missing. After filling, some pixel information of the 25 single-spectrum images is still missing.

[0054] Step 6: Use interpolation to supplement the missing pixel information in all single-spectrum images;

[0055] Step 7: According to the method of step 2 to step 6, the time window can be slid to solve the spectrum data cube of the subsequent frames, and the spectrum information of the mosaic video spectrometer can be sequentially reconstructed. The restored image can extract the single spectrum image restored by the method of the present invention, such as Figure 7 shown.

[0056] The obtained optical flow and motion trajectory can continue to be used, and only the optical flow and motion trajectory related to the newly added frame need to be sequentially filled.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A multi-frame fusion mosaic video spectrometer spectrum information reconstruction method, characterized in that: The following steps are involved: Step 1, obtaining the original multispectral mosaic image of the mosaic video spectrometer; Step 2: randomly select M consecutive frames of images as a time window, take the first frame as a reference, perform demosaicing on each frame of the image in the time window, and obtain a demosaiced image; M ≥ 2 and is an integer; Step 3, respectively calculating the optical flow between two adjacent frames in the forward and reverse directions in the time window of the demosaiced image, and obtaining the optical flow maps of the previous and next frames in the forward and reverse directions; Step 4: Generate the motion trajectory of the pixel points on each frame image in the time window according to the optical flow information in the obtained optical flow map, specifically: 4.

1. Project the reverse optical flow between two adjacent frames onto the previous frame image in sequence, and perform interpolation estimation on each pixel point of the previous frame image; 4.

2. Add the projected optical flow of each pixel on each frame of the image to the forward optical flow and calculate the modulus to obtain the optical flow deviation value. Pixels whose optical flow deviation values ​​are less than the set threshold t are retained. Pixels whose optical flow deviation values ​​are greater than the set threshold t are considered unreliable and the pixel is discarded. 4.

3. Connect the optical flow of each pixel on each frame image to obtain the motion trajectory of each pixel on each frame image; Step 5: Use the mosaic template to extract multiple single-spectral band images in the first frame of the original mosaic multi-spectral image within the spectrum of the spectrometer to obtain the corresponding atlas data cube, obtain the motion trajectory of the pixel points according to step 4, determine the band position of each pixel point in the first frame in the subsequent M-1 frame image and the spectral information at the corresponding band position, and fill the spectral information of the determined band position into the pixel position corresponding to the single-spectral band image in sequence; Step 6: Use interpolation to supplement the missing pixel information in all single-spectrum images; Step 7, slide the time window and follow the methods of steps 2 to 6 to solve the spectral data cube of the remaining M-1 frames and realize the sequential reconstruction of the spectral information of the mosaic video spectrometer.

2. The method for reconstructing spectral information of a mosaic video spectrometer by multi-frame fusion according to claim 1, characterized in that: Step 2 is as follows: 2.

1. Select M consecutive frames of images as a time window, eliminate the grayscale changes in the multispectral mosaic image within the time window, and make the grayscale levels of all pixels in each frame of the image in the same horizontal range; 2.

2. Further filter out the high-frequency signals in the horizontal and vertical directions of the image to obtain a demosaiced image.

3. The method for reconstructing spectral information of a multi-frame fusion mosaic video spectrometer according to claim 2, characterized in that: In step 2.1, the grayscale equalization method is used to eliminate the sudden grayscale changes in the multispectral mosaic image.

4. The method for reconstructing spectral information of a mosaic video spectrometer by multi-frame fusion according to claim 3, characterized in that: In step 2.2, the wavelet transform method is used to filter out high-frequency signals in the horizontal and vertical directions of the image.

5. The method for reconstructing spectral information of a multi-frame fusion mosaic video spectrometer according to claim 4, characterized in that: In step 4.2, the threshold value t=0.2 is set.

6. The method for reconstructing spectral information of a mosaic video spectrometer by multi-frame fusion according to claim 5, characterized in that: In step 2, M=10.

7. The method for reconstructing spectral information of a mosaic video spectrometer by multi-frame fusion according to claim 6, characterized in that: In step 7, the sliding time window refers to sliding backward one frame.

8. The method for reconstructing spectral information of a multi-frame fusion mosaic video spectrometer according to claim 7, characterized in that: In step 7, when solving the atlas data cube of the remaining M-1 frames, the optical flow information and motion trajectory of the newly added frames after the sliding time window are sequentially filled based on the optical flow information and motion trajectory obtained in steps 3 and 4 when solving the atlas data cube of the previous frame.

Images