Snapshot spectrum camera real-time spectrum image enhancement method

Abstract

The invention discloses a real-time spectral image enhancement method for a snapshot spectral camera. Belongs to the technical field of multispectral image processing and particularly relates to the technical field of spectral image enhancement processing. The technical problem that the physical property of insufficient transmittance of a multispectral optical filter array has significant limitation on the imaging performance of a snapshot spectrum camera and seriously affects the spectral imaging quality is solved. The transmittance of each channel is calculated through the gray average value of the spectral image, and the image brightness can be quickly and effectively improved. The homomorphic filtering is carried out through the single-parameter Gaussian transfer function, the amplified image noise can be removed, the image contrast can be enhanced, only one parameter exists, the adjustment workload can be reduced, and meanwhile the calculation amount is reduced. And each channel is mapped, so that the spectral information and the spatial details of the spectral image of each channel are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multispectral image processing, and particularly relates to the technical field of spectral image enhancement processing. Background Art

[0002] Spectral imaging technology combines spectral information with spatial information to achieve multi-dimensional feature characterization of target objects, and is widely used in many fields such as agricultural monitoring, biomedical diagnosis, industrial inspection, and remote sensing detection. As a new type of spectral imaging technology, the snapshot spectral camera breaks through the limitation of the traditional spectral camera that needs to scan point by point or line by line, realizes the acquisition of multi-channel spectral data in a single exposure, and has the advantages of small volume, low cost, and strong real-time performance. However, the physical property of insufficient transmittance of the core component, the multispectral filter array, significantly limits the imaging performance of the snapshot spectral camera and seriously affects the spectral imaging quality. Therefore, in order to solve the influence of transmittance on imaging quality, it is imperative to explore a real-time spectral image enhancement method for snapshot spectral cameras. Summary of the Invention

[0003] In order to solve the technical problem that the physical property of insufficient transmittance of the multispectral filter array significantly limits the imaging performance of the snapshot spectral camera and seriously affects the spectral imaging quality, the present invention provides a real-time spectral image enhancement method for a snapshot spectral camera. The method includes the following steps: S1. Calculate the transmittance of each spectral channel; S2. Adjust the brightness of the spectral image of each channel according to the calculated transmittance to obtain the image brightness enhancement result of each channel; S3. Construct a single-channel reference image, and perform normalization processing on the reference image to obtain a normalized reference image; S4. First perform logarithmic processing on the normalized reference image, and then perform Fourier transform; S5. Perform homomorphic filtering on the image according to the Gaussian-type transfer function; S6. Perform inverse Fourier transform and exponential processing on the homomorphic filtering result to obtain a homomorphic filtering enhanced image; S7. Use the homomorphic filtering enhanced image to map the image brightness enhancement result of each channel; S8. Adjust the brightness of the mapping processing result to obtain the spectral image enhancement result.

[0004] The beneficial effects of the method of the present invention are as follows: It mainly solves the problem of poor imaging quality caused by insufficient transmittance of the multispectral filter array in existing snapshot spectral cameras. This method innovatively introduces a transmittance parameter based on the Retinex theory to more accurately describe the imaging model of snapshot spectral cameras. By calculating the transmittance of each channel through the gray mean of the spectral image, the image brightness can be quickly and effectively improved. Through homomorphic filtering with a single-parameter Gaussian transfer function, not only can the amplified image noise be removed, but also the image contrast can be enhanced. At the same time, there is only one parameter, which can reduce the workload of adjustment and calculation amount. By mapping each channel separately, the spectral information and spatial details of the spectral image of each channel are improved. This method not only improves the image brightness and image contrast of the spectral image, but also enhances the spectral information and spatial details of the spectral image, with short processing time and good real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 It is a schematic diagram of the imaging process of a snapshot spectral camera; Figure 2 It is a flowchart of the real-time enhancement method for the snapshot spectral camera in the embodiment of the present invention; Figure 3 It is a schematic diagram of the filter array channels in the embodiment of the present invention; Figure 4 It is a comparison diagram of the spectral image enhancement results of different methods in the embodiment of the present invention; Figure 5 It is a comparison diagram of the results of the spectral image enhancement quantization index (information entropy) of different methods in the embodiment of the present invention; Figure 6 It is a comparison diagram of the results of the spectral image enhancement quantization index (average gradient) of different methods in the embodiment of the present invention; Figure 7 It is a comparison diagram of the results of the spectral image enhancement quantization index (standard deviation) of different methods in the embodiment of the present invention; Figure 8 It is a comparison diagram of the results of the spectral image enhancement quantization index (spatial frequency) of different methods in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0006] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0007] Embodiment 1, The imaging principle of the snapshot multispectral camera is as Figure 1As shown in the figure. The light source emits light. The incident light passes through an observed object and is reflected by the object. The reflected light is received by the snapshot spectral camera, separated by spectral channels through MSFA, and imaged through the photoelectric effect on the monochromatic sensor to obtain spectral image data. The MSFA of the snapshot multispectral camera used in this paper adopts a 3×3 filter unit, and its arrangement is shown in the enlarged area of the yellow border in the figure. A 9-channel multispectral image can be obtained through one acquisition.

[0008] The Retinex theory holds that the human visual system perceives the true color of an object by comparing the brightness and color of different regions in the same scene. Even under changing lighting conditions, humans can maintain a relatively consistent perception of the color of an object. This phenomenon is called color constancy. Considering the influence of the multispectral filter array, the core component of the snapshot spectral camera, on the imaging quality, and the inconsistent transmittance of each spectral channel of the MSFA, based on the Retinex theory, introducing the transmittance parameter, the imaging model of the snapshot spectral camera is: .

[0009] In the formula, represents the acquired image, is the reflected light image of the observed object, which is determined by the nature of the object itself and directly reflects the intrinsic attributes of the image, is the incident light image of the observed object, which directly determines the dynamic range that the pixels in the image can reach, represents the pixel coordinates in the image, , c represents the k color channels of the snapshot spectral camera, is the transmittance parameter, . For fast calculation, the calculation method of the transmittance parameter introduced in this paper is: .

[0010] In the formula, is the gray-scale mean value of all elements of the spectral image in the c channel, D is the maximum gray-scale value of the image data, and its calculation formula is as follows: .

[0011] In the formula, n is the bit depth of the image data. The common bit depth of RGB image data is 8 bits. The bit depth of the image data of the snapshot spectral camera used in this embodiment is 10 bits, which can represent higher brightness levels, and the gray-scale value range is from 0 to 1023. Such an image can display finer brightness changes and belongs to a high dynamic range (HDR) image.

[0012] Such as Figure 2As shown in the figure, this embodiment provides a real-time spectral image enhancement method for a snapshot spectral camera. The method includes the following steps: S1. Calculate the transmittance of each spectral channel; S2. Adjust the brightness of the spectral image of each channel according to the calculated transmittance to obtain the image brightness enhancement result of each channel; S3. Construct a single-channel reference image, and perform normalization processing on the reference image to obtain a normalized reference image; S4. First perform logarithmic processing on the normalized reference image, and then perform Fourier transform; S5. Perform homomorphic filtering on the image according to the Gaussian transfer function; S6. Perform inverse Fourier transform and exponential processing on the homomorphic filtering result to obtain a homomorphic filtering enhanced image; S7. Use the homomorphic filtering enhanced image to map the image brightness enhancement result of each channel; S8. Adjust the brightness of the mapping result to obtain the spectral image enhancement result.

[0013] Embodiment 2 This embodiment further limits Embodiment 1. The technical solution after supplementing the technical details is as follows: S1. By calculating the gray mean value of all pixels of the spectral image of each channel, dividing it by the upper limit value of pixel gray value, and then multiplying by 2 times to obtain the transmittance parameter of each spectral channel.

[0014] The gray value of an image pixel refers to the brightness value represented by each pixel in a grayscale image, which is an integer, and its range depends on the bit depth of the image. For the most common 8-bit grayscale image, the range of pixel gray values is from 0 to 255, and the upper limit is 255; the image data bit depth of the snapshot spectral camera used in this embodiment is 10 bits, which can represent higher brightness levels, and the gray value range is from 0 to 1023, and the corresponding upper limit value of pixel gray value is 1023.

[0015] S2. According to the imaging model of the snapshot spectral camera, divide the pixel gray value of the spectral image of each channel by the transmittance parameter of the corresponding channel. According to the definition of transmittance, if the average gray value of the original spectral image is low, the transmittance parameter of the corresponding channel is also low, and the image brightness enhancement effect is stronger. On the contrary, if the average gray value of the spectral image is high, the transmittance parameter of the corresponding channel is also high, and the image brightness enhancement effect is weaker. In this way, according to the transmittance parameter, appropriate image brightness enhancement can be realized for the spectral image of each channel to obtain the image brightness enhancement result of each channel.

[0016] S3. Since the spectral image data has a high dimension and a large computational complexity, in order to meet the real-time requirement of the algorithm, it is necessary to reduce the dimension of the spectral image. By calculating the gray average value corresponding to multiple channels of each pixel, a single-channel reference image is constructed, and at the same time, the gray values of all pixels of the reference image are normalized to facilitate subsequent calculation and processing.

[0017] S4. The obtained normalized reference image is first subjected to logarithmic transformation and then Fourier transformation to be converted to the frequency domain.

[0018] S5. In the frequency domain, a single-parameter Gaussian transfer function is used to perform homomorphic filtering on the normalized reference image, which can not only remove image noise but also improve the contrast of the image, and obtain the result of the homomorphic filtering processed image.

[0019] S6. The result of the homomorphic filtering process is subjected to inverse Fourier transformation and then exponential transformation to restore the image to the spatial domain, and the homomorphic filtering enhancement result in the spatial domain is obtained, denoted as the homomorphic filtering enhanced image.

[0020] S7. For the image brightness enhancement result of each channel, the homomorphic filtering enhanced result is used for mapping respectively. The mapping method is that the gray value of each pixel of the image brightness enhancement result is multiplied by the gray value at the corresponding position of the homomorphic filtering enhanced result and divided by the gray value at the corresponding position of the normalized reference image, so as to realize the enhancement processing of the spectral image of each channel, obtain the enhancement result of the spectral image of each channel, and improve the spectral information and spatial details of the spectral image.

[0021] S8. Since the data has been normalized, it is also necessary to restore the image gray value. By multiplying the gray values of all pixels of the enhancement result of the spectral image of each channel by the upper limit value of the pixel gray value, the normalized numerical value of the image is restored to the original pixel gray range.

[0022] Embodiment 3 This embodiment provides experimental data to further illustrate the beneficial effects of the method of the present invention.

[0023] The method of the present invention combines the advantages of the snapshot multi-spectral camera imaging model and the improved homomorphic filtering method. The theory of the imaging model is simple and has good robustness, and it is widely applicable to various snapshot multi-spectral cameras. The improved homomorphic filtering method can enhance the quality of the spectral image while removing noise.

[0024] To verify the effectiveness of the proposed method, subjective visual effects and a variety of objective evaluation metrics were used to systematically evaluate the method. All experiments were conducted under the following configuration: the hardware platform was a desktop computer with a 64-bit Windows 10 operating system, the CPU used was a 12th Gen Intel(R) Core(TM) i5-12400F @ 2.50 GHz processor, and the running memory was 16GB; all methods of the present invention were programmed and implemented in the MATLAB R2022b software environment.

[0025] To verify the comprehensive performance of the method proposed in the present invention, the power-law (gamma) transformation (Method1) enhancement method, the logarithmic transformation (Method2) enhancement method, and the method of the present invention (Method3) were used for multi-faceted comparison. At the same time, to verify the effectiveness of the method of the present invention, spectral images collected by a snapshot multi-spectral camera were used for testing. For a 9-channel mosaic snapshot multi-spectral camera corresponding to the 600-800nm band, the schematic diagram of the filter array channels is as Figure 3 shown. The spectral channel image resolution of the collected image data was 341*341*9, and the data format was 10bit RAW. When shooting, the camera was set to a fixed exposure time of 20ms, and the image format was saved as a 16-bit tif format. By referring to the relevant work on multi-spectral image enhancement, five objective evaluation metrics, namely image information entropy, average gradient, standard deviation, spatial frequency, and calculation time, were used to evaluate the performance of different methods.

[0026] The comparison of the spectral image enhancement results of different methods is as Figure 4 shown. Each row of images is the original spectral image of a single channel and the enhancement processing results of different methods. The processing result of the power-law transformation (Method1) enhancement method makes the overall image brighter, but the contrast between light and dark is poor, the light and dark details are not clear, and the details are blurred. Although the processing result of the logarithmic transformation (Method2) enhancement method has less overall brightness improvement of the image, the image contrast is better, and the clarity is better than that of the power-law transformation (Method1) enhancement method. However, the processing effect on the dark area is poor. The processing result of the method of the present invention (Method3) has a significant brightness improvement, and good visual effects can be achieved for both bright details such as vehicle reflections and artificially whitewashed areas on tree trunks, and dark details such as the surrounding areas of the image.

[0027] The comparison of the quantization index results of the 9-channel spectral image enhancement of different methods is as Figures 5 - 8 shown, Figures 5 - 8The quantified indicators represented respectively are information entropy, average gradient, standard deviation, and spatial frequency. The quality of the original spectral image is the worst, so the results obtained in the four indicators are all the lowest. In terms of the image information entropy indicator, the result obtained by the power-law transformation (Method1) enhancement method is only slightly higher than that of the original image and is the worst among the three methods. However, the logarithmic transformation (Method2) enhancement method achieves a sub-optimal result, showing a significant improvement compared to the result of the original image. The method of the present invention (Method3) still achieves the optimal result with a remarkable improvement effect. In terms of the three indicators of average gradient, image standard deviation, and spatial frequency, the results obtained by the three methods are similar in trend. The result obtained by the power-law transformation (Method1) enhancement method is the worst, and the result obtained by the logarithmic transformation (Method2) enhancement method is the second. The improvement effects of the two methods compared to the original image are limited. The result obtained by the method of the present invention (Method3) is the best, which is consistent with the subjective effect and is about 7 times higher than the result of the original image. The processing result of the method of the present invention not only has good image quality and rich details, and each index is significantly better than the original image and other methods, but also has good robustness. It will not be affected by different imaging bands and can well adapt to snapshot spectral cameras with different imaging bands.

[0028] In order not to affect the imaging frame rate of the snapshot spectral camera, a high requirement is put forward for the real-time performance of the method. After the experiment of spectral image enhancement (wavelength in 600 - 800nm), the processing time consumed by different methods is shown in Table 1.

[0029] Table 1:

[0030] As can be seen from Table 1, the calculation time of the power-law transformation (Method1) enhancement method is the longest, the calculation time of the method of the present invention (Method3) is the second, and the calculation time of the logarithmic transformation (Method2) enhancement method is the shortest. Although the processing time of the method of the present invention (Method3) is not the shortest, the average value of the processing time is only 44% of that of the power-law transformation (Method1) enhancement method.

[0031] The processing time of the method of the present invention (Method3) is only 14.2ms. Converted to the frame rate, it can process the spectral images of snapshot spectral cameras with an imaging frame rate not greater than 70FPS in real time.

[0032] In summary, compared with the power-law transformation enhancement method and the logarithmic transformation enhancement method, the method of the present invention achieves the best results on the 9-channel snapshot spectral camera in the 600-800 nm band, whether in terms of subjective vision or objective index quantitative results. The spectral image enhancement results have richer information, higher brightness, clearer details, and better contrast. At the same time, the algorithm of the present invention has good robustness and short processing time, and can meet the requirements of real-time spectral image enhancement of snapshot multispectral cameras.

Claims

1. A real-time spectral image enhancement method for a snapshot spectral camera, characterized in that, The method includes the following steps: S1. Calculate the transmittance of each spectral channel; S2. Adjust the brightness of the spectral image of each channel according to the calculated transmittance to obtain the image brightness enhancement result of each channel; S3. Construct a single-channel reference image and perform normalization processing on the reference image to obtain a normalized reference image; S4. First perform logarithmic processing on the normalized reference image and then perform Fourier transform; S5. Perform homomorphic filtering on the image according to the Gaussian transfer function; S6. Perform inverse Fourier transform and exponential processing on the homomorphic filtering result to obtain a homomorphic filtering enhanced image; S7. Use the homomorphic filtering enhanced image to map the image brightness enhancement result of each channel; S8. Adjust the brightness of the mapping processing result to obtain the spectral image enhancement result.

2. The real-time spectral image enhancement method for the snapshot spectral camera according to claim 1, characterized in that, Step S1 is specifically: By calculating the gray mean value of all pixels of the spectral image of each channel, dividing it by the upper limit value of pixel gray value, and then multiplying by 2 to obtain the transmittance parameter of each spectral channel.

3. The real-time spectral image enhancement method of the snapshot spectral camera according to claim 2, characterized in that The adjustment of the brightness of the spectral image of each channel is specifically: Divide the pixel gray value of the spectral image of each channel by the transmittance parameter of the corresponding channel.

4. The real-time spectral image enhancement method for the snapshot spectral camera according to claim 3, characterized in that Step S3 is specifically: By calculating the gray average value corresponding to multiple channels of each pixel, construct a single-channel reference image, and at the same time perform normalization processing on all pixel gray values of the reference image.

5. The real-time spectral image enhancement method of the snapshot spectral camera according to claim 4, wherein Using the homomorphic filtering enhanced image to map the image brightness enhancement result of each channel is specifically: Multiply the gray value of each pixel in the image brightness enhancement result by the gray value at the corresponding position of the homomorphic filtering enhanced image and divide by the gray value at the corresponding position of the normalized reference image.

6. The real-time spectral image enhancement method for a snapshot spectral camera according to claim 5, wherein Step S8 is specifically: Multiply all pixel gray values of the mapping processing result by the upper limit value of pixel gray value to restore the image normalization value to the original pixel gray range.

7. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-6.

8. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instruction is executed by the processor, it implements the steps of the method according to any one of claims 1-6.

Images