A real-time spectral image enhancement method for snapshot spectral cameras

Abstract

A real-time spectral image enhancement method for a snapshot spectral camera. The method belongs to the field of multispectral image processing technology, and specifically relates to the field of spectral image enhancement processing technology. The method solves the technical problem that the physical characteristic of insufficient transmittance of a multispectral filter array significantly limits the imaging performance of a snapshot spectral camera, seriously affecting the quality of spectral imaging. By calculating the transmittance of each channel through the grayscale mean of the spectral image, the image brightness can be quickly and effectively improved. By performing homomorphic filtering through a single-parameter Gaussian transfer function, not only can the amplified image noise be removed, but also the image contrast can be enhanced. At the same time, there is only one parameter, which can reduce the workload of adjustment and the amount of calculation. By mapping each channel separately, the spectral information and spatial details of the spectral image of each channel are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multispectral image processing, and particularly relates to the technical field of spectral image enhancement processing. Background Art

[0002] Spectral imaging technology combines spectral information with spatial information to achieve multi-dimensional feature characterization of target objects. It is widely used in many fields, including agricultural monitoring, biomedical diagnosis, industrial testing, and remote sensing. As a new type of spectral imaging technology, snapshot spectral cameras break through the limitations of traditional spectral cameras that require point-by-point or line-by-line scanning, and can acquire multi-channel spectral data in a single exposure. They have the advantages of small size, low cost, and strong real-time performance. However, the physical property of insufficient transmittance of its core component, the multispectral filter array, significantly limits the imaging performance of snapshot spectral cameras, seriously affecting the quality of spectral imaging. Therefore, in order to address the impact of transmittance on imaging quality, it is imperative to explore real-time spectral image enhancement methods for snapshot spectral cameras. Summary of the Invention

[0003] In order to solve the technical problem that the physical property of insufficient transmittance of a multispectral filter array significantly limits the imaging performance of a snapshot spectral camera and seriously affects the quality of spectral imaging, the present invention provides a real-time spectral image enhancement method for a snapshot spectral camera, the method comprising the following steps:

[0004] S1. Calculate the transmittance of each spectral channel;

[0005] S2. Adjust the brightness of each channel spectral image according to the calculated transmittance to obtain the image brightness enhancement result of each channel;

[0006] S3, constructing a single-channel reference image, and normalizing the reference image to obtain a normalized reference image;

[0007] S4, performing logarithmic processing on the normalized reference image and then performing Fourier transform;

[0008] S5, performing homomorphic filtering on the image according to a Gaussian transfer function;

[0009] S6. Perform inverse Fourier transform and exponential processing on the homomorphic filtering result to obtain a homomorphic filtering enhanced image;

[0010] S7, mapping the image brightness enhancement result of each channel using homomorphic filtering to enhance the image;

[0011] S8. Perform brightness adjustment on the mapping processing result to obtain a spectral image enhancement result.

[0012] The beneficial effects of the method described in the present invention are: it mainly solves the problem of poor imaging quality caused by insufficient transmittance of the multi-spectral filter array of the existing snapshot spectral camera. The method innovatively introduces the transmittance parameter based on the Retinex theory to more accurately describe the snapshot spectral camera imaging model. By calculating the transmittance of each channel through the grayscale mean of the spectral image, the image brightness can be quickly and effectively improved. By performing homomorphic filtering through a single-parameter Gaussian transfer function, not only can the amplified image noise be removed, but also the image contrast can be enhanced. At the same time, there is only one parameter, which can reduce the adjustment workload and the amount of calculation. By mapping each channel separately, the spectral information and spatial details of the spectral image of each channel are improved. This method not only improves the image brightness and image contrast of the spectral image, but also improves the spectral information and spatial details of the spectral image, with short processing time and good real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of the imaging process of a snapshot spectral camera;

[0014] Figure 2 This is a processing flow chart of a real-time enhancement method for a snapshot spectral camera according to an embodiment of the present invention;

[0015] Figure 3 Schematic diagram of the filter array channel in an embodiment of the present invention;

[0016] Figure 4 A comparison chart of spectral image enhancement results using different methods in an embodiment of the present invention;

[0017] Figure 5 This is a comparison chart of the quantitative index (information entropy) of spectral image enhancement using different methods in an embodiment of the present invention;

[0018] Figure 6 This is a comparison chart of the quantitative index (average gradient) of spectral image enhancement results of different methods in the embodiment of the present invention;

[0019] Figure 7 This is a comparison chart of the quantitative index (standard deviation) of spectral image enhancement using different methods in the embodiments of the present invention;

[0020] Figure 8 1 is a comparison chart of the quantitative index (spatial frequency) of spectral image enhancement using different methods in the embodiments of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] Example 1

[0023] The imaging principle of snapshot multispectral camera is as follows Figure 1 As shown in the figure, a light source emits light, which is reflected by an observation object. The reflected light is then received by a snapshot spectral camera, where it is separated into spectral channels by the MSFA. The image is then formed on a monochromatic sensor via the photoelectric effect, producing spectral image data. The snapshot multispectral camera used in this article uses a 3×3 filter element MSFA, whose arrangement is shown in the enlarged area outlined in yellow in the figure. This allows a 9-channel multispectral image to be acquired in a single acquisition.

[0024] Retinex theory posits that the human visual system perceives the true color of objects by comparing the brightness and color of different areas within the same scene. Even under changing lighting conditions, humans maintain a relatively consistent perception of object color. This phenomenon is known as color constancy. Considering the impact of the multispectral filter array (MSFA), the core component of snapshot spectral cameras, on imaging quality, and the inconsistent transmittance of each spectral channel in the MSFA, based on Retinex theory, the transmittance parameter is introduced, resulting in the snapshot spectral camera imaging model: .

[0025] Where, represents the acquired image, The reflected light image of the observed object is determined by the properties of the object itself and directly reflects the intrinsic properties of the image. It is the incident light image of the observed object, which directly determines the dynamic range that the pixels in the image can achieve. represents the pixel coordinates in the image, , c Represents snapshot spectral camera k color channels, is the transmittance parameter, In order to calculate quickly, the transmittance parameter introduced in this paper is The calculation method is: .

[0026] Where, is the grayscale mean of all elements of the c-channel spectral image, D is the maximum grayscale value of the image data, and its calculation formula is as follows: .

[0027] Where, n The image data bit depth is 8 bits, while the typical RGB image data bit depth is 8 bits. The snapshot spectral camera used in this embodiment has an image data bit depth of 10 bits, which can represent higher brightness levels, with a grayscale value range of 0 to 1023. Such images can display more subtle brightness changes and are considered high dynamic range (HDR) images.

[0028] like Figure 2 As shown, this embodiment provides a real-time spectral image enhancement method for a snapshot spectral camera, the method comprising the following steps:

[0029] S1. Calculate the transmittance of each spectral channel;

[0030] S2. Adjust the brightness of each channel spectral image according to the calculated transmittance to obtain the image brightness enhancement result of each channel;

[0031] S3, constructing a single-channel reference image, and normalizing the reference image to obtain a normalized reference image;

[0032] S4, performing logarithmic processing on the normalized reference image and then performing Fourier transform;

[0033] S5, performing homomorphic filtering on the image according to a Gaussian transfer function;

[0034] S6. Perform inverse Fourier transform and exponential processing on the homomorphic filtering result to obtain a homomorphic filtering enhanced image;

[0035] S7, mapping the image brightness enhancement result of each channel using homomorphic filtering to enhance the image;

[0036] S8. Perform brightness adjustment on the mapping processing result to obtain a spectral image enhancement result.

[0037] Example 2

[0038] This embodiment further limits the embodiment 1, and the technical solution after supplementing the technical details is as follows:

[0039] S1. Calculate the grayscale mean of all pixels in the spectral image of each channel, divide it by the upper limit of the pixel grayscale, and then multiply it by 2 to obtain the transmittance parameter of each spectral channel.

[0040] Image pixel grayscale refers to the brightness value represented by each pixel in a grayscale image. It is an integer whose range depends on the image's bit depth. For the most common 8-bit grayscale images, pixel grayscale values ​​range from 0 to 255, with an upper limit of 255. The snapshot spectral camera image data used in this example has a 10-bit bit depth, which can represent higher brightness levels. The grayscale value range is 0 to 1023, corresponding to a pixel grayscale upper limit of 1023.

[0041] S2. Based on the snapshot spectral camera imaging model, the pixel grayscale value of each channel's spectral image is divided by the corresponding channel's transmittance parameter. According to the definition of transmittance, if the average grayscale value of the original spectral image is low, the transmittance parameter of the corresponding channel is also low, and the image brightness enhancement effect is stronger. Conversely, if the average grayscale value of the spectral image is high, the transmittance parameter of the corresponding channel is also high, but the image brightness enhancement effect is weaker. In this way, based on the transmittance parameter, appropriate image brightness enhancement can be achieved for each channel's spectral image, resulting in an image brightness enhancement result for each channel.

[0042] S3. Due to the high dimensionality and computational complexity of spectral image data, dimensionality reduction is necessary to meet the real-time requirements of the algorithm. A single-channel reference image is constructed by calculating the grayscale average of multiple channels for each pixel. All pixel grayscale values ​​in the reference image are normalized to facilitate subsequent computational processing.

[0043] S4. Perform logarithmic transformation on the obtained normalized reference image, and then perform Fourier transformation on the image to convert it into the frequency domain.

[0044] S5. In the frequency domain, a single-parameter Gaussian transfer function is used to perform homomorphic filtering on the normalized reference image. This not only removes image noise but also improves the image contrast, thereby obtaining a homomorphic filtering image processing result.

[0045] S6. Perform inverse Fourier transform on the homomorphic filtering result, and then perform exponential transform to restore the image to the spatial domain, thereby obtaining a homomorphic filtering enhancement result in the spatial domain, which is recorded as a homomorphic filtering enhanced image.

[0046] S7. The image brightness enhancement results of each channel are mapped separately using the homomorphic filtering enhancement results. The mapping method is to multiply the grayscale value of each pixel of the image brightness enhancement result by the grayscale value of the corresponding position of the homomorphic filtering enhancement result and divide it by the grayscale value of the corresponding position of the normalized reference image, so as to realize the spectral image enhancement processing of each channel, obtain the spectral image enhancement result of each channel, and improve the spectral information and spatial details of the spectral image.

[0047] S8. Since the data has been normalized, the image grayscale value needs to be restored. By multiplying all pixel grayscale values ​​of the spectral image enhancement results of each channel by the pixel grayscale upper limit value, the image normalization value is restored to the original pixel grayscale range.

[0048] Example 3

[0049] This example provides experimental data to further illustrate the beneficial effects of the method of the present invention.

[0050] The method described in the present invention combines the advantages of a snapshot multispectral camera imaging model and an improved homomorphic filtering method. The imaging model is theoretically simple and robust, and is widely applicable to various snapshot multispectral cameras. The improved homomorphic filtering method can enhance the quality of spectral images while removing noise.

[0051] To validate the effectiveness of the proposed method, we systematically evaluated it using subjective visual effects and multiple objective evaluation metrics. All experiments were conducted on a desktop computer running a 64-bit Windows 10 operating system, a 12th Gen Intel(R) Core(TM) i5-12400F @ 2.50 GHz processor, and 16GB of RAM. All methods were implemented using MATLAB R2022b.

[0052] In order to verify the comprehensive performance of the method proposed in this invention, the power law (gamma) transformation (Method 1) enhancement method, the logarithmic transformation (Method 2) enhancement method and the method of this invention (Method 3) were used for multiple comparisons. At the same time, in order to verify the effectiveness of the method of this invention, the spectral images collected by the snapshot multispectral camera were tested. The 9-channel mosaic snapshot multispectral camera corresponding to the 600-800nm ​​band has a filter array channel schematic diagram as shown below. Figure 3 As shown in the figure, the image data captured has a spectral channel resolution of 341*341*9 and is in 10-bit RAW format. The camera was set to a fixed exposure time of 20ms, and the images were saved in 16-bit tif format. By referring to related work on multispectral image enhancement, the performance of different methods was evaluated using five objective evaluation metrics: image information entropy, mean gradient, standard deviation, spatial frequency, and computation time.

[0053] Comparison of spectral image enhancement results using different methods Figure 4As shown, each row of images shows the original spectral image of a single channel and the enhancement results using different methods. The power-law transformation (Method 1) enhancement method results in a brighter overall image, but the contrast between light and dark is poor, and the light and dark details are unclear and blurred. The logarithmic transformation (Method 2) enhancement method, while improving the overall image brightness slightly, has better contrast and clarity than the power-law transformation (Method 1) enhancement method. However, the effect on dark areas is poor. The method of the present invention (Method 3) significantly improves brightness, and bright details such as vehicle reflections and artificially whitened tree trunks, as well as dark details such as the surrounding areas of the image, all have a better visual effect.

[0054] Comparison of quantitative index results of 9-channel spectral image enhancement using different methods Figure 5-8 As shown,

[0055] Figure 5-8 The quantitative indicators represented are information entropy, mean gradient, standard deviation, and spatial frequency, respectively. The original spectral image had the worst quality, resulting in the lowest results for all four indicators. Regarding image information entropy, the power-law transform (Method 1) enhancement method achieved results only slightly higher than the original image, the worst of the three methods. However, the logarithmic transform (Method 2) enhancement method achieved suboptimal results, with a significant improvement compared to the original image. The method of the present invention (Method 3) still achieved the best results, with a significant improvement. Regarding mean gradient, image standard deviation, and spatial frequency, the results of the three methods showed similar trends. The power-law transform (Method 1) enhancement method achieved the worst results, followed by the logarithmic transform (Method 2). Both methods achieved limited improvement compared to the original image. The method of the present invention (Method 3) achieved the best results, consistent with the subjective effect, with an improvement of approximately 7 times compared to the original image. The processing results of the method of the present invention not only have good image quality and rich details, and various indicators are significantly better than the original images and other methods, but also have good robustness. The effects achieved will not be different due to different imaging bands, and can be well adapted to snapshot spectral cameras with different imaging bands.

[0056] In order not to affect the imaging frame rate of the snapshot spectral camera, high real-time performance is required. After experiments on spectral image enhancement (wavelength between 600-800 nm), the processing time consumed by different methods is shown in Table 1.

[0057] Table 1:

[0058]

[0059] As shown in Table 1, the computation time for the power-law transformation (Method 1) enhancement method is the longest, followed by the method of the present invention (Method 3), and the computation time for the logarithmic transformation (Method 2) enhancement method is the shortest. Although the processing time for the method of the present invention (Method 3) is not the shortest, its average processing time is only 44% of that for the power-law transformation (Method 1) enhancement method.

[0060] The processing time of the method (Method 3) of the present invention is only 14.2 ms, which can be converted into a frame rate and can process the spectral image of a snapshot spectral camera with an imaging frame rate of no more than 70 FPS in real time.

[0061] In summary, compared with power-law and logarithmic transformation enhancement methods, the proposed method achieves the best results on a 9-channel snapshot spectral camera in the 600-800 nm band, both in terms of subjective visual perception and objective quantitative results. The resulting spectral image enhancement features richer information, higher brightness, clearer details, and better contrast. Furthermore, the proposed algorithm exhibits excellent robustness and short processing time, meeting the requirements for real-time spectral image enhancement for snapshot multispectral cameras.

Claims

1. A real-time spectral image enhancement method for a snapshot spectral camera, characterized in that: The method comprises the following steps: S1. Calculate the transmittance of each spectral channel; The transmittance parameter of each spectral channel is obtained by calculating the grayscale mean of all pixels in the spectral image of each channel, dividing it by the upper limit of the pixel grayscale, and then multiplying it by 2; S2. Adjust the brightness of the spectral image of each channel according to the calculated transmittance, and divide the pixel grayscale value of the spectral image of each channel by the transmittance parameter of the corresponding channel to obtain the image brightness enhancement result of each channel; S3, constructing a single-channel reference image, and normalizing the reference image to obtain a normalized reference image; S4, performing logarithmic processing on the normalized reference image and then performing Fourier transform; S5, performing homomorphic filtering on the image according to a Gaussian transfer function; S6. Perform inverse Fourier transform and exponential processing on the homomorphic filtering result to obtain a homomorphic filtering enhanced image; S7, mapping the image brightness enhancement result of each channel using homomorphic filtering to enhance the image; S8. Perform brightness adjustment on the mapping processing result to obtain a spectral image enhancement result.

2. The method for enhancing real-time spectral images of a snapshot spectral camera according to claim 1, wherein: Step S3 specifically comprises: constructing a single-channel reference image by calculating the grayscale average value corresponding to multiple channels of each pixel, and normalizing the grayscale values ​​of all pixels in the reference image.

3. The real-time spectral image enhancement method for a snapshot spectral camera according to claim 2, characterized in that: The image brightness enhancement result of each channel is mapped using the homomorphic filtering enhanced image. Specifically, the grayscale value of each pixel in the image brightness enhancement result is multiplied by the grayscale value of the corresponding position in the homomorphic filtering enhanced image and divided by the grayscale value of the corresponding position in the normalized reference image.

4. The method for enhancing real-time spectral images of a snapshot spectral camera according to claim 3, wherein: Step S8 specifically includes: multiplying all pixel grayscale values ​​of the mapping processing result by the pixel grayscale upper limit value, and restoring the image normalization value to the original pixel grayscale range.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images