Texture information guided enhancement method and system for filtered array spectral images

Abstract

The present invention proposes a texture information guided enhancement method and system for filtered array spectral images, which belongs to the field of computer vision and computational photography technology. The present invention designs a method for extracting high-frequency texture information from an image, and designs a texture guided enhancement network and a texture guided loss function based on this method. During the training process, the texture guided enhancement network extracts texture information after averaging the input image, and embeds the extracted information into the network as a guide; the texture guided loss function extracts texture information on the target image, and uses this information as a guide to redistribute the weight of each pixel point, so that the network can pay more attention to high-frequency texture information, thereby obtaining a better enhancement effect. At the same time, a fully automated filtered array spectral image enhancement system is proposed, which can effectively solve the problem of poor imaging quality of filtered array spectral cameras under low light conditions.

Description

Technical Field

[0001] The present invention relates to a texture information guided enhancement method and system for filtering array spectral images, belonging to the technical field of computer vision and computational photography. Background Art

[0002] Filter array spectral cameras are increasingly being used in the field of multispectral imaging. Extending from traditional color filter arrays, multispectral filter arrays enable filter array spectral cameras to capture multispectral images in a snapshot format. However, compared to color filter arrays, each pixel in a multispectral filter array can pass a narrower spectral band and transmit less light. Therefore, in poor lighting conditions, the spectral images captured by filter array spectral cameras can contain significant noise, significantly affecting image quality.

[0003] To enhance the image quality of filter array spectral cameras in low-light environments and remove image noise, several mature solutions have been developed. These enhancement and denoising methods can be categorized as model-based and learning-based. Model-based methods primarily rely on the physical properties of spectral images, manually designing prior information. Incorporating this physical prior information, they transform the image enhancement and denoising problem into an optimization problem, thereby removing noise from the spectral image. However, a challenge with these methods is that the manually designed prior information is not accurate or comprehensive enough. In contrast, learning-based methods can effectively avoid the complications associated with manually designed priors, as well as their inaccuracy and incompleteness. Leveraging the powerful learning capabilities of deep neural networks, learning-based methods can automatically learn the underlying features in the data and have achieved considerable success in spectral image enhancement and denoising.

[0004] However, existing learning-based methods overlook a crucial issue: during image enhancement, restoring high-frequency texture details is more difficult than restoring low-frequency components. This results in images processed using learning-based methods losing some high-frequency texture information, resulting in oversmoothing. Therefore, there is an urgent need for an image enhancement method that can effectively restore high-frequency texture information and apply it to real-world scenarios. Summary of the Invention

[0005] The purpose of the present invention is to solve the technical problem of poor imaging quality of filter array spectral cameras under insufficient lighting conditions, and creatively propose a texture information guided enhancement method and system for filter array spectral images.

[0006] The innovation of this invention lies in extracting texture maps as a guide for high-frequency information, designing a texture-guided enhancement network and a texture-guided loss function for filtered array spectral images. This effectively addresses the issues of high image noise and low signal-to-noise ratio (SNR) in low-light environments often faced by filtered array spectral cameras. Compared to existing methods, this method effectively preserves high-frequency texture information in the original image.

[0007] In order to achieve the above objectives, the present invention adopts the following technical solutions.

[0008] A texture information guided enhancement method for filtered array spectral images includes a training phase and an application phase:

[0009] Step 1: During the training phase, the model parameters are iteratively optimized using a paired low-light-normal-light filtered array spectral image dataset.

[0010] Specifically, step 1 includes the following steps:

[0011] Step 1.1: Randomly select N pairs of paired data from the paired low-light-normal-light filter array spectral images and preprocess the data, including data brightening, augmentation, normalization, and formatting.

[0012] Step 1.2: Input the preprocessed filtered array spectral image into the texture-guided enhancement network to obtain an enhanced filtered array spectral image;

[0013] Step 1.3: Feed the enhanced filtered array spectral image and the corresponding normal light filtered array spectral image into the texture guided loss function to calculate the loss function value;

[0014] Preferably, the texture-guided enhancement network and the texture-guided loss function use a texture map extracted from the image as a guide.

[0015] Furthermore, the present invention proposes a method for extracting a texture map from an image.

[0016] First, perform Gaussian filtering on the original image to obtain a blurred image;

[0017] Then, the blurred image is subtracted from the original image to obtain a difference map, and the difference map is Gaussian filtered again to obtain a blurred difference map;

[0018] Finally, the blurred difference map is normalized to obtain the texture map.

[0019] Let the input image be I and the texture map be M, then M is expressed as:

[0020] M=h(g2(I-g1(I)))

[0021] Among them, g1 and g2 are Gaussian filters, and h is the normalization function, which is defined as:

[0022]

[0023] Where X represents the image to be normalized.

[0024] Step 1.4: According to the backpropagation algorithm, the gradient values ​​of each parameter are calculated and backpropagated through the loss function value, and the calculated gradient values ​​are used to update each parameter to complete one training iteration;

[0025] Step 1.5: Repeat steps 1.1 to 1.4 until the number of iterations reaches the preset upper limit, and save the enhanced network structure and parameter values.

[0026] Step 2: Use stage, load the model structure and parameters in step 1, input the low-light filter array spectral image into the model, and obtain the enhanced image.

[0027] Preferably, the texture-guided enhancement network described in this method includes a backbone network and a texture-guided branch, wherein:

[0028] The backbone network adopts an encoder-decoder structure, consisting of 9 residual blocks, of which 4 residual blocks serve as encoders and 4 residual blocks serve as decoders. Each encoder is followed by a downsampling layer to halve the length and width of the feature map and increase the number of channels. Each decoder is preceded by an upsampling layer to double the length and width of the feature map. Skip connections are used to pass features extracted by the shallow encoder to the decoder.

[0029] The texture guidance branch averages the multi-channel spectral image in the channel dimension, extracts the texture map on the averaged single-channel image, and downsamples the texture map to five different scales. It is connected to the feature map in the backbone network as a guide for the backbone network information.

[0030] Preferably, the texture-guided loss function described in this method includes an image restoration part and a texture-guided part. The image restoration part adopts the common L1 loss function. Specifically, let the image obtained after network enhancement be O and the target image be T, then the image restoration part is:

[0031] L1=‖OT‖1,

[0032] Among them, ‖·‖1 is the L1 norm.

[0033] The texture guidance part extracts the texture map of the target image band by band, uses the obtained texture map as a guide, multiplies the difference between the restored image and the target image element by element, and calculates its L1 norm. If the image obtained after network enhancement is O, the target image is T, and the extracted texture map is M, then the texture guidance part is:

[0034] L TG =‖M×(OT)‖1

[0035] The final texture-guided loss function is expressed as:

[0036] L=L1+α×L TG

[0037] Among them, α is used to balance the image restoration part and the texture guidance part.

[0038] On the other hand, the present invention further proposes a texture information guided enhancement system for filter array spectral images, comprising a filter array spectral image preprocessing subsystem, a texture guided enhancement network subsystem, a texture guided loss function subsystem, and a filter array spectral imaging subsystem, as well as paired low-light-normal-light filter array spectral image datasets;

[0039] A paired low-light-normal-light filtered array spectral image dataset for supervised training of the texture-guided enhancement network subsystem and optimization of network parameters;

[0040] The filter array spectral image preprocessing subsystem is used to perform data brightening, augmentation, normalization, and formatting on the filter array spectral image during the training phase;

[0041] The texture-guided enhancement network subsystem is used to perform supervised training using paired low-light and normal-light filter array spectral image datasets during the training phase to obtain the network subsystem's enhancement of the filter array spectral images; during the use phase, it is used to enhance the low-light and normal-light images acquired by the filter array spectral imaging subsystem;

[0042] The texture-guided loss function subsystem is used to supervise the training of the texture-guided enhancement network and optimize network parameters. This can significantly improve the performance of the network subsystem.

[0043] The filter array spectral imaging subsystem is used to obtain unprocessed low-light filter array spectral images.

[0044] The output of the filter array spectral image preprocessing subsystem is connected to the input of the texture-guided enhancement network subsystem, and the output of the texture-guided enhancement network subsystem is connected to the input of the texture-guided loss function subsystem. The output of the filter array spectral camera imaging subsystem is connected to the input of the filter array spectral image preprocessing subsystem.

[0045] The processing process of the system is as follows:

[0046] During the system training phase:

[0047] Step 1: Randomly select N pairs of data from the paired low-light-normal light filter array spectral image dataset and input them into the filter array spectral image preprocessing subsystem for data brightening, augmentation, normalization and formatting. The processed paired low-light-normal light spectral images are stored in the memory.

[0048] Step 2: Input the low-light spectrum image into the texture-guided enhancement network subsystem for processing to obtain an enhanced filtered array spectrum image;

[0049] Step 3: The enhanced filtered array spectral image and the stored normal light spectral image are fed into the texture guided loss function subsystem to calculate the loss function value;

[0050] Step 4: Through the gradient backpropagation algorithm, the value of the loss function is propagated back to each parameter in the enhanced network subsystem, and the parameters are updated according to the gradient values ​​propagated back to each parameter, completing a training iteration;

[0051] Step 5: Repeat steps 1 to 4 until the total number of iterations reaches the preset upper limit, stop training, and save the network structure and network parameters in the texture-guided enhancement network subsystem.

[0052] During the application deployment phase:

[0053] Step 6: Read the trained network structure and parameter files into the texture-guided enhancement network subsystem;

[0054] Step 7: Use the filter array spectral camera imaging subsystem to acquire a spectral image and input it into the filter array spectral image preprocessing subsystem for data normalization and formatting;

[0055] Step 8: The spectral image after normalization and formatting by the filter array spectral camera preprocessing subsystem will be input into the texture guided enhancement network subsystem for accurate processing to obtain the enhanced filter array spectral image.

[0056] Beneficial effects

[0057] Compared with the prior art, the present invention has the following advantages:

[0058] The present invention designs a texture-guided enhancement method and system for filtered array spectral images, which can effectively enhance filtered array spectral images and achieve excellent enhancement results. The method extracts a texture map from the filtered array spectral image and introduces it into the enhancement method as a guide, enabling the designed method to better restore high-frequency texture information in the filtered array spectral image, achieving enhancement for the filtered array spectral image. This effectively addresses the pain points of high noise and poor imaging quality in filtered array spectral images in low-light environments, significantly broadening the application scenarios of filtered array spectral cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of the process of this method.

[0060] Figure 2 Schematic diagram of a texture information guided enhancement method for filtered array spectral images provided by an embodiment of the present invention.

[0061] Figure 3 Schematic diagram of the structure of a texture information guided enhancement system for filtering array spectral images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to better illustrate the purpose and advantages of the present invention, the inventive method is further described below with reference to the accompanying drawings and examples.

[0063] Example

[0064] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar components or components having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, but are not to be construed as limiting the present invention.

[0065] like Figure 1 As shown, the following steps are included:

[0066] Step 1: Randomly select data from the dataset and preprocess it;

[0067] Step 2: Feed the data into the texture-guided enhancement network for enhancement;

[0068] Step 3: Feed the enhanced result and target data into the texture-guided loss function to calculate the loss function value;

[0069] Step 4: Update the parameters of the enhanced network according to the back propagation algorithm;

[0070] Step 5: Repeat the above steps until the number of iterations reaches the upper limit, and save the network structure and parameter values;

[0071] Step 6: Load the saved trained model and input low-light images for enhancement.

[0072] The texture-guided enhancement method for filtered array spectral images in this application embodiment collects texture maps from the data itself and uses them as a guide for the enhancement method. This method is then trained using paired low-light and normal-light datasets to achieve image enhancement. This solves the problem of poor image quality and noise-induced image overwhelm in low-light environments for filtered array spectral cameras.

[0073] Furthermore, in one embodiment of the present application, a method for extracting a texture map from an image includes first performing a Gaussian filter on the original image to obtain a blurred image; then subtracting the blurred image from the original image to obtain a difference map, and performing a Gaussian filter on the difference map again to obtain a blurred difference map; and finally normalizing the blurred difference map to obtain a texture map.

[0074] Let the input image be I and the texture map be M, then M is expressed as:

[0075] M=h(g2(I-g1(I)))

[0076] Among them, g1 and g2 are Gaussian filters, and h is the normalization function, which is defined as:

[0077]

[0078] Furthermore, in one embodiment of the present application, the texture-guided enhancement network can be divided into a backbone network and a texture-guided branch;

[0079] The backbone network adopts an encoder-decoder structure, consisting of 9 residual blocks, 4 of which serve as encoders and 4 as decoders. Each encoder is followed by a downsampling layer to halve the length and width of the feature map and increase the number of channels. Each decoder is preceded by an upsampling layer to double the length and width of the feature map. Skip connections are used to pass features extracted by the shallow encoder to the decoder.

[0080] The texture guidance branch averages the multi-channel spectral image in the channel dimension, extracts the texture map on the averaged single-channel image, and downsamples the texture map to five different scales. It is connected to the feature map in the backbone network as a guide for the backbone network information.

[0081] Furthermore, in one embodiment of the present application, the texture-guided loss function includes an image restoration part and a texture-guided part;

[0082] The image restoration part uses the common L1 loss function. Specifically, let the image obtained after network enhancement be O and the target image be T, then the image restoration loss function is

[0083] L1=‖OT‖1,

[0084] Among them, ‖·‖1 is the L1 norm;

[0085] The texture guidance part extracts the texture map of the target image band by band, and uses the obtained texture map as a guide to multiply it element by element with the difference between the restored image and the target image, and calculate its L1 norm; if the image obtained after network enhancement is O, the target image is T, and the extracted texture map is M, then the texture guidance loss function is

[0086] L TG =‖M×(OT)‖1

[0087] The final texture-guided loss function can be expressed as

[0088] L=L1+α×L TG

[0089] Among them, α is used to balance the image restoration part and the texture guidance part.

[0090] Figure 2 This is a schematic diagram of a texture information guided enhancement method for a filtered array spectral image according to an embodiment of the present application. After the low-light filtered array spectral image is fed into the texture guided enhancement network, the network extracts a texture map from the averaged input image and connects it to the feature map of the backbone network as a guide. After outputting the enhancement result, the texture guided loss function calculates the loss function value of the result and the target image, and passes the gradient back to the network and updates the network parameters. Compared with other spectral image enhancement methods, the enhancement method in the embodiment of the present invention can better retain the high-frequency texture information in the image, and the final restoration result is closer to the real image.

[0091] Figure 3 This is a schematic structural diagram of a texture information guided enhancement system for filtered array spectral images provided by an embodiment of the present invention. The system includes a training phase and an actual deployment and application phase.

[0092] During the system training phase, it involves paired low-light-normal-light filter array spectral image datasets, filter array spectral image preprocessing subsystems, texture-guided enhancement network subsystems, and texture-guided loss function subsystems:

[0093] A paired low-light-normal-light filtered array spectral image dataset, used for supervised training of the texture-guided enhancement network subsystem and optimization of network parameters;

[0094] The filter array spectral image preprocessing subsystem is used to brighten, augment, normalize and format the input low-light filter array spectral image data;

[0095] A texture-guided enhancement network subsystem is used for supervised training using paired low-light-normal-light filter array spectral image datasets to obtain a network subsystem for enhancing filter array spectral images;

[0096] The texture-guided loss function subsystem is used to supervise the training of the texture-guided enhancement network, optimize the network parameters, and improve the performance of the network subsystem.

[0097] In the system deployment and application stage, it involves the filter array spectral imaging subsystem, the filter array spectral image preprocessing subsystem and the texture guided enhancement network subsystem;

[0098] A filter array spectral imaging subsystem that uses a filter array spectral camera to acquire unprocessed low-light filter array spectral images in deployment scenarios;

[0099] A filter array spectral image preprocessing subsystem, used for normalizing and formatting the filter array spectral image acquired by the filter array spectral imaging subsystem;

[0100] The texture-guided enhancement network subsystem is used to enhance the low-light images acquired by the filter array spectral imaging subsystem.

[0101] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A texture information guided enhancement method for filtered array spectral images, characterized in that: The following steps are involved: Step 1: In the training phase, paired low-light-normal-light filter array spectral image datasets are used to iteratively optimize the model parameters. Step 1.1: Randomly select N pairs of paired data from the paired low-light-normal-light filter array spectral images and preprocess the data, including data brightening, augmentation, normalization, and formatting; Step 1.2: Input the preprocessed low-light filter array spectral image into the texture-guided enhancement network to obtain the enhanced filter array spectral image; Texture-guided enhancement network, including a backbone network and a texture-guided branch; The backbone network adopts an encoder-decoder structure, consisting of 9 residual blocks, 4 of which serve as encoders and 4 as decoders. Each encoder is followed by a downsampling layer to halve the length and width of the feature map and increase the number of channels. Each decoder is preceded by an upsampling layer to double the length and width of the feature map. Skip connections are used to transfer features extracted by the shallow encoder to the decoder. The texture guidance branch averages the multi-channel spectral image in the channel dimension, extracts the texture map from the averaged single-channel image, and downsamples the texture map to five different scales. The texture map is then connected to the feature map in the backbone network as a guide for the backbone network information. Step 1.3: Feed the enhanced filtered array spectral image and the corresponding normal light filtered array spectral image into the texture guided loss function to calculate the loss function value; Texture-guided loss function, including image restoration part and texture-guided part; The image restoration part uses the L1 loss function. Let the image obtained after network enhancement be O and the target image be T. Then the image restoration part is: L1=‖OT‖1 Among them, ‖·‖1 is the L1 norm; The texture guidance part extracts the texture map of the target image band by band, uses the obtained texture map as a guide, multiplies the difference between the restored image and the target image element by element, and calculates its L1 norm; let the image obtained after network enhancement be O, the target image be T, and the extracted texture map be M, then the texture guidance part is: L TG =‖M×(O-T)‖1 The final texture-guided loss function is expressed as: L=L1+α×L TG Among them, α is used to balance the image restoration part and the texture guidance part; Step 1.4: According to the backpropagation algorithm, the gradient values ​​of each parameter are calculated and backpropagated through the loss function value, and the calculated gradient values ​​are used to update each parameter to complete one training iteration; Step 1.5: Repeat steps 1.1 to 1.4 until the number of iterations reaches the preset upper limit, and save the enhanced network structure and parameter values; Step 2: Use stage, load the model structure and parameters in step 1, input the spectral image with low-light filter array into the model, and obtain the enhanced image.

2. The texture information guided enhancement method for filtered array spectral images according to claim 1, characterized in that: In step 1, the texture-guided enhancement network uses the texture map extracted from the preprocessed filtered array spectral image as a guide, and the texture-guided loss function uses the texture map extracted from the target image as a guide.

3. The texture information guided enhancement method for filtered array spectral images according to claim 2, characterized in that: The method to extract the texture map is as follows: First, perform Gaussian filtering on the original image to obtain a blurred image; Then, the blurred image is subtracted from the original image to obtain a difference map, and the difference map is Gaussian filtered again to obtain a blurred difference map; Finally, the blurred difference map is normalized to obtain the texture map; Let the input image be I and the texture map be M, then M is expressed as: M=h(g2(I-g1(I))) Among them, g1 and g2 are Gaussian filters, and h is the normalization function, which is defined as: Where X represents the image to be normalized.

4. A texture information guided enhancement system for filtering array spectral images implementing the method of claim 1, characterized in that: It includes a filter array spectral image preprocessing subsystem, a texture-guided enhancement network subsystem, a texture-guided loss function subsystem, a filter array spectral imaging subsystem, and a paired low-light-normal-light filter array spectral image dataset; A paired low-light-normal-light filtered array spectral image dataset for supervised training of the texture-guided enhancement network subsystem and optimization of network parameters; The filter array spectral image preprocessing subsystem is used to perform data augmentation, normalization and formatting on the filter array spectral image during the training phase; The texture-guided enhancement network subsystem is used to perform supervised training using paired low-light-normal-light filter array spectral image datasets during the training phase to obtain the texture-guided enhancement network subsystem for enhancing the filter array spectral image; During the use phase, it is used to enhance the images acquired by the filter array spectral imaging subsystem; Texture-guided loss function subsystem, used to supervise the training of texture-guided enhancement network and optimize network parameters; A filter array spectral imaging subsystem for acquiring unprocessed low-light filter array spectral images; The output end of the filter array spectral image preprocessing subsystem is connected to the input end of the texture guided enhancement network subsystem, and the output end of the texture guided enhancement network subsystem is connected to the input end of the texture guided loss function subsystem; the output end of the filter array spectral camera imaging subsystem is connected to the input end of the filter array spectral image preprocessing subsystem; The processing process of the system is as follows: During the training phase: Step 1: Randomly select N pairs of data from the paired low-light-normal-light filter array spectral image dataset and input them into the filter array spectral image preprocessing subsystem for data augmentation, normalization, and formatting. The processed paired low-light-normal-light spectral images are then stored. Step 2: Input the low-light spectrum image into the texture-guided enhancement network subsystem for processing to obtain an enhanced filtered array spectrum image; Step 3: The enhanced filtered array spectral image and the stored normal light spectral image are fed into the texture guided loss function subsystem to calculate the loss function value; Step 4: Through the gradient backpropagation algorithm, the value of the loss function is backpropagated to each parameter in the texture-guided enhancement network subsystem, and the parameters are updated according to the gradient values ​​backpropagated to each parameter, completing one training iteration; Step 5: Repeat steps 1 to 4 until the total number of iterations reaches the preset upper limit, stop training, and save the network structure and network parameters in the texture-guided enhancement network subsystem; During the use phase: Step 6: Read the trained network structure and parameter files into the texture-guided enhancement network subsystem; Step 7: Use the filter array spectral camera imaging subsystem to acquire a spectral image and input it into the filter array spectral image preprocessing subsystem for data normalization and formatting; Step 8: The processed spectral image is input into the texture-guided enhancement network subsystem for enhancement processing to obtain an enhanced filtered array spectral image.

Images