Method, device, medium and application for spectral synergy enhanced visible light image

Abstract

The application discloses a method, device, medium and application for spatial-spectrum synergistically enhancing a visible light image, and relates to the technical field of image processing. In the spatial-spectrum synergistically enhancing network of the visible light image, a spatial saliency enhancement link, a frequency domain texture enhancement link and a global feature enhancement link are connected with an enhanced feature fusion module, the enhanced feature fusion module is connected with an element-by-element addition layer, and the spatial saliency enhancement link is connected with the frequency domain texture enhancement link; and after identity mapping of the visible light image, the visible light image is also input into the element-by-element addition layer. The visible light enhanced image obtained by the method has a low distortion degree, a high structural similarity and better image quality.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a method, device, medium, and application for spatial-spectral synergistic enhancement of visible light images. Background Technology

[0002] Most existing deep learning-based image enhancement methods only perform feature processing in the spatial domain, lacking a frequency domain texture information reconstruction process guided by spatial saliency. This results in low structural similarity and high distortion between the enhanced visible light image and the original visible light image, which is detrimental to subsequent downstream target detection tasks. To address this, this application provides a method, device, medium, and application for spatial-spectral co-enhancing of visible light images. Summary of the Invention

[0003] To overcome the shortcomings of the prior art, this application provides a method, apparatus, medium, and application for spatial-spectral synergistic enhancement of visible light images.

[0004] This application is achieved through the following technical solution: A method for spatial-spectral co-enhancing of visible light images includes the following steps: The visible light image to be processed is preprocessed to obtain a standardized visible light image. The standardized visible light image is then input into the visible light image spatial-spectral collaborative enhancement network model and propagated forward once to obtain an enhanced visible light feature map. The enhanced visible light feature map is then subjected to inverse standardization and inverse normalization processing in sequence to obtain the enhanced visible light image. The visible light image spatial-spectral collaborative enhancement network model is obtained by training the visible light image spatial-spectral collaborative enhancement network; The visible light image spatial-spectral collaborative enhancement network includes a spatial saliency enhancement link, a frequency domain texture enhancement link, a global feature enhancement link, an enhancement feature fusion module, and an element-wise additive layer; wherein, the spatial saliency enhancement link, the frequency domain texture enhancement link, and the global feature enhancement link are all connected to the enhancement feature fusion module, the enhancement feature fusion module is connected to the element-wise additive layer, and the spatial saliency enhancement link is also connected to the frequency domain texture enhancement link; This application also performs an identity mapping on the standardized visible light image to obtain a feature map with the original information of the visible light image, and inputs the feature map with the original information of the visible light image into the element-wise addition layer; the element-wise addition layer outputs an enhanced visible light feature map.

[0005] Preferably, the spatial saliency enhancement link includes a Sobel X operator, a Sobel Y operator, a gradient magnitude calculation layer, a Laplacian operator, a high-frequency detail extraction layer, and a spatial weight graph calculation layer connected in sequence.

[0006] Preferably, the frequency domain texture enhancement link includes a Fourier transform layer, which is connected to a decoupling layer. The decoupling layer outputs an amplitude spectrum and a phase spectrum. The amplitude spectrum is sent to a mask layer, and the phase spectrum is sent to an inverse Fourier transform layer. The mask layer is sequentially connected to an element-wise multiplication unit and an inverse Fourier transform layer. The input of the element-wise multiplication unit is also connected to the output of the spatial weight map calculation layer in the spatial saliency enhancement link. The inverse Fourier transform layer outputs a frequency domain texture enhancement feature map.

[0007] Preferably, the global feature enhancement chain includes an average pooling layer, a convolutional layer, a normalization layer, and a sigmoid activation layer connected in sequence; the sigmoid activation layer outputs a global enhancement weight map.

[0008] Preferably, the enhanced feature fusion module includes a feature dimension alignment layer, three weight prediction branches with identical network structures, three element-wise multiplication units, an element-wise addition layer, and a convolutional layer; wherein the outputs of the spatial saliency enhancement link, the frequency domain texture enhancement link, and the global feature enhancement link are all connected to the input of the feature dimension alignment layer; the feature dimension alignment layer is connected to the first weight prediction branch, the second weight prediction branch, and the third weight prediction branch, respectively; the outputs of the first weight prediction branch and the feature dimension alignment layer are both connected to the first element-wise multiplication unit; the outputs of the second weight prediction branch and the feature dimension alignment layer are both connected to the second element-wise multiplication unit; the outputs of the third weight prediction branch and the feature dimension alignment layer are both connected to the third element-wise multiplication unit; the first, second, and third element-wise multiplication units are all connected to the element-wise addition layer; the element-wise addition layer is connected to the convolutional layer; and the convolutional layer outputs a fused enhanced feature map.

[0009] Preferably, each weight prediction branch includes a global average pooling layer, a first convolutional layer, a second convolutional layer, and a sigmoid activation layer connected in sequence.

[0010] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described method for spatial-spectral co-enhanced visible light image.

[0011] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for spatial-spectral co-enhanced visible light image.

[0012] The application of the visible light image spatial-spectral co-enhancement network in the above-mentioned method for spatial-spectral co-enhancement of visible light images in the field of target detection.

[0013] A method for applying a visible light image spatial-spectral collaborative enhancement network to target detection includes the following steps: Step 1) Construct a target detection network that includes a visible light image spatial-spectral co-enhancement network; Step 2) Train the target detection network that includes the visible light image spatial-spectral co-enhancement network to obtain the target detection network model; Step 3) Preprocess both the visible light image and the infrared image to be processed to obtain paired standardized visible light images and standardized infrared images; input the paired standardized visible light images into the visible light image spatial spectrum co-enhancement network of the target detection network model, and input the paired standardized infrared images into the infrared noise reduction and enhancement module of the target detection network model. After one forward propagation, the target detection result can be obtained.

[0014] Preferably, the target detection network including the visible light image spatial-spectral collaborative enhancement network comprises a visible light image spatial-spectral collaborative enhancement network, an infrared noise reduction enhancement network, a cross-modal fusion module, and a multi-scale decoupled target detection head; wherein, the visible light image spatial-spectral collaborative enhancement network and the infrared noise reduction enhancement network are both connected to the cross-modal fusion module, and the cross-modal fusion module is connected to the multi-scale decoupled target detection head; the multi-scale decoupled target detection head outputs target detection results, which include target category, target bounding box, and target confidence.

[0015] Preferably, the infrared noise reduction and enhancement network includes a Fourier transform layer, a decoupling layer, a low-frequency masking layer, a high-frequency masking layer, a soft-threshold noise reduction layer, an element-wise addition layer, a convolutional layer (with a kernel size of 3×3), and an inverse Fourier transform layer. The Fourier transform layer is connected to the decoupling layer, which outputs an amplitude spectrum and a phase spectrum. The amplitude spectrum is sent to the low-frequency masking layer and the high-frequency masking layer, respectively, while the phase spectrum is sent to the inverse Fourier transform layer. The high-frequency masking layer is connected to the soft-threshold noise reduction layer. Both the low-frequency masking layer and the soft-threshold noise reduction layer are connected to the element-wise addition layer. The element-wise addition layer is sequentially connected to the convolutional layer and the inverse Fourier transform layer. The inverse Fourier transform layer outputs a noise-reduced infrared feature map.

[0016] Preferably, the cross-modal fusion module includes two depthwise separable convolutional layers, eight element-wise multiplication units (i.e., the first to the eighth element-wise multiplication units), three element-wise addition layers (i.e., the first to the third element-wise addition layers), and one convolutional layer (with a kernel size of 3×3). The infrared noise reduction module and the visible light image spatial-spectral collaborative enhancement network are each connected to a depthwise separable convolutional layer, and both depthwise separable convolutional layers are connected to a polarization decomposition layer. The polarization decomposition layer performs polarization decomposition on the outputs of the two depthwise separable convolutional layers to obtain K′ feature maps, V'_s feature maps, V'_o feature maps, Q_sim feature maps, Q_opp feature maps, K" feature maps, V"_s feature maps, and V"_o feature maps, respectively. In this process, the K' feature map and the V'_o feature map are input into the first element-wise multiplication unit, the K' feature map and the V'_s feature map are input into the second element-wise multiplication unit, the output of the second element-wise multiplication unit and the Q'_sim feature map are both input into the third element-wise multiplication unit, the output of the first element-wise multiplication unit and the Q'_opp feature map are both input into the fourth element-wise multiplication unit, and the third element-wise multiplication unit and the fourth element-wise multiplication unit are both connected to the first element-wise addition layer; The K' feature map and V'_o feature map are input to the fifth element-wise multiplication unit. The K' feature map and V'_s feature map are input to the sixth element-wise multiplication unit. The output of the sixth element-wise multiplication unit and the Q'_sim feature map are both input to the seventh element-wise multiplication unit. The output of the fifth element-wise multiplication unit and the Q'_opp feature map are both input to the eighth element-wise multiplication unit. The seventh and eighth element-wise multiplication units are both connected to the second element-wise addition layer. The first and second element-wise additive layers are both connected to the third element-wise additive layer, which is connected to a convolutional layer. The convolutional layer outputs a cross-modal fusion enhanced feature map.

[0017] Preferably, the multi-scale decoupled target detection head includes a target category acquisition branch, a bounding box coordinate offset value acquisition branch, a foreground target score acquisition branch, a confidence threshold filtering layer, and a non-maximum suppression (NMS) layer. The target category acquisition branch, the bounding box coordinate offset value acquisition branch, and the foreground target score acquisition branch are all connected to the confidence threshold filtering layer. The confidence threshold filtering layer is connected to the non-maximum suppression (NMS) layer. The non-maximum suppression (NMS) layer outputs the target detection result, which includes the target category, the target bounding box, and the target confidence.

[0018] Preferably, the target category acquisition branch includes three convolutional layers and a sigmoid activation layer connected in sequence, with the sigmoid activation layer outputting the target category confidence score.

[0019] Preferably, the bounding box coordinate offset value acquisition branch includes three convolutional layers connected in sequence. The bounding box coordinate offset value acquisition branch includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence, and the third convolutional layer outputs the bounding box coordinate offset value.

[0020] Preferably, the foreground target score acquisition branch includes three convolutional layers and a sigmoid activation layer connected in sequence, with the sigmoid activation layer outputting the foreground target score.

[0021] Compared with the prior art, the beneficial technical effects of this application are as follows: In the visible light image spatial-spectral collaborative enhancement network constructed in this application, the spatial saliency enhancement link can acquire spatial saliency enhancement feature maps and spatial weight maps; wherein, the spatial saliency enhancement feature maps are sent to the enhancement feature fusion module, and the spatial weight maps are sent to the frequency domain texture enhancement link; the frequency domain texture enhancement link can acquire frequency domain texture enhancement feature maps, and the setting of the frequency domain texture enhancement link realizes the reconstruction and enhancement of frequency domain texture information guided by spatial saliency; the global feature enhancement link can acquire a global enhancement weight map; the enhancement feature fusion module can perform feature fusion on the spatial saliency enhancement feature maps, frequency domain texture enhancement feature maps, and global enhancement weight maps to obtain fused enhancement feature maps; the element-wise addition layer can perform element-wise addition on the fused enhancement feature maps output by the enhancement feature fusion module and the feature maps with original visible light image information obtained by identity mapping to obtain enhanced visible light feature maps with rich texture details and high structural fidelity. Tests show that, compared with histogram equalization methods, the method described in this application produces test visible light enhanced images with lower distortion, higher structural similarity, and higher structural fidelity than the original test visible light images, resulting in better image quality. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the network structure of the visible light image spatial-spectral collaborative enhancement network; Figure 2 for Figure 1 A schematic diagram of the network structure of the spatial weight graph computation layer; Figure 3 for Figure 1 A schematic diagram of the network structure of the enhanced feature fusion module; Figure 4 A schematic diagram of the network structure of a target detection network with a visible light image spatial-spectral collaborative enhancement network; Figure 5 for Figure 4 A schematic diagram of the network structure of the mid-infrared noise reduction and enhancement module; Figure 6 for Figure 4 A schematic diagram of the network structure of the mid-to-span modal fusion module; Figure 7 for Figure 6 Schematic diagram of the network structure of the intermediate polarization decomposition layer; Figure 8 for Figure 4 A schematic diagram of the network structure of a multi-scale decoupled target detection head. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0024] Example 1: A method for spatial-spectral co-enhancing of visible light images includes the following steps: S1. Construct a spatial-spectral collaborative enhancement network for visible light images; The network structure of the visible light image spatial-spectral collaborative enhancement network is as follows: Figure 1 As shown, the visible light image spatial-spectral collaborative enhancement network includes a spatial saliency enhancement link, a frequency domain texture enhancement link, a global feature enhancement link, an enhancement feature fusion module, and an element-wise additive layer; wherein, the spatial saliency enhancement link, the frequency domain texture enhancement link, and the global feature enhancement link are all connected to the enhancement feature fusion module, the enhancement feature fusion module is connected to the element-wise additive layer, and the spatial saliency enhancement link is also connected to the frequency domain texture enhancement link; A visible light image is preprocessed to obtain a standardized visible light image. The preprocessing of the visible light image includes the following steps: First, the resolution of the visible light image is uniformly scaled to 640×640 pixels to obtain a scaled image; then, the pixel values ​​of the scaled image are normalized from the range of 0 to 255 to the range of (0,1) to obtain a normalized visible light image; the normalized visible light image is then standardized to obtain a standardized visible light image.

[0025] In this application, the standardized visible light image is input into the spatial saliency enhancement link, the frequency domain texture enhancement link, and the global feature enhancement link, respectively. Furthermore, this application performs an identity mapping operation on the standardized visible light image, directly outputting the input standardized visible light image through this operation. Therefore, the output feature map contains the original information of the visible light image. This output feature map is called a feature map with the original information of the visible light image. The feature map with the original information of the visible light image is identical to the input standardized visible light image in terms of the number of channels and spatial size, and the pixel values ​​at the same pixel position and in the same channel remain unchanged. In this application, the feature map with the original information of the visible light image is input into an element-wise additive layer. The dimension of the feature map with the original information of the visible light image is consistent with the dimension of the standardized visible light image.

[0026] In the visible light image spatial-spectral collaborative enhancement network, the spatial saliency enhancement link is used to obtain a spatial saliency enhancement feature map and a spatial weight map; wherein, the spatial saliency enhancement feature map is sent to the enhancement feature fusion module, and the spatial weight map is sent to the frequency domain texture enhancement link; the frequency domain texture enhancement link is used to obtain a frequency domain texture enhancement feature map; the global feature enhancement link is used to obtain a global enhancement weight map; the enhancement feature fusion module is used to perform feature fusion on the spatial saliency enhancement feature map, the frequency domain texture enhancement feature map, and the global enhancement weight map to obtain a fused enhancement feature map; the element-wise addition layer is used to perform element-wise addition on the fused enhancement feature map output by the enhancement feature fusion module and the feature map with original visible light image information obtained by identity mapping to obtain an enhanced visible light feature map with rich texture details and high structural fidelity.

[0027] In this application, the spatial saliency enhancement link includes a Sobel X operator, a Sobel Y operator, a gradient magnitude calculation layer, a Laplacian operator, a high-frequency detail extraction layer, and a spatial weight map calculation layer connected in sequence; the spatial weight map calculation layer outputs a spatial saliency enhancement feature map and a spatial weight map; Among them, the network structure of the spatial weight graph computation layer, such as Figure 2 As shown, the spatial weight map calculation layer includes a global average pooling layer, a global max pooling layer, a feature concatenation layer, a convolutional layer (with a kernel size of 1×1), a sigmoid activation layer, and an identity mapping layer. The global average pooling layer and the global max pooling layer are both connected to the feature concatenation layer, which in turn connects to the convolutional layer and the sigmoid activation layer. The sigmoid activation layer outputs the spatial weight map. The spatial weight map calculation layer also includes an identity mapping layer, which performs identity mapping to obtain a spatially saliency-enhanced feature map. In this application, the frequency domain texture enhancement link includes a Fourier transform layer, which is connected to a decoupling layer. The decoupling layer outputs an amplitude spectrum and a phase spectrum. The amplitude spectrum is sent to a mask layer, and the phase spectrum is sent to an inverse Fourier transform layer. The mask layer is sequentially connected to an element-wise multiplication unit and an inverse Fourier transform layer. The input of the element-wise multiplication unit is also connected to the output of the spatial weight map calculation layer in the spatial saliency enhancement link. The inverse Fourier transform layer outputs a frequency domain texture enhancement feature map. In this application, the global feature enhancement chain includes an average pooling layer, a convolutional layer (with a kernel size of 1×1), a normalization layer, and a sigmoid activation layer connected in sequence; the sigmoid activation layer outputs a global enhancement weight map.

[0028] In this application, the network structure of the enhanced feature fusion module is as follows: Figure 3As shown, the enhanced feature fusion module includes a feature dimension alignment layer, three weight prediction branches with identical network structures (i.e., the first weight prediction branch, the second weight prediction branch, and the third weight prediction branch), three element-wise multiplication units (i.e., the first element-wise multiplication unit, the second element-wise multiplication unit, and the third element-wise multiplication unit), an element-wise addition layer, and a convolutional layer (with a kernel size of 3×3). The outputs of the spatial saliency enhancement link, the frequency domain texture enhancement link, and the global feature enhancement link are all connected to the input of the feature dimension alignment layer. The feature dimension alignment layer connects to the first weight prediction branch, the second weight prediction branch, and the third weight prediction branch, respectively. The outputs of the prediction branches and the feature dimension alignment layer are both connected to the first element-wise multiplication unit. The outputs of the second weight prediction branch and the feature dimension alignment layer are both connected to the second element-wise multiplication unit. The outputs of the third weight prediction branch and the feature dimension alignment layer are both connected to the third element-wise multiplication unit. The first, second, and third element-wise multiplication units are all connected to the element-wise addition layer, which is connected to the convolutional layer. The convolutional layer outputs a fused and enhanced feature map. Each weight prediction branch contains an average pooling layer, a first convolutional layer (1×1 kernel size), a second convolutional layer (1×1 kernel size), and a Sigmoid activation layer connected in sequence. The Sigmoid activation layer outputs normalized weights, which are the outputs of the weight prediction branches.

[0029] In this application, the function of the spatial saliency enhancement link is as follows: After standardizing the visible light image input spatial saliency enhancement link using the Sobel X operator, the Sobel X operator extracts horizontal edge gradient information from the visible light image, resulting in a feature map with prominent horizontal edge and texture information. The Sobel Y operator extracts vertical edge gradient information from the feature map with prominent horizontal edge and texture information, resulting in a feature map with prominent edge and texture information in both horizontal and vertical directions. The gradient magnitude calculation layer calculates the gradient magnitude of the feature map with prominent edge and texture information in both horizontal and vertical directions, resulting in a feature map with significant edge intensity and prominent texture details, completing the initial detail enhancement of the visible light image. The Laplacian operator performs a second-order differential convolution operation on the feature map with significant edge intensity and prominent texture details, resulting in a high-frequency detail feature map containing positive and negative pixel responses. The high-frequency detail extraction layer performs Laplacian second-order differential high-frequency detail extraction on the high-frequency detail feature map containing positive and negative pixel responses, resulting in a feature map with enhanced high-frequency details and edge magnitude. The high-frequency detail extraction layer uses Laplacian second-order derivative high-frequency detail extraction in the following way: In the high-frequency detail extraction layer, a 3×3 Laplacian second-order derivative operator is used to perform pixel-by-pixel convolution operations on the high-frequency detail feature map containing positive and negative pixel responses. The convolution kernel of the Laplacian second-order derivative operator is as follows: the first row is 0, 1, 0; the second row is 1, -4, 1; and the third row is 0, 1, 0. The pixel-by-pixel convolution operation is as follows: for each pixel position in the high-frequency detail feature map containing positive and negative pixel responses, the pixel value at that position is subtracted from the sum of the pixel values ​​at the four adjacent pixel positions above, below, to the left, and to the right of that pixel position to obtain the Laplacian second-order derivative response value for that pixel position. The Laplacian second-order derivative response value can be positive or negative; the larger its absolute value, the stronger the high-frequency details and edges at that pixel position. Then, the absolute value of the Laplacian second-order derivative response value is taken to obtain a feature map where both high-frequency details and edge amplitudes are enhanced. The specific functions of the spatial weight graph calculation layer are as follows: In the high-frequency detail feature map input spatial weight map computation layer, the global average pooling layer, global max pooling layer, and identity mapping layer perform global average pooling on the high-frequency detail feature map, the global max pooling layer performs global max pooling on the high-frequency detail feature map, the feature concatenation layer concatenates the feature maps output by the global average pooling layer and the global max pooling layer, the convolutional layer performs convolution on the feature map output by the feature concatenation layer, and the Sigmoid activation layer performs a non-linear transformation on the feature map output by the convolutional layer, generating a value between (0,1) for each element. The spatial enhancement weight map of the interval is output to the element-wise multiplication unit of the frequency domain texture enhancement link; while the identity mapping layer performs identity mapping on the high-frequency detail feature map to obtain the spatial saliency enhancement feature map, which is output to the enhancement feature fusion module. Among them, the identity mapping layer performs identity mapping operation on the high-frequency detail feature map. The identity mapping operation does not perform any parameterization transformation on the input high-frequency detail feature map, but directly outputs the high-frequency detail feature map as the spatial saliency enhancement feature map. That is, the output spatial saliency enhancement feature map is exactly the same as the input high-frequency detail feature map in terms of the number of channels and spatial size, and the pixel value at the same pixel position and the same channel remains unchanged, realizing the effective transmission of high-frequency detail feature map information.

[0030] The function of the frequency domain texture enhancement link is: After the standardized visible light image is input into the Fourier transform layer of the frequency domain texture enhancement link, the Fourier transform layer converts the spatial domain features of the standardized visible light image to the frequency domain features through a two-dimensional fast Fourier transform, obtaining a frequency domain feature map. The decoupling layer performs a decoupling operation on the frequency domain feature map, separating the amplitude spectrum and the phase spectrum. The masking layer performs a masking operation on the amplitude spectrum based on a preset high-frequency mask, filtering out high-frequency texture components to obtain an amplitude spectrum with prominent high-frequency information. The preset high-frequency mask is obtained by performing FFT shift on the frequency domain feature map. A shifting process is performed to move high-frequency components to the spectral center of the frequency domain feature map; an ideal high-pass filter mask with the same size as the frequency domain feature map is constructed, with the spectral center as the center of the ideal high-pass filter mask, and the cutoff frequency D0 of the ideal high-pass filter mask is set to 40, where D0 is a pixel distance threshold; values ​​are assigned to each position of the ideal high-pass filter mask, specifically: regions with a distance to the spectral center less than or equal to D0 are assigned a value of 0, and regions with a distance to the spectral center greater than D0 are assigned a value of 1, thus obtaining the preset high-frequency mask; step by step... The element-wise multiplication unit multiplies the amplitude spectrum highlighting high-frequency information and the spatial weight map output from the spatial weight map calculation layer in the spatial saliency enhancement link element-wise to obtain a saliency-weighted amplitude spectrum with enhanced high-frequency texture. The inverse Fourier transform layer performs an inverse two-dimensional fast Fourier transform on the saliency-weighted amplitude spectrum with enhanced high-frequency texture and the phase spectrum output from the mask layer to convert the frequency domain features back to spatial domain features, resulting in a feature map with enhanced frequency domain texture information. In this application, the setting of the frequency domain texture enhancement link realizes the reconstruction and enhancement of frequency domain texture information guided by spatial saliency. In the domain texture enhancement link, the decoupling layer performs decoupling operations in the following way: the decoupling layer separates the complex value at each frequency domain coordinate position in the frequency domain feature map into an amplitude spectrum and a phase spectrum. The amplitude spectrum takes the modulus of the complex value at the frequency domain coordinate position, and the phase spectrum takes the argument of the complex value at the frequency domain coordinate position. In the frequency domain texture enhancement link, the mask layer performs a masking operation on the amplitude spectrum based on a preset high-frequency mask in the following way: the preset high-frequency mask and the amplitude spectrum are multiplied element-wise at the same frequency domain coordinate position. That is, for each frequency domain coordinate position, the product of the amplitude spectrum value at that frequency domain coordinate position and the preset high-frequency mask value is taken as the amplitude spectrum value after masking at that frequency domain coordinate position. The preset high-frequency mask is a soft mask with continuously taking values, and the preset high-frequency mask value range is [0,1]. At high-frequency coordinate positions far from the center of the spectrum, the preset high-frequency mask value approaches 1, and the amplitude spectrum component at the corresponding position is preserved to a greater extent. At low-frequency coordinate positions close to the center of the spectrum, the preset high-frequency mask value approaches 0, and the amplitude spectrum component at the corresponding position is suppressed and attenuated. The above settings enable this application to effectively filter out high-frequency texture components and obtain an amplitude spectrum with prominent high-frequency information. The function of the global feature enhancement link is: In the average pooling layer of the global feature enhancement chain, which is input to a standardized visible light image, the average pooling layer performs an average pooling operation on the standardized visible light image to compress the spatial dimension of the visible light image and capture the global brightness information of the visible light image, resulting in a feature map with global brightness information. A convolutional layer (with a kernel size of 1×1) performs a convolution operation on the feature map with global brightness information. A normalization layer normalizes the output of the convolutional layer. A sigmoid activation layer performs a nonlinear transformation on the output of the normalization layer to generate a global enhancement weight map with each element taking values ​​in the range (0,1). This global enhancement weight map contains the global brightness information of the visible light image.

[0031] In this application, the functionality of the enhanced feature fusion module is as follows: The feature dimension alignment layer is used to uniformly align the input spatial saliency enhancement feature map, frequency domain texture enhancement feature map, and global enhancement weight map in terms of channel dimension and spatial size. The feature dimension alignment layer outputs the aligned spatial saliency enhancement feature map, aligned frequency domain texture enhancement feature map, and aligned global enhancement weight map. The aligned spatial saliency enhancement feature map is fed to the first weight prediction branch and the first element-wise multiplication unit, the aligned frequency domain texture enhancement feature map is fed to the second weight prediction branch and the second element-wise multiplication unit, and the aligned global enhancement weight map is fed to the third weight prediction branch and the third element-wise multiplication unit. The first weight prediction branch obtains prediction weights with spatial saliency awareness based on the aligned spatial saliency enhancement feature map, the second weight prediction branch obtains prediction weights with frequency domain texture awareness based on the aligned frequency domain texture enhancement feature map, and the third weight prediction branch obtains prediction weights with global brightness and semantic awareness based on the aligned global enhancement weight map. The first element-wise multiplication unit is used to process the prediction weights output by the first weight prediction branch and the feature dimension... The aligned spatial saliency enhancement feature map output by the alignment layer is multiplied element-wise to obtain a spatially saliency-enhanced feature map. The second element-wise multiplication unit is used to multiply the predicted weights output by the second weight prediction branch and the aligned frequency domain texture enhancement feature map output by the feature dimension alignment layer element-wise to obtain a frequency domain texture enhancement feature map. The third element-wise multiplication unit is used to multiply the predicted weights output by the third weight prediction branch and the aligned global enhancement weight map output by the feature dimension alignment layer element-wise to obtain a feature map with enhanced global brightness and semantic information. The element-wise addition layer performs feature aggregation on the feature maps output by the first, second, and third element-wise multiplication units to obtain a feature map with complementary aggregation of spatial saliency enhancement information, frequency domain texture enhancement information, and global brightness and semantic information. The convolutional layer (with a kernel size of 3×3) performs convolution operation on the feature map with complementary aggregation of spatial saliency enhancement information, frequency domain texture enhancement information, and global brightness and semantic information to obtain a fused enhanced feature map with rich spatial saliency information, rich frequency domain texture information, and rich global brightness and semantic information. In this application, the average pooling layer in each weight prediction branch is used to perform average pooling operation on the input feature map, the first convolutional layer (with a kernel size of 1×1) and the second convolutional layer (with a kernel size of 1×1) are used to perform convolution operation, and the Sigmoid activation layer outputs normalized weights, which are the outputs of the weight prediction branch.

[0032] S2. Train the visible light image spatial-spectral co-enhancement network to obtain the visible light image spatial-spectral co-enhancement network model; including the following steps: S2-1. Obtain the training set, validation set, and test set, specifically including the following steps: The LLVIP dataset was used as the original dataset A, which contains 15,488 pairs of spatiotemporally rigorous registered low-light visible-infrared image pairs; the M3FD dataset was used as the original dataset B, which contains 4,200 pairs of registered infrared-visible image pairs.

[0033] The visible light images in both original datasets A and B are preprocessed, including the following steps: First, the resolution of all visible light images in both datasets A and B is uniformly scaled to 640×640 pixels to obtain scaled images; then, the pixel values ​​of the scaled images are normalized from the range of 0 to 255 to the range of (0,1) to obtain normalized visible light images; the normalized visible light images are then standardized to obtain the standardized visible light images corresponding to original datasets A and B.

[0034] In this application, the standardized visible light images corresponding to the original dataset A are subjected to random data augmentation, including random horizontal flipping, random cropping and scaling (scaling range of 0.8 to 1.2), random brightness and contrast perturbation (brightness perturbation range of -0.2 to 0.2, contrast perturbation range of -0.2 to 0.2), and random Gaussian noise injection (the injected Gaussian noise has a mean of 0 and a standard deviation of 0.01), to obtain the data-augmented visible light images. The probability of random horizontal flipping is 0.5. The data-augmented visible light images are divided into training and testing sets in a 9:1 ratio. In this application, the standardized visible light images corresponding to the original dataset B constitute the validation set.

[0035] S2-2. Train the visible light image spatial-spectral co-enhancement network based on the training set and validation set to obtain the visible light image spatial-spectral co-enhancement network model; including the following steps: Augmented visible light images are read from the training set in batches (batch size 16). These augmented images are then input into the visible light image spatial-spectral co-augmentation network for forward propagation. The total loss of the network is calculated, and backpropagation is performed based on this loss to update the network's weight parameters. This completes one epoch of training. The next batch of visible light images is then read for the next epoch of training, and this process is repeated until a preset iteration threshold is reached (100 in this embodiment). In this application, validation is performed every 5 epochs using a validation set. The weight parameters and accuracy of the network during validation are saved. The weight parameters of the network corresponding to the validation process with the highest accuracy are used as the weight parameters of the visible light image spatial-spectral co-augmentation network model, thus obtaining the model.

[0036] In this application, validation is performed every 5 epochs using a validation set. The validation process includes the following steps: reading standardized visible light images from the validation set in batches (batch size is 16), inputting the read standardized visible light images into the visible light image spatial-spectral co-enhancement network, propagating forward, outputting the accuracy, and saving the weight parameters of the visible light image spatial-spectral co-enhancement network.

[0037] In this application, the total loss of the visible light image spatial-spectral collaborative enhancement network refers to the pixel-level average absolute error loss. The pixel-level average absolute error loss is calculated as follows: the enhanced visible light feature map output by the visible light image spatial-spectral collaborative enhancement network is compared with the reference visible light feature map at each pixel position and each channel, and the absolute value is taken to obtain the absolute error at that pixel position; the absolute errors of all pixel positions and all channels are summed and divided by the product of the total number of pixel positions and the number of channels to obtain the pixel-level average absolute error loss; wherein, the reference visible light feature map is the feature map obtained by the identity mapping operation on the standardized visible light image input to the visible light image spatial-spectral collaborative enhancement network, and the enhanced visible light feature map and the reference visible light feature map have the same spatial size and number of channels.

[0038] In this embodiment, the training of the visible light image spatial-spectral co-enhancement network is implemented based on an NVIDIA A800 GPU, and the Adam optimizer is used to iteratively optimize the network parameters. The initial learning rate is set to 1×10⁻⁶. -4 The weight decay coefficient is set to 5×10. -4 The learning rate employs a cosine annealing decay strategy, with a minimum learning rate set to 1×10⁻⁶.-6 .

[0039] S3. Preprocess the visible light image to be processed to obtain a standardized visible light image. Input the standardized visible light image into the visible light image spatial-spectral collaborative enhancement network model obtained in step S2. Propagate forward once to obtain the enhanced visible light feature map. Perform inverse standardization and inverse normalization processing on the enhanced visible light feature map in sequence to obtain the enhanced visible light image. The steps for preprocessing the visible light image to be processed are the same as those for preprocessing the visible light image in step S1.

[0040] Test 1: To verify the image enhancement effect of the spatial-spectral collaborative enhancement method for visible light images described in this application, this application first utilizes the spatial-spectral collaborative enhancement method for visible light images described in this application (hereinafter referred to as the Ours method) and the histogram equalization method (derived from "Digital Image Processing (Third Edition)", RC Gonzalez, RE (Woods, translated by Ruan Qiuqi, Ruan Yuzhi, et al., published by Electronic Industry Press) Based on a test set, an enhanced visible light feature map is generated. This enhanced visible light feature map is then subjected to inverse normalization and inverse standardization processing to obtain an enhanced visible light image, referred to as the test enhanced visible light image. Subsequently, the visible light image in the test set corresponding to the test enhanced visible light image is called the original test visible light image. Using the original test visible light image as a benchmark, the test enhanced visible light image and the original test visible light image are compared using full-reference image quality evaluation metrics to obtain Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The results of PSNR and SSIM are shown in Table 1. PSNR and SSIM serve as image enhancement quality evaluation metrics to assess the quality improvement effect of the enhanced image.

[0041] Table 1 shows the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) results for the Ours method and histogram equalization method. Histogram equalization method 16.82 0.523 Ours method 24.36 0.827

[0042] As shown in Table 1, the peak signal-to-noise ratio (PSNR) obtained by the Ours method is (24.36-16.82) / 16.82=44.83% higher than that obtained by the histogram equalization method. This indicates that, compared with the histogram equalization method, the Ours method results in a lower degree of distortion and better image quality between the enhanced test visible light image and the original test visible light image. The structural similarity (SSIM) obtained by Ours method is improved by (0.827-0.523) / 0.523=58.13% compared to the structural similarity (SSIM) obtained by histogram equalization. This indicates that, compared to histogram equalization, the test visible light enhanced image obtained by Ours method has higher structural similarity, higher structural fidelity, and better image quality than the original test visible light image.

[0043] Example 2: A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described method for spatial-spectral co-enhanced visible light image.

[0044] Example 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for spatial-spectral co-enhanced visible light image.

[0045] Example 4: The application of the visible light image spatial-spectral co-enhancement network in the above-mentioned method for spatial-spectral co-enhancement of visible light images in the field of target detection.

[0046] Example 5: A method for applying a visible light image spatial-spectral collaborative enhancement network to target detection includes the following steps: Step 1) Construct a target detection network that includes a visible light image spatial-spectral co-enhancement network; The network structure of the target detection network, which includes a visible light image spatial-spectral collaborative enhancement network, is as follows: Figure 4 As shown, this includes a visible light image spatial-spectral collaborative enhancement network (such as...). Figure 1 The system consists of an infrared noise reduction and enhancement network, a cross-modal fusion module, and a multi-scale decoupled target detection head. The visible light image spatial-spectral collaborative enhancement network and the infrared noise reduction and enhancement network are both connected to the cross-modal fusion module, which in turn is connected to the multi-scale decoupled target detection head. The multi-scale decoupled target detection head outputs target detection results, including target category, target bounding box, and target confidence. Among them, the visible light image spatial-spectral collaborative enhancement network is used to obtain the enhanced visible light feature map, and the infrared denoising enhancement network is used to obtain the denoised infrared feature map; the cross-modal fusion module is used to perform feature fusion on the enhanced visible light feature map and the denoised infrared feature map to obtain the cross-modal fused enhanced feature map; the multi-scale decoupled target detection head is used to predict the cross-modal fused enhanced feature map to obtain the target detection result, which includes the target bounding box, category and confidence score.

[0047] In this application, the network structure of the infrared noise reduction and enhancement network is as follows: Figure 5 As shown, the infrared noise reduction and enhancement network includes a Fourier transform layer, a decoupling layer, a low-frequency masking layer, a high-frequency masking layer, a soft-threshold noise reduction layer, an element-wise addition layer, a convolutional layer (with a kernel size of 3×3), and an inverse Fourier transform layer. The Fourier transform layer is connected to the decoupling layer, which outputs an amplitude spectrum and a phase spectrum. The amplitude spectrum is sent to the low-frequency masking layer and the high-frequency masking layer, respectively, while the phase spectrum is sent to the inverse Fourier transform layer. The high-frequency masking layer is connected to the soft-threshold noise reduction layer. Both the low-frequency masking layer and the soft-threshold noise reduction layer are connected to the element-wise addition layer. The element-wise addition layer is sequentially connected to the convolutional layer and the inverse Fourier transform layer. The inverse Fourier transform layer outputs a noise-reduced infrared feature map.

[0048] In this application, the network structure of the cross-modal fusion module is as follows: Figure 6 As shown, the cross-modal fusion module includes two depthwise separable convolutional layers, one polarization decomposition layer, eight element-wise multiplication units (i.e., the first to the eighth element-wise multiplication units), three element-wise addition layers (i.e., the first to the third element-wise addition layers), and one convolutional layer (with a kernel size of 3×3). The infrared noise reduction module and the visible light image spatial-spectral collaborative enhancement network are each connected to a depthwise separable convolutional layer. Both depthwise separable convolutional layers are connected to the polarization decomposition layer, which performs polarization decomposition on the outputs of the two depthwise separable convolutional layers to obtain the K′ feature map, V′_s feature map, V′_o feature map, Q_sim feature map, Q_opp feature map, K" feature map, V"_s feature map, and V"_o feature map, respectively. The feature maps are as follows: K′ feature map contains semantic information of the target in the visible light image; V′_s feature map contains contour and structural information of the target in the visible light image; V′_o feature map contains color information of the target in the visible light image; Q_sim feature map contains contour and structural information of the target that exists in both infrared and visible light images of the same scene; Q_opp feature map contains information other than contour and structural information of the target that exists in both infrared and visible light images of the same scene; K" feature map contains thermal radiation information of the target location in the infrared image; V"_s feature map contains contour and structural information of the target in the infrared image; V"_o feature map contains thermal radiation and texture information of the target in the infrared image. Specifically, the K′ feature map and V′_o feature map are input to the first element-wise multiplication unit; the K′ feature map and V′_s feature map are input to the second element-wise multiplication unit; the output of the second element-wise multiplication unit and the Q_sim feature map are both input to the third element-wise multiplication unit; the output of the first element-wise multiplication unit and the Q_opp feature map are both input to the fourth element-wise multiplication unit; and the third and fourth element-wise multiplication units are both connected to the first element-wise addition layer. The K′ feature map and V′_o feature map are input to the fifth element-wise multiplication unit; the K′ feature map and V′_s feature map are input to the sixth element-wise multiplication unit; the output of the sixth element-wise multiplication unit and the Q_sim feature map are both input to the seventh element-wise multiplication unit; the output of the fifth element-wise multiplication unit and the Q_opp feature map are both input to the eighth element-wise multiplication unit; and the seventh and eighth element-wise multiplication units are both connected to the second element-wise addition layer. The first and second element-wise additive layers are both connected to the third element-wise additive layer. The third element-wise additive layer is connected to a convolutional layer (with a kernel size of 3×3). The convolutional layer (with a kernel size of 3×3) outputs a cross-modal fusion enhanced feature map.

[0049] In this application, the network structure of the multi-scale decoupled target detection head, such as... Figure 8 As shown, the multi-scale decoupled target detection head includes a target category acquisition branch, a bounding box coordinate offset acquisition branch, a foreground target score acquisition branch, a confidence threshold filtering layer, and a non-maximum suppression (NMS) layer. The target category acquisition branch, the bounding box coordinate offset acquisition branch, and the foreground target score acquisition branch are all connected to the confidence threshold filtering layer. The confidence threshold filtering layer is connected to the non-maximum suppression (NMS) layer, and the non-maximum suppression (NMS) layer outputs the target detection result. The target detection results include target category, target bounding box, and target confidence score. The target category acquisition branch consists of three sequentially connected convolutional layers (each with a 3×3 kernel) and a sigmoid activation layer, with the sigmoid activation layer outputting the target category confidence score. The bounding box coordinate offset acquisition branch consists of three sequentially connected convolutional layers (each with a 3×3 kernel), and the third convolutional layer outputs the bounding box coordinate offset value. The foreground target score acquisition branch consists of three sequentially connected convolutional layers (each with a 3×3 kernel) and a sigmoid activation layer, with the sigmoid activation layer outputting the foreground target score. The confidence threshold filtering layer uses a preset confidence threshold (in this embodiment, the preset confidence threshold is 0.25) as a benchmark, and filters candidate target boxes based on the target category confidence, bounding box coordinate offset value and foreground target score to obtain the filtered candidate target boxes and their corresponding target categories and coordinate information. The filtered candidate target boxes and their corresponding target categories and coordinate information constitute the candidate target box set. In this application, the confidence threshold filtering layer uses a preset confidence threshold as a benchmark and filters candidate target boxes based on target category confidence, bounding box coordinate offset, and foreground target score. Specifically, the foreground target score of each pixel position output by the foreground target score acquisition branch is compared with the preset confidence threshold. For pixel positions where the foreground target score is greater than or equal to the preset confidence threshold, the candidate target box corresponding to that pixel position, along with the target category confidence and bounding box coordinate offset, is retained. For pixel positions where the foreground target score is less than the preset confidence threshold, the candidate target box corresponding to that pixel position, along with the target category confidence and bounding box coordinate offset, is removed. The retained candidate target boxes and their corresponding target categories and coordinate information together constitute a candidate target box set. The Non-Maximum Suppression (NMS) layer performs redundant overlapping detection box suppression and deduplication on the candidate target box set output by the confidence threshold filtering layer, and outputs the target detection results including target category, target bounding box, and target confidence. In this application, the Non-Maximum Suppression (NMS) layer performs redundant overlapping detection box suppression and deduplication on the candidate target box set as follows: First, all candidate target boxes in the candidate target box set are sorted from high to low according to the target confidence. The candidate target box with the highest current ranking is selected as the retained box. Then, the intersection-union ratio (IUR) between the remaining candidate target boxes and the retained box is calculated. Candidate target boxes with an IUR greater than a preset IUR threshold (0.45 in this embodiment) are identified as redundant detection boxes that overlap with the retained box and are removed. Then, the highest-ranked candidate target box is selected again from the remaining candidate target boxes as the retained box, and the above sorting and removal process is repeated until there are no candidate target boxes with an IUR greater than the preset IUR threshold in the candidate target box set. The redundant overlapping detection box suppression and deduplication operation is then completed. In this application, the IUR is the ratio of the area of ​​the intersection region to the area of ​​the union region between two candidate target boxes.

[0050] In this application, after preprocessing, a normalized visible light image is obtained, and the normalized visible light image is input into the visible light image spatial-spectral collaborative enhancement network; after preprocessing, an infrared image is obtained, and the normalized infrared image is input into the infrared noise reduction enhancement network; wherein, the method of preprocessing the visible light image and the method of preprocessing the infrared image in Embodiment 5 are the same as the method of preprocessing the visible light image in Embodiment 1.

[0051] In this application, the infrared noise reduction and enhancement network functions as follows: After a standardized infrared image is input into the Fourier transform layer of the infrared noise reduction and enhancement network, the Fourier transform layer converts the spatial domain features of the standardized infrared image to the frequency domain features through a two-dimensional fast Fourier transform, obtaining a frequency domain feature map. A decoupling layer decouples the frequency domain feature map to obtain the amplitude spectrum and phase spectrum. A low-frequency masking layer performs a masking operation on the amplitude spectrum based on a low-frequency mask, filtering out low-frequency components with the target thermal radiation profile, resulting in a low-frequency amplitude spectrum with rich thermal radiation profile information. A high-frequency masking layer performs a masking operation on the amplitude spectrum based on a high-frequency mask, filtering out noisy high-frequency components, resulting in a high-frequency amplitude spectrum with prominent high-frequency noise. A soft-threshold noise reduction layer performs soft-threshold noise reduction processing on the high-frequency amplitude spectrum with prominent high-frequency noise, suppressing high-frequency components with amplitudes smaller than a preset noise threshold (0.08 in this embodiment), and suppressing high-frequency components with large amplitudes. The high-frequency spectral coefficients at a preset noise threshold retain their sign information and relative amplitude, resulting in a high-frequency amplitude spectrum with weak edge-corresponding high-frequency feature components and effectively suppressed noise. An element-wise addition layer adds the low-frequency amplitude spectrum with rich thermal radiation contour information and the high-frequency amplitude spectrum with weak edge-corresponding high-frequency feature components and effectively suppressed noise element-wise to obtain a frequency-domain denoised fused amplitude spectrum. A convolutional layer (with a 3×3 kernel) performs a convolution operation on the frequency-domain denoised fused amplitude spectrum to obtain a denoised amplitude spectrum. An inverse Fourier transform layer performs an inverse two-dimensional fast Fourier transform on the denoised amplitude spectrum and the phase spectrum output from the decoupling layer, converting the frequency domain features back to spatial domain features to obtain a denoised infrared feature map. In this application, the infrared denoising enhancement network not only effectively preserves the target thermal radiation contour information but also effectively suppresses high-frequency noise. The low-frequency mask is obtained as follows: The frequency domain feature map is shifted using an FFTshift operation to move the low-frequency components to the center of the frequency domain feature map's spectrum; an ideal low-pass filter mask with the same size as the frequency domain feature map is constructed, with the center of the spectrum as the center and the cutoff frequency D0 of the ideal low-pass filter mask set to 40, where D0 is a pixel distance threshold; values ​​are assigned to each position of the ideal low-pass filter mask, specifically: regions with a distance to the center of the spectrum less than or equal to D0 are assigned a value of 1, and regions with a distance to the center of the spectrum greater than D0 are assigned a value of 0, thus obtaining the low-frequency mask. The high-frequency mask is obtained as follows: The frequency domain feature map is shifted using an FFTshift operation to move the high-frequency components to the center of the frequency domain feature map's spectrum; an ideal high-pass filter mask with the same size as the frequency domain feature map is constructed, with the center of the spectrum as the center and the cutoff frequency D0 of the ideal high-pass filter mask set to 40, where D0 is a pixel distance threshold; values ​​are assigned to each position of the ideal high-pass filter mask, specifically: regions with a distance to the center of the spectrum less than or equal to D0 are assigned a value of 0, and regions with a distance to the center of the spectrum greater than D0 are assigned a value of 1, thus obtaining the high-frequency mask. In this application, the masking operation of the low-frequency mask layer specifically involves multiplying the low-frequency mask and the amplitude spectrum element-wise at the same frequency domain coordinate position. That is, for each frequency domain coordinate position, the product of the amplitude spectrum value at that frequency domain coordinate position and the low-frequency mask value is taken as the amplitude spectrum value after masking at that frequency domain coordinate position. The low-frequency mask is a continuously valued soft mask with a value range of [0,1]. At low-frequency coordinate positions closer to the center of the spectrum, the low-frequency mask value tends to be close to 1, and the low-frequency amplitude component representing the thermal radiation profile at the corresponding position is preserved to the greatest extent. At high-frequency coordinate positions farther from the center of the spectrum, the low-frequency mask value tends to be close to 0, and the high-frequency amplitude component at the corresponding position is suppressed and attenuated. Through the masking operation settings of the low-frequency mask layer, this application can effectively filter out the low-frequency amplitude component carrying the target thermal radiation profile information.

[0052] The masking operation of the high-frequency masking layer is as follows: the high-frequency mask and the amplitude spectrum are multiplied element-wise at the same frequency domain coordinate position. That is, for each frequency domain coordinate position, the product of the amplitude spectrum value at that frequency domain coordinate position and the high-frequency mask value is taken as the amplitude spectrum value after masking at that frequency domain coordinate position. The high-frequency mask is a continuously valued soft mask with a value range of [0,1]. At high-frequency coordinate positions far from the center of the spectrum, the high-frequency mask value approaches 1, and the high-frequency amplitude components containing noise at the corresponding positions are preserved to a greater extent. At low-frequency coordinate positions close to the center of the spectrum, the high-frequency mask value approaches 0, and the low-frequency amplitude components at the corresponding positions are suppressed and attenuated. Through the masking operation settings of the high-frequency masking layer, this application can effectively filter out high-frequency amplitude components containing noise information.

[0053] In this application, the cross-modal fusion module has the following functions: The enhanced visible light feature map and the denoised infrared feature map are respectively input into a depthwise separable convolutional layer in the cross-modal fusion module; the two depthwise separable convolutional layers perform depthwise separable convolution operations on the enhanced visible light feature map and the denoised infrared feature map, respectively, to complete the feature projection and dimension matching of the channel dimension, resulting in visible light projection feature map and infrared projection feature map with consistent dimensions; the polarization decomposition layer performs polarization decomposition operations on the visible light projection feature map and the infrared projection feature map, respectively, to obtain K′ feature map, V′_s feature map, V′_o feature map, Q_sim feature map, Q_opp feature map, K" feature map, V"_s feature map, and V"_o feature map; the first element-wise multiplication unit performs element-wise multiplication on the K′ feature map and the V′_o feature map to obtain a feature map containing target semantic information and target color information; the second element-wise multiplication unit performs element-wise multiplication on the K′ feature map and the V′_s feature map. The feature maps are multiplied element-wise to obtain a feature map containing target semantic information, target contour information, and target structural information. The third element-wise multiplication unit multiplies the Q_sim feature map and the feature map output by the second element-wise multiplication unit element-wise to obtain a feature map enhanced with contour and structural information. The fourth element-wise multiplication unit multiplies the Q_opp feature map and the feature map output by the first element-wise multiplication unit element-wise to obtain a feature map enhanced with target semantic information and target color information. The first element-wise addition layer adds the feature maps output by the third and fourth element-wise multiplication units element-wise to obtain a feature map that enhances target semantic information, target color information, target contour information, and target structural information. The fifth element-wise multiplication unit multiplies the K" feature map and the V"_o feature map element-wise to obtain a feature map containing thermal radiation information and texture information. The sixth element-wise multiplication unit multiplies the K" feature map and the V"_s feature map element-wise to obtain a feature map containing... The system first generates a feature map containing thermal radiation, contour, and structural information of the target location. The seventh element-wise multiplication unit performs element-wise multiplication on the Q_sim feature map and the feature map output from the sixth element-wise multiplication unit to obtain a feature map enhanced with contour and structural information. The eighth element-wise multiplication unit performs element-wise multiplication on the Q_opp feature map and the feature map output from the fifth element-wise multiplication unit to obtain a feature map enhanced with thermal radiation and texture information. The second element-wise addition layer performs element-wise addition on the feature maps output from the seventh and eighth element-wise multiplication units to obtain a feature map enhanced with contour, structural, thermal radiation, and texture information. The third element-wise addition layer performs element-wise addition on the feature maps output from the first and second element-wise addition layers to obtain a feature map with dual-modal complementary information. A convolutional layer (with a 3×3 kernel) performs convolution operations on the feature map with dual-modal complementary information, outputting a cross-modal fusion enhanced feature map. In this application, the network structure of the polarization decomposition layer is as follows: Figure 7 As shown, the polarization decomposition layer includes eight convolutional layers (each with a kernel size of 1×1), which are the first to the eighth convolutional layers; the visible light projection feature map output from one of the depth-separable convolutional layers is input to the first to the fifth convolutional layers, and the infrared projection feature map output from the other depth-separable convolutional layer is input to the fourth to the eighth convolutional layers. In the polarization decomposition layer, the first to third convolutional layers are used to perform convolution operations on the visible light projection feature map, respectively, to realize the channel dimension mapping transformation of the features in the visible light projection feature map, and respectively obtain the K′ feature map containing the semantic information of the target in the visible light image, the V′_s feature map containing the contour information and structural information of the target in the visible light image, and the V′_o feature map containing the color information of the target in the visible light image; The fourth convolutional layer performs convolution operations on the visible light projection feature map and the infrared projection feature map, realizing channel dimension mapping transformation of the features in the visible light projection feature map and the infrared projection feature map, and obtaining the Q_sim feature map containing the target contour information and target structure information that exist in both the infrared image and the visible light image in the same scene. The fifth convolutional layer performs convolution operations on the visible light projection feature map and the infrared projection feature map, and performs channel dimension mapping transformation on the features in the visible light projection feature map and the infrared projection feature map to obtain a Q_opp feature map containing information other than the target contour information and target structure information that exist in both infrared images and visible light images in the same scene. The sixth to eighth convolutional layers are used to perform convolution operations on the infrared projection feature map, respectively, to perform channel dimension mapping transformation on the features in the visible light projection feature map, and respectively obtain the K" feature map containing the thermal radiation information of the target position in the infrared image, the V"_s feature map containing the target contour information and target structure information in the infrared image, and the V"_o feature map containing the target thermal radiation information and texture information in the infrared image. In this application, the convolutional layer performs convolution operations to achieve channel dimension mapping transformation. Specifically, a convolutional layer with a kernel size of 1×1 is used to perform convolution operations on the input feature map. That is, for each pixel position in the input feature map, the pixel values ​​of all channels at that pixel position are multiplied by the weight values ​​of the convolutional kernel on that channel and summed to obtain the pixel value of the corresponding channel in the output feature map at that pixel position. The convolution operation only performs linear weighted mapping and channel number transformation in the channel dimension. The number of input channels of the convolutional kernel (the number of input channels of the convolutional kernel is 256) is equal to the number of channels of the input feature map, and the number of output channels of the convolutional kernel (the number of output channels of the convolutional kernel is 128) is equal to the number of channels of the output feature map, thereby achieving channel dimension mapping transformation.

[0054] In this application, the function of the multi-scale decoupled target detection head is as follows: The cross-modal fusion enhanced feature map is input into the target category acquisition branch, bounding box coordinate offset value acquisition branch, and foreground target score acquisition branch of the multi-scale decoupled target detection head, respectively; among them... In the target category acquisition branch, three sequentially connected convolutional layers perform layer-by-layer feature extraction and semantic encoding on the cross-modal fusion enhancement feature map. The Sigmoid activation layer performs a non-linear activation operation on each value in the output feature map of the third convolutional layer to map it to the (0,1) interval, thereby obtaining the target category confidence corresponding to each pixel position. In the bounding box coordinate offset value acquisition branch, three sequentially connected convolutional layers perform layer-by-layer feature extraction and position encoding on the cross-modal fusion enhancement feature map. The third convolutional layer outputs the bounding box coordinate offset value corresponding to each pixel position, which is used for the localization regression of the target bounding box. In the foreground target score acquisition branch, three sequentially connected convolutional layers perform layer-by-layer feature extraction and foreground discrimination encoding on the cross-modal fusion enhancement feature map. The Sigmoid activation layer performs non-linear activation operation on each value in the output feature map of the third convolutional layer to map it to the (0,1) interval, thereby obtaining the foreground target score corresponding to each pixel position. The target category confidence score, bounding box coordinate offset value, and foreground target score are all input into the confidence threshold filtering layer; In this application, the confidence threshold filtering layer uses a preset confidence threshold (in this embodiment, the preset confidence threshold is 0.25) as a benchmark, and filters candidate target boxes based on the target category confidence, bounding box coordinate offset value and foreground target score to obtain the filtered candidate target boxes and their corresponding target categories and coordinate information. The filtered candidate target boxes and their corresponding target categories and coordinate information constitute a candidate target box set. The Non-Maximum Suppression (NMS) layer performs non-maximum suppression deduplication on the candidate bounding box set output by the confidence threshold filtering layer, and outputs the target detection result, which includes the target category, target bounding box and target confidence.

[0055] Step 2) Train the target detection network that includes the visible light image spatial-spectral co-enhancement network to obtain the target detection network model; specifically, this includes the following steps: Step 2-1) Obtain the training set X′ and test set Y′ based on the LLVIP dataset, specifically including the following steps: The LLVIP dataset was used as the original dataset, which contains 15,488 spatiotemporally rigorous registered low-light visible-infrared image pairs.

[0056] The visible light and infrared images in the LLVIP dataset are preprocessed, including the following steps: First, the resolution of all visible light and infrared images is uniformly scaled to 640×640 pixels to obtain scaled images of visible light and infrared images; then, the pixel values ​​of the scaled images of visible light and infrared images are normalized from the range of 0 to 255 to the range of (0,1) to obtain normalized visible light and infrared images; then, the normalized visible light and infrared images are standardized to obtain the standardized visible light and infrared images corresponding to the LLVIP dataset.

[0057] In this application, random data augmentation is performed on the standardized visible light images and standardized infrared images corresponding to the LLVIP dataset. This includes random horizontal flipping, random cropping and scaling (scaling range of 0.8 to 1.2), random brightness and contrast perturbation (brightness perturbation range of -0.2 to 0.2, contrast perturbation range of -0.2 to 0.2), and random Gaussian noise injection (injected Gaussian noise with a mean of 0 and a standard deviation of 0.01) to obtain paired augmented images. The probability of random horizontal flipping is 0.5. The paired augmented images refer to paired visible light images and data-augmented infrared images. The paired augmented images are divided in a 9:1 ratio to obtain a training set X′ and a test set Y′, respectively. The partitioning process maintains the pairing relationship of the bimodal images (i.e., visible light images and infrared images). The training set X′ is used to train the target detection network, and the test set Y′ is used to test the target detection performance.

[0058] Step 2-2): Train the object detection network based on the training set X′ to obtain the object detection network model; including the following steps: The training set X′ is used to read augmented paired images in batches (batch size 16). These augmented paired images are then input into the target detection network. Specifically, the augmented paired images include augmented visible light images and augmented infrared images. The augmented visible light images are input into the visible light image spatial-spectral co-enhancement network, and the augmented infrared images are input into the infrared noise reduction and enhancement module. Forward propagation is then performed to calculate the total loss of the target detection network. Guided by the total loss, backpropagation is performed to update the weight parameters of the target detection network, thus completing one epoch of training. The next batch of augmented paired images is then read for the next epoch of training. This process is repeated until the number of iterations reaches a preset threshold, completing the training of the target detection network and obtaining the trained target detection network model. In this embodiment, the preset threshold is 100.

[0059] In this application, the total loss L of the object detection network total Including image enhancement loss L enhance And target detection loss L det The target detection network L total The total loss is calculated as shown in equation (1); L total =λ enhance × L enhance +λ det ×L det (1) In equation (1), λ enhance and λ det All are balancing weight coefficients, λ enhance Set to 0.2, λ det Set to 1.0; L enhance and L det These represent the image enhancement loss and the object detection loss, respectively. The image enhancement loss L in equation (1) enhance The calculation method is the same as that for pixel-level average absolute error loss; The target detection loss L in equation (1) det Including classification loss L cls Bounding box regression loss L reg And target confidence loss L obj Among them, the target detection loss L det The calculation method is shown in equation (2): L det =λ1×L cls +λ2×L reg +λ3×L obj (2) In equation (2), λ1, λ2, and λ3 are all balance coefficients, and λ1, λ2, and λ3 are set to 0.5, 0.75, and 0.5, respectively;

[0060] The classification loss L in equation (2) cls The calculation method is shown in equation (3): (3) In equation (3), N is the total number of pixel positions in the target category confidence score output by the target category acquisition branch; y i Let y be the ground truth value of the target category at the i-th pixel position. i The value of is 0 or 1, where y i =1 indicates that the i-th pixel position belongs to the target category, y i =0 indicates that the i-th pixel position does not belong to the target category; p ip represents the target category confidence at the i-th pixel location. i The value range of H(y) is (0,1); i ,p i ) represents the binary cross-entropy at the i-th pixel position; In equation (3), the binary cross-entropy H(y) i ,p i The calculation method for ) is shown in equation (4): H(y i ,p i )=-[y i ×log(p i )+(1-y i )×log(1-p i (4) In equation (4), y i y is the true value of the target category. i The value can be 0 or 1; p i For the confidence level of the target category, p i The range of values ​​is (0,1); log() is the logarithmic function with the natural number e as the base; when y i When =1, H(y) i ,p i )=-log(p i ), p i The closer H(y) is to 1, the better. i ,p i The smaller y is; i When =0, H(y) i ,p i )=-log(1-p i ), p i The closer H(y) is to 0, the better. i ,p i The smaller the value of H(yi,pi), the closer the target class confidence is to the target class truth value. The smaller the value of H(yi,pi), the closer the target class confidence is to the target class truth value. The bounding box regression loss L in equation (2) reg The calculation method is shown in equation (5): L reg =1-IoU+ρ 2 (b,bgt) / c 2 +α×v (5) In Equation (5), IoU is the intersection-union ratio of the predicted bounding box and the true bounding box, that is, the ratio of the area of ​​the intersection region of the predicted bounding box and the area of ​​the union region; ρ(b,bgt) is the Euclidean distance between the center point b of the predicted bounding box and the center point bgt of the true bounding box; c is the diagonal length of the smallest bounding rectangle that simultaneously contains the predicted bounding box and the true bounding box; α is the tradeoff coefficient, and v is the aspect ratio consistency measure. In equation (5), α is calculated as shown in equation (6): α=v / [(1-IoU)+v] (6) In equation (6), v is the aspect ratio consistency metric, and IoU is the intersection-union ratio between the predicted bounding box and the true bounding box; In equations (5) and (6), the calculation method of v is shown in equation (7): v=(4 / π 2 )×[arctan(w gt / h gt )-arctan(w / h)] 2 (7) In equation (7), w is the width of the predicted bounding box, h is the height of the predicted bounding box, and w gt h is the width of the actual bounding box. gt is the height of the actual bounding box, arctan() is the arctangent function, and π is pi; The target confidence loss L in equation (2) obj The calculation method is shown in equation (8): (8) In equation (8), N is the total number of pixel positions in the foreground target score output of the foreground target score acquisition branch; Q i Let Q be the ground truth value of the foreground object at the i-th pixel position. i The value of Q is either 0 or 1. i =1 indicates that there is a foreground object at the i-th pixel position, Q i =0 indicates that there is no foreground object at the i-th pixel position; q i The score for the foreground object at the i-th pixel position, q i The value range is (0,1).

[0061] In this embodiment, the training process of the object detection network is based on the PyTorch 2.1 framework, runs on a server equipped with an NVIDIA A800 GPU, and uses the AdamW optimizer to iteratively optimize the network parameters. The initial learning rate is set to 1×10⁻⁶. -4 The weight decay coefficient is set to 5×10. -4The learning rate employs a cosine annealing decay strategy, with a minimum learning rate set to 1×10⁻⁶. -6 .

[0062] Step 3) Preprocess both the visible light image and the infrared image to be processed to obtain paired standardized visible light images and standardized infrared images; input the paired standardized visible light images into the visible light image spatial spectrum co-enhancement network of the target detection network model, and input the paired standardized infrared images into the infrared noise reduction and enhancement module of the target detection network model. After one forward propagation, the target detection result can be obtained; the preprocessing steps for the visible light image and the infrared image to be processed are the same as those for the visible light image and the infrared image in Step 1).

[0063] Example 6: The difference between Example 6 and Example 5 is that: (1) Step 2-1 of Example 6 is: obtain the training set X" and the test set Y" based on the M3FD dataset; wherein, in Example 6, the M3FD dataset is used as the original dataset, and the M3FD dataset contains 4200 pairs of registered visible light-infrared image pairs; the method of preprocessing both the visible light images and infrared images in the M3FD dataset in Example 6 is the same as the method of preprocessing both the visible light images and infrared images in the LLVIP dataset in Example 5; (2) Step 2-2 of Example 6 is: train the target detection network based on the training set X" to obtain the target detection network model; during the training process, the preset iteration number threshold in Example 6 is 200.

[0064] Test 2: To verify the target detection performance of the visible light image spatial-spectral collaborative enhancement network method described in this application, this application specifically conducts target detection tests on the method described in this application based on the test set Y′; specifically: First, the target detection results corresponding to the method described in this application are obtained based on the test set Y′. Then, using the labeled ground truth of the LLVIP dataset corresponding to the test set Y′ as a benchmark, the object detection results corresponding to the method described in this application are quantitatively evaluated using two general object detection evaluation metrics, mAP50 and mAP50-95.

[0065] Wherein, mAP50 represents the average accuracy with an intersection-union ratio (IU) threshold of 0.5, used to measure the ability to detect targets; mAP50-95 represents the average accuracy with an IU threshold ranging from 0.5 to 0.95, used to measure the accuracy of locating target bounding boxes.

[0066] Table 2 shows the test results of mAP50 and mAP50-95 obtained by the method described in this application based on the test set Y′. The method described in this application 96.4 64.5

[0067] As shown in Table 2, the mAP50 test result obtained by the method described in this application is 96.4%, and the mAP50-95 test result obtained by the method described in this application is 64.5%. Since the test set Y′ is obtained based on the LLVIP dataset, which contains 15488 pairs of spatiotemporally strictly registered low-light visible light-infrared image pairs, the test results in Table 2 can demonstrate that the method described in this application also has strong target detection capability in low-light scenes, and the target bounding box localization is also relatively accurate.

[0068] Test 3: To verify the target detection performance of the visible light image spatial-spectral collaborative enhancement network method described in this application, this application specifically tested the method described in this application on the test set Y" for target detection; specifically: First, the method described in this application is used to obtain target detection results based on the test set Y". Then, using the labeled ground truth of the M3FD dataset corresponding to the test set Y" as a benchmark, the object detection results corresponding to the method described in this application are quantitatively evaluated using two general object detection evaluation metrics, mAP50 and mAP50-95.

[0069] Wherein, mAP50 represents the average accuracy with an intersection-union ratio (IU) threshold of 0.5, used to measure the ability to detect targets; mAP50-95 represents the average accuracy with an IU threshold ranging from 0.5 to 0.95, used to measure the accuracy of locating target bounding boxes.

[0070] Table 3 shows the test results of mAP50 and mAP50-95 obtained by the method described in this application based on test set Y". The method described in this application 91.3 60.0

[0071] As shown in Table 3, the method described in this application achieves an mAP50 test result of 91.3% and an mAP50-95 test result of 60.0%. This indicates that, faced with the more complex scenarios and more diverse target categories in the M3FD dataset, the method described in this application has strong target detection capabilities and accurate target bounding box localization.

Claims

1. A method for spatial-spectral synergistic enhancement of visible light images, characterized in that: Includes the following steps: A standardized visible light image is obtained from the visible light image to be processed, and then input into the visible light image spatial spectrum collaborative enhancement network model for forward propagation to obtain an enhanced visible light feature map. The feature map is then subjected to inverse standardization and inverse normalization processing in sequence to obtain an enhanced visible light image. The model was trained using a visible light image spatial-spectral co-enhancement network. In the network, the spatial saliency enhancement link, the frequency domain texture enhancement link, and the global feature enhancement link are all connected to the enhanced feature fusion module. The enhanced feature fusion module is connected to the element-wise addition layer. The spatial saliency enhancement link is connected to the frequency domain texture enhancement link. After performing an identity mapping on the standardized visible light image, it is also input into the element-wise addition layer. The element-wise additive layer outputs an enhanced visible light feature map.

2. The method for spatial-spectral synergistic enhancement of visible light images according to claim 1, characterized in that: The spatial saliency enhancement link consists of the Sobel X operator, the Sobel Y operator, the gradient magnitude calculation layer, the Laplacian operator, the high-frequency detail extraction layer, and the spatial weight map calculation layer, which are connected in sequence.

3. The method for spatial-spectral synergistic enhancement of visible light images according to claim 2, characterized in that: The frequency domain texture enhancement link includes a Fourier transform layer, which is connected to a decoupling layer. The decoupling layer outputs an amplitude spectrum and a phase spectrum. The amplitude spectrum is sent to a mask layer, and the phase spectrum is sent to an inverse Fourier transform layer. The mask layer is connected to an element-wise multiplication unit and an inverse Fourier transform layer in sequence. The spatial weight map calculation layer is connected to the element-wise multiplication unit.

4. The method for spatial-spectral synergistic enhancement of visible light images according to claim 1, characterized in that: The global feature enhancement chain consists of an average pooling layer, a convolutional layer, a normalization layer, and a sigmoid activation layer connected in sequence.

5. The method for spatial-spectral synergistic enhancement of visible light images according to claim 1, characterized in that: The enhanced feature fusion module includes a feature dimension alignment layer, three weight prediction branches, three element-wise multiplication units, an element-wise addition layer, and a convolutional layer. The spatial saliency enhancement link, frequency domain texture enhancement link, and global feature enhancement link are all connected to the feature dimension alignment layer. The feature dimension alignment layer connects to the first to third weight prediction branches. The first weight prediction branch and the feature dimension alignment layer are both connected to the first element-wise multiplication unit. The second weight prediction branch and the feature dimension alignment layer are both connected to the second element-wise multiplication unit. The third weight prediction branch and the feature dimension alignment layer are both connected to the third element-wise multiplication unit. The first to third element-wise multiplication units are all connected to the element-wise addition layer, and the element-wise addition layer is connected to the convolutional layer.

6. The method for spatial-spectral co-enhancing of visible light images according to claim 5, characterized in that: Each weight prediction branch contains a global average pooling layer, a first convolutional layer, a second convolutional layer, and a sigmoid activation layer connected in sequence.

7. A computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method for spatial-spectral co-enhancing visible light image according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for spatial-spectral co-enhancing visible light image according to any one of claims 1 to 6.

9. The application of visible light image spatial-spectral collaborative enhancement network in the field of target detection, characterized by: The visible light image spatial-spectral collaborative enhancement network is the visible light image spatial-spectral collaborative enhancement network in the method for spatial-spectral collaborative enhancement of visible light images as described in claim 1.

10. A method for applying a visible light image spatial-spectral collaborative enhancement network to target detection, characterized in that: Includes the following steps: Step 1) Construct a target detection network that includes a visible light image spatial-spectral co-enhancement network; the visible light image spatial-spectral co-enhancement network is the visible light image spatial-spectral co-enhancement network in the method for spatial-spectral co-enhancement of visible light images as described in claim 1; Step 2) Train the target detection network that includes the visible light image spatial-spectral co-enhancement network to obtain the target detection network model; Step 3) Preprocess both the visible light image and the infrared image to be processed to obtain paired standardized visible light images and standardized infrared images; input the paired standardized visible light images into the visible light image spatial spectrum co-enhancement network of the target detection network model, and input the paired standardized infrared images into the infrared noise reduction and enhancement module of the target detection network model. After one forward propagation, the target detection result can be obtained.

Images