Weak light target reconstruction method based on multi-polarization fusion generative adversarial network

Abstract

The application discloses a weak light target reconstruction method based on a multi-polarization fusion generative adversarial network, comprising the following steps: 1, data collection of low-light polarization images of different scenes; 2, construction of a multi-polarization fusion generative adversarial network; 3, establishment of a loss function between polarization information and target brightness; 4, training of a weak light target reconstruction network by using low-light polarization images, and reconstruction of low-light polarization images by using the trained model. According to the polarization characteristics difference between backscattering light and target signals, the application captures more target information by polarization imaging, and based on the fusion design of multi-polarization images, can effectively extract the related information of different polarization images, has obvious effects in terms of image fidelity, contrast enhancement and color restoration, and has extraordinary significance for the perception and detection of weak targets.

Description

Technical Field

[0001] This invention belongs to the field of low-light sensing and imaging methods, specifically involving a method for reconstructing color targets under low light conditions by using deep learning algorithms for multi-polarization fusion. Background Technology

[0002] In real-world imaging scenarios, the brightness and sharpness of acquired images are often reduced due to variables such as insufficient illumination and short exposure times. These images frequently lose local details and overall contrast, which has a significant negative impact on visual tasks and human subjective experience. Therefore, high-quality imaging under low light conditions is of great research significance and has broad application prospects in numerous applications such as autonomous driving, mobile detection, and environmental monitoring. Polarized images contain more target information than visible light images, including the texture and shape of low-contrast objects. To improve target information, multi-polarization information can be fused, thereby enhancing image quality and texture. In contrast, a single detector captures less target information under low-light conditions, while the increased information dimensionality provides richer features for target reconstruction.

[0003] Visualizing images in low-light environments typically requires enhancing illumination. However, traditional methods that rely solely on increasing brightness often fail to achieve satisfactory results due to a lack of detail, color imbalance, and poor contrast. With the increasing processing power of computers and the rapid development of computational imaging technology, convolutional neural networks (CNNs) have found widespread applications in various fields. CNNs have gradually become an essential tool for solving imaging problems, and the rapid development of deep learning (DL)-based network models in computer vision has led to the belief that deep learning-based methods can overcome the limitations of traditional methods and have achieved significant success in low-light enhancement tasks. These methods far surpass traditional methods in terms of accuracy, speed, fidelity, contrast enhancement, and color reproduction. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing a low-light target reconstruction method based on multi-polarization fusion generative adversarial networks. This method aims to overcome the challenge of detecting the brightness and clarity of targets under low-light conditions, reconstruct the brightness and texture of weak targets, and overcome the limitations of perception imaging based on the structure of objects made of the same material, as well as target perception imaging across structures and materials. This enables the generalized reconstruction of target information that has been severely damaged under real low-light conditions.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The present invention provides a method for reconstructing weak light targets based on multi-polarization fusion generative adversarial networks, characterized by the following steps: Step 1: Obtain the polarization direction in the nth scene. They are respectively A set of original polarization images This yields original polarization images for N scenes, forming a low-light polarization image dataset; among them, This indicates that the polarization direction in the nth scene is... The original polarization image, ; Step 2: Calculate the original polarization image of the nth scene. Stokes vector ,in, This represents the total light intensity image of the nth scene. This represents the difference in polarization intensity in the horizontal and vertical directions of the original polarization image of the nth scene. This represents the difference in polarization intensity between 45 degrees and 135 degrees in the original polarization image of the nth scene; Step 3: Construct a multi-polarization fusion generative adversarial network, which consists of two identical generators and a discriminator; each generator consists of an encoder, a fusion module, a gradient extraction module, and a decoder. Step 3.1, the first generator pair Processing is performed to obtain the reconstructed image of the nth scene. ; Step 3.2, The image is input into a second generator for processing to obtain the reconstructed image of the nth scene. ;in, Represents the nth scene The reconstructed image, Represents the nth scene The reconstructed image, Represents the nth scene The reconstructed image; Step 3.3, the discriminator pair Process the data and output the true / false probabilities. ; Step 4: Construct the total loss function of the multi-polarization fusion generative adversarial network. ; Step 5: Train the multi-polarization fusion generative adversarial network using gradient descent and minimize the total loss function. The network parameters are updated until the maximum number of training iterations is reached, thereby obtaining the optimal multi-polarization fusion generative adversarial model after training. This model is then used to process the input low-light polarization image to obtain the reconstructed image, thus achieving the reconstruction of weak-light targets.

[0006] The weak light target reconstruction method based on multi-polarization fusion generative adversarial network described in this invention is characterized in that the encoder in step 3 is composed of three basic units STL based on the transformer model; wherein each basic unit sequentially includes two multilayer perceptrons (MLPs), one WMSA unit and one SWMSA unit; each multilayer perceptron and WMSA unit is connected to an LN layer, and the input and output of each unit in the basic unit are connected by residual connections; The first basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ; The second basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ; The third basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ,in, These represent the height, width, and number of channels of the self-attention feature map, respectively. Self-attention feature map for the nth scene Perform a concatenation operation to obtain the total self-attention feature map of the nth scene. ,in, This represents the total number of channels in the self-attention polarization feature map; The fusion module in the first generator After processing, the fused polarization feature map of the nth scene is obtained. ,in, The number of channels in the polarization feature map; The gradient extraction module in the first generator includes: a first gradient residual branch and a second gradient residual branch; The first gradient residual branch in the first generator uses the Sobel operator to fuse the polarization feature map. Processing is performed to obtain Strong texture of edge features ; Will and After element-wise addition, the intermediate polarization characteristic map is obtained. ,in, This indicates the number of channels in the intermediate polarization feature map; The second gradient residual branch in the first generator uses the Laplacian operator pair. Processing is performed to obtain weak texture of edge features ; Will and After adding the elements, we obtain the refined feature map of the nth scene. ,in, The number of channels in the refined feature map; The decoder in the first generator is... The layers consist of convolutional layers with LReLU, and... After processing, the reconstructed image of the nth scene is output. ;in, This indicates the number of channels in the reconstructed image.

[0007] In step 4, the total loss function of the multi-polarization fusion generative adversarial network is constructed using equation (1). : (1) In equation (1), It is to counteract losses, and is obtained from equation (2). It is the cyclic consistency loss, and is obtained from equation (3). It is the loss of identity invariance, and is obtained from equation (4). It is the perceived loss, and is obtained from equation (7). It is the polarization loss, and is obtained from equation (8). , , and These represent the four hyperparameters that balance each loss term; (2) In equation (2), Expressing expectations; (3) In equation (3), yes The true label, yes For any i-th image, express For any i-th reconstructed image, It is an L2 canonical operation; H represents the height of the image, and W represents the width of the image; (4) In equation (4), and These are weighting coefficients. The strength loss is represented by equation (5). The loss of color consistency is represented by equation (6); (5) (6) In equations (5) and (6), C It is the number of channels. Is it pixels? R , G and B The calculated value of discrete cosine similarity in the channel. M Indicates the number of pixels; In the nth scenario Real tag images in the channel The m-th pixel, In the nth scenario The m-th pixel of the image reconstructed in the channel; (7) In equation (7), VGG express VGG-19 General model; (8) In equation (8), This indicates polarization loss.

[0008] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the low-light target reconstruction method, and the processor is configured to execute the program stored in the memory.

[0009] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the low-light target reconstruction method.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a generative adversarial network based on multi-polarization fusion and uses labeled data to supervise deep neural network learning, thereby obtaining a robust weak target reconstruction model. This solves the limitations of high exposure and low contrast in statistical models using physical prior models and mathematical transformation algorithms, and improves the effect of weak target color reconstruction.

[0011] 2. The multi-polarization fusion-based generative adversarial network constructed in this invention introduces multi-dimensional polarization information. The different reflectivities of each target object lead to polarization differences throughout the image, resulting in richer target information in the polarization data. This characteristic solves the problem of missing information in the reconstruction of weakly colored targets.

[0012] 3. The multi-polarization fusion-based generative adversarial network constructed in this invention introduces polarization loss to improve the stability of network training. In the training process of existing generative adversarial networks, there is an instability in the results due to adversarial game, resulting in varying degrees of artifacts and color loss in the reconstructed results. This invention effectively solves this problem. Attached Figure Description

[0013] Figure 1 This is the experimental flowchart of the present invention.

[0014] Figure 2 This is a schematic diagram of the generative adversarial network based on multi-polarization fusion in this invention.

[0015] Figure 3 This is a schematic diagram of different modules in the multi-polarization fusion generative adversarial network of this invention.

[0016] Figure 4 This diagram compares the present invention with other methods for targets with different structures and materials under simulated low-light conditions.

[0017] Figure 5 This diagram compares the present invention with other methods for targets with different structures and materials under real low-light conditions.

[0018] Figure 6 This describes the target reconstruction under different low-light conditions in a real low-light environment. Detailed Implementation

[0019] In this embodiment, a low-light target reconstruction method based on multi-polarization fusion generative adversarial networks utilizes Stokes vectors to highlight the polarization characteristics of materials, fusing different polarization information to improve target information, thereby increasing the information dimension and ensuring richer features. This promotes the neural network to learn the relationship between polarization information and target brightness using its powerful data mining and fitting capabilities, achieving the reconstruction of targets with different structures and materials under low-light conditions. Specifically, the method includes the following steps: Step 1: Obtain the polarization direction in the nth scene. They are respectively A set of original polarization images This yields original polarization images for N scenes, forming a low-light polarization image dataset; among them, This indicates that the polarization direction in the nth scene is... The original polarization image, ; In this embodiment, a low-light polarized image dataset was captured using a focal plane polarization camera (LUCID, PHX055S-PC), which contains N=100 polarized image scenes. Each image has a width of 1224 and a height of 1024.

[0020] Step 2: Calculate the original polarization image of the nth scene. Stokes vector ,in, This represents the total light intensity image of the nth scene. This represents the difference in polarization intensity in the horizontal and vertical directions of the original polarization image of the nth scene. This represents the difference in polarization intensity between 45 degrees and 135 degrees in the original polarization image of the nth scene; In this embodiment, four original polarization angle images demonstrate the differences in the scene, which directly affect the Stokes vector, a crucial parameter of polarization. The resulting color intensity image has three channels and is the same size as the original polarization angle images. The polarization image provides more information about the target, encompassing the texture and shape of low-contrast objects.

[0021] Step 3: Construct a multi-polarization fusion generative adversarial network, which consists of two identical generators and a discriminator; each generator consists of an encoder, a fusion module, a gradient extraction module, and a decoder. Step 3.1: The encoder consists of three basic STL units based on the transformer model; each basic unit contains two multilayer perceptrons (MLPs), one WMSA unit, and one SWMSA unit in sequence; each MLP and WMSA unit is connected to an LN layer, and the input and output of each unit in the basic unit are connected by residuals. In this embodiment, the encoder corresponds to Figure 3 The encoder in the middle; the STL corresponding backbone network Figure 2 In , and Among them, multilayer perceptron (MLP), WMSA and SWMSA respectively in Figure 3 One-to-one correspondence.

[0022] The first basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ; The second basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ; The third basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ,in, These represent the height, width, and number of channels of the self-attention feature map, respectively. In this embodiment, the basic unit STL will have a size of Input Resize to The tensor, and divide it into groups of size Using non-overlapping windows, calculate the self-attention features within each window. M=8, H=W=128, V=32.

[0023] Self-attention feature map for the nth scene Perform a concatenation operation to obtain the total self-attention feature map of the nth scene. ,in, This represents the total number of channels in the self-attention polarization feature map; in this embodiment, =96.

[0024] Step 3.2, the fusion module in the first generator... After processing, the fused polarization feature map of the nth scene is obtained. ,in, The number of channels in the polarization feature map; In this embodiment, the self-attention feature map Polarization feature map after processing by the fusion module Width, height, and number of channels are all 128, 128, and 64, respectively.

[0025] The fusion module consists of The Resblock basic unit is composed of layers, and the Resblock basic unit includes: Each convolutional kernel is Mainstream 2D ​​convolutional layers and Each convolutional kernel is The branch of the two-dimensional convolutional layer; In this embodiment, the fusion module corresponds to Figure 2 The fusion module in Resblock. The basic unit of Resblock corresponds to... Figure 3 Resblock in the middle. =3, =3, =1; =3, =1.

[0026] Step 3.3: The gradient extraction module in the first generator includes: the first gradient residual branch and the second gradient residual branch; In this embodiment, the gradient extraction module corresponds to Figure 3 Gradient extraction block (GEB) in the middle.

[0027] The first gradient residual branch in the first generator uses the Sobel operator to fuse the polarization feature map. Processing is performed to obtain Strong texture of edge features ; Will and After element-wise addition, the intermediate polarization characteristic map is obtained. ,in, This indicates the number of channels in the intermediate polarization feature map.

[0028] The second gradient residual branch in the first generator uses the Laplacian operator pair. Processing is performed to obtain weak texture of edge features ; Will and After adding the elements, we obtain the refined feature map of the nth scene. ,in, surface In this embodiment, the Sobel operator and the Laplacian operator correspond to Figure 3 Sobel operator and Laplacian operator in the text; =3, =64, =64.

[0029] Step 3.4, the decoder in the first generator is... The layers consist of convolutional layers with LReLU, and... After processing, the reconstructed feature map of the nth scene is output. ;in, This represents the number of channels in the reconstructed feature map; in this embodiment, the decoder corresponds to... Figure 3 The decoder in the middle; =4, =3, =3.

[0030] Step 3.5, The image is input into a second generator for processing to obtain the reconstructed image of the nth scene. ;in, Represents the nth scene The reconstructed image, Represents the nth scene The reconstructed image, Represents the nth scene The reconstructed image.

[0031] Step 3.6, Discriminator Pair Process the data and output the true / false probabilities. ; In this embodiment, the discriminator is constructed by five convolutional layers with LReLU, and the kernel size is a 3×3 two-dimensional convolution.

[0032] Step 4: Construct the total loss function of the multi-polarization fusion generative adversarial network using equation (1). : (1) In equation (1), It is to counteract losses, and is obtained from equation (2). It is the cyclic consistency loss, and is obtained from equation (3). It is the loss of identity invariance, and is obtained from equation (4). It is the perceived loss, and is obtained from equation (7). It is the polarization loss, and is obtained from equation (8). , , and These represent the four hyperparameters that balance each loss term; in this embodiment, the hyperparameters are... , , and The values ​​are 10, 1, 40 and 8 × 10, respectively. e -6 .

[0033] (2) In equation (2), Expressing expectations; (3) In equation (3), yes The true label, yes For any i-th image, express For any i-th reconstructed image, It is an L2 normalized operation; H represents the height of the image, and W represents the width of the image.

[0034] (4) In equation (4), and These are weighting coefficients. The strength loss is represented by equation (5). The loss of color consistency is represented by equation (6); in this embodiment, and Set them to 20 and 25 respectively.

[0035] (5) (6) In equations (5) and (6), C It is the number of channels. Is it pixels? R , G and B The calculated value of discrete cosine similarity in the channel, where m represents the number of pixels. Representing an image R , G and B Channel, In the nth scenario Real tag images in the channel The m-th pixel, In the nth scenario The m-th pixel of the reconstructed image in the channel; in this embodiment, C=3, M=128×128.

[0036] (7) In equation (7), VGG express VGG-19 A general model; in this embodiment, VGG is a pre-trained VGG-19 model. Only the last two layers of the VGG-19 network are selected for deep feature extraction to compute... and Feature similarity between them.

[0037] (8) In equation (8), This indicates polarization loss.

[0038] Step 5: Train the multi-polarization fusion generative adversarial network using gradient descent and minimize the total loss function. The network parameters are updated until the maximum number of training iterations is reached, thus obtaining the optimal multi-polarization fusion generative adversarial model after training, which is used to process input low-light polarization images. After processing, the reconstructed version is obtained. Images are used to reconstruct targets in low light.

[0039] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0040] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0041] Figure 4 and Figure 5 This document compares the results of this invention with other current methods for weak target image reconstruction. Specifically, Ours represents the weak light target reconstruction method based on multi-polarization fusion generative adversarial networks; LDR is a contrast enhancement method based on two-dimensional histogram hierarchical difference representation; SPI is a weighted variational model method for simultaneously estimating reflectance and illuminance; NPE is a naturalness enhancement algorithm for non-uniformly lit images; RetinexNet is a depth retinal decomposition method for weak light enhancement; RUAS is a Retinex heuristic expansion and cooperative prior architecture search method for weak light image enhancement; Zero-DCE is a zero-reference depth curve estimation method for weak light image enhancement; TBEFN is a bi-branch exposure fusion network method for weak light image enhancement; EnlightenGAN is a depth light enhancement method without paired supervision; and LTC is a method for enhancing low-light video effects by learning temporal consistency from a single image. This invention outperforms these methods in image fidelity, contrast enhancement, and color reproduction.

[0042] Table 1 Comparison of experimental results for different reconstruction methods

[0043] Table 1 compares the experimental results of this invention with those of different reconstruction methods. PSNR is Peak Signal-to-Noise Ratio, SSIM is Structural Similarity, UQI is Universal Image Quality Index, VIF is Visual Information Fidelity, and SAM is Spectral Angle Mapper. Among these indicators, a smaller SAM indicates less interference in the reconstruction result and better reconstruction performance. The other indicators are positively correlated with reconstruction performance. Table 1 shows that, under the current indicators, the overall performance of this invention is superior to the aforementioned image reconstruction methods.

Claims

1. A method for reconstructing weak-light targets based on multi-polarization fusion generative adversarial networks, characterized in that, The procedure is as follows: Step 1: Obtain the polarization direction in the nth scene. They are respectively A set of original polarization images This yields original polarization images for N scenes, forming a low-light polarization image dataset; among them, This indicates that the polarization direction in the nth scene is... The original polarization image, ; Step 2: Calculate the original polarization image of the nth scene. Stokes vector ,in, This represents the total light intensity image of the nth scene. This represents the difference in polarization intensity in the horizontal and vertical directions of the original polarization image of the nth scene. This represents the difference in polarization intensity between 45 degrees and 135 degrees in the original polarization image of the nth scene; Step 3: Construct a multi-polarization fusion generative adversarial network, which consists of two identical generators and a discriminator; each generator consists of an encoder, a fusion module, a gradient extraction module, and a decoder. Step 3.1, the first generator pair Processing is performed to obtain the reconstructed image of the nth scene. ; Step 3.2, The image is processed in the second generator to obtain the reconstructed image of the nth scene. ;in, Represents the nth scene The reconstructed image, Represents the nth scene The reconstructed image, Represents the nth scene The reconstructed image; Step 3.3, the discriminator pair Process the data and output the true / false probabilities. ; Step 4: Construct the total loss function of the multi-polarization fusion generative adversarial network. ; Step 5: Train the multi-polarization fusion generative adversarial network using gradient descent and minimize the total loss function. The network parameters are updated until the maximum number of training iterations is reached, thereby obtaining the optimal multi-polarization fusion generative adversarial model after training. This model is then used to process the input low-light polarization image to obtain the reconstructed image, thus achieving the reconstruction of weak-light targets.

2. The method for reconstructing weak-light targets based on multi-polarization fusion generative adversarial networks according to claim 1, characterized in that, The encoder in step 3 consists of three basic units STL based on the transformer model; each basic unit contains two multilayer perceptrons (MLPs), one WMSA unit, and one SWMSA unit; each MLP and WMSA unit is connected to an LN layer, and the input and output of each unit in the basic unit are connected using residual connections. The first basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ; The second basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ; The third basic unit STL pair in the encoder of the first generator Process and output. Self-attention feature map ,in, These represent the height, width, and number of channels of the self-attention feature map, respectively. Self-attention feature map for the nth scene Perform a concatenation operation to obtain the total self-attention feature map of the nth scene. ,in, This represents the total number of channels in the self-attention polarization feature map; The fusion module in the first generator After processing, the fused polarization feature map of the nth scene is obtained. ,in, The number of channels in the polarization feature map; The gradient extraction module in the first generator includes: a first gradient residual branch and a second gradient residual branch; The first gradient residual branch in the first generator uses the Sobel operator to fuse the polarization feature map. Processing is performed to obtain Strong texture of edge features ; Will and After element-wise addition, the intermediate polarization characteristic map is obtained. ,in, This indicates the number of channels in the intermediate polarization feature map; The second gradient residual branch in the first generator uses the Laplacian operator pair. Processing is performed to obtain weak texture of edge features ; Will and After adding the elements, we obtain the refined feature map of the nth scene. ,in, The number of channels in the refined feature map; The decoder in the first generator is... The layers consist of convolutional layers with LReLU, and... After processing, the reconstructed image of the nth scene is output. ;in, This indicates the number of channels in the reconstructed image.

3. The method for reconstructing weak-light targets based on multi-polarization fusion generative adversarial networks according to claim 2, characterized in that, In step 4, the total loss function of the multi-polarization fusion generative adversarial network is constructed using equation (1). : (1) In equation (1), It is to counteract losses, and is obtained from equation (2). It is the cyclic consistency loss, and is obtained from equation (3). It is the loss of identity invariance, and is obtained from equation (4). It is the perceived loss, and is obtained from equation (7). It is the polarization loss, and is obtained from equation (8). , , and These represent the four hyperparameters that balance each loss term; (2) In equation (2), Expressing expectations; (3) In equation (3), yes The true label, yes For any i-th image, express For any i-th reconstructed image, It is an L2 canonical operation; H represents the height of the image, and W represents the width of the image; (4) In equation (4), and These are weighting coefficients. The strength loss is represented by equation (5). The loss of color consistency is represented by equation (6); (5) (6) In equations (5) and (6), C It is the number of channels. Is it pixels? R , G and B The calculated value of discrete cosine similarity in the channel. M Indicates the number of pixels; In the nth scenario Real tag images in the channel The m-th pixel, In the nth scenario The m-th pixel of the reconstructed image in the channel; (7) In equation (7), VGG express VGG-19 General model; (8) In equation (8), This indicates polarization loss.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing any of the low-light target reconstruction methods of claims 1-3, and the processor is configured to execute the programs stored in the memory.

5. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the low-light target reconstruction method according to any one of claims 1-3.

Images