Enhanced information generation method and system for low-light night vision goggles

Abstract

The invention discloses an enhanced information generation method and system of a low-light-level night vision goggle, and relates to the field of image data processing, and the method comprises the steps: collecting a visible light image of a low-light-level night vision scene through the low-light-level night vision goggle, and collecting a depth image of the low-light-level night vision scene through an infrared depth sensor; preprocessing the collected visible light image and the depth image; according to the preprocessed depth image, constructing a three-dimensional model of a low-light night vision scene through a three-dimensional reconstruction algorithm; fusing the three-dimensional model and the acquired visible light image by using a GAN (Generative Adversarial Network) to generate an enhanced night vision image; aiming at geometric distortion of a scene in a low-light night vision enhanced image in the prior art, a depth image of the low-light night vision goggles scene is collected, and a three-dimensional geometric model of the scene is constructed through the depth image. And fusing the three-dimensional model and the visible light image by using the generative adversarial network, generating an enhanced night vision image and the like, retaining a scene geometric structure, and generating a more vivid night vision image.

Description

Technical Field

[0001] The present application relates to the field of image data processing, and in particular to a method and system for generating enhanced information for low-light-level night vision goggles. Background Art

[0002] With the continuous development of night vision imaging technology, low-light-level night vision goggles have been widely used in military reconnaissance, security monitoring and other fields. By enhancing weak visible light and near-infrared light, low-light-level night vision goggles can perform imaging under low-light conditions, expanding the observation ability of the human eye. However, due to the weak low-light imaging signal and low signal-to-noise ratio, the direct imaging quality is poor, and it is urgent to enhance the low-light image to improve the visual perception effect.

[0003] Traditional low-light-level night vision enhancement methods mainly optimize image contrast, brightness, clarity and other characteristics. Among them, histogram equalization improves image contrast by stretching the pixel value distribution; Retinex theory achieves image dynamic range compression and color constancy correction by estimating the illumination component and reflection component; multi-scale fusion method enhances image texture details by fusing features such as multi-scale gradients of images. The above methods improve the visual quality of low-light images to a certain extent, but do not fully utilize the prior information of the imaging scene, making it difficult to restore the scene geometry, and the enhanced image lacks realism.

[0004] In recent years, the rise of deep learning technology has brought new opportunities for low-light night vision image enhancement. Convolutional neural networks can directly model image quality enhancement through end-to-end training, and generative adversarial networks can learn the probability distribution of real images and generate realistic enhanced images. However, most existing deep learning-based methods regard low-light image enhancement as a mapping from two-dimensional images to images, ignoring the three-dimensional structural information of the scene. The low-light imaging process easily introduces optical distortion and noise, resulting in inaccurate estimated depth information and geometric distortion of the enhanced image. In addition, the network structure and loss function design of existing methods are not sophisticated enough, making it difficult to well constrain the local details and global consistency of the enhanced image. Summary of the invention

[0005] In response to the problem of scene geometric distortion in low-light-level night vision enhanced images existing in the prior art, the present application provides a method and system for generating enhanced information for low-light-level night vision goggles, which fully utilizes the depth information of the scene and generative adversarial networks, retains the scene geometric structure, and generates more realistic night vision images.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] One aspect of the present application provides a method for generating enhanced information for a low-light-level night vision goggle, comprising: using a low-light-level night vision goggle to collect a visible light image of a low-light-level night vision scene, and using an infrared depth sensor to collect a depth image of the low-light-level night vision scene; preprocessing the collected visible light image and depth image; constructing a three-dimensional model of the low-light-level night vision scene through a three-dimensional reconstruction algorithm based on the preprocessed depth image; and using a generative adversarial network (GAN) to fuse the three-dimensional model with the collected visible light image to generate an enhanced night vision image.

[0008] Among them, low-light-level night vision goggles are a type of optoelectronic device used for imaging under low-light conditions, which can amplify weak visible light and near-infrared light to achieve nighttime observation. Its core component is a low-light-level image intensifier, which mainly includes a photocathode, a microchannel plate, a fluorescent screen, etc. In this application, low-light-level night vision goggles can obtain scene images at night and in low-light environments, but the image quality is usually not high. This application scheme uses low-light-level night vision goggles to collect visible light images as input for night vision enhancement, and introduces an infrared depth sensor to collect depth images of the scene to provide depth information for subsequent three-dimensional reconstruction. Post-processing and enhancement of low-light-level night vision goggles imaging through algorithms can further improve their imaging performance and user experience.

[0009] Among them, low-light-level night vision scenes refer to scenes observed with low-light-level night vision goggles at night or in low-light environments. Such scenes usually have extremely weak light that is difficult to observe with the naked eye, but there is a small amount of visible light and near-infrared light. Specifically, the visible light illumination in the scene is usually less than 0.1 lux, which is a low-light environment.

[0010] The collected visible light image is preprocessed, including: dividing the visible light image collected by the low-light night vision goggles into blocks to obtain multiple local sub-images; according to each local sub-image, extracting a local area pixel by pixel on the sub-image by means of a sliding window; for each extracted local area, counting the variance of the pixel values ​​in the local area as the local noise variance; according to the local noise variance, setting the parameters of the median filter; performing median filtering on the corresponding local area using the median filter after setting the parameters; performing median filtering on all local areas to obtain a denoised visible light image; performing Retinex decomposition on the denoised visible light image to obtain an illumination map reflecting the brightness distribution of the image and a reflectivity map reflecting the image; using a stretching algorithm to expand the range of the illumination map to enhance the brightness contrast of the image to obtain an enhanced illumination map; using a guided filtering algorithm to smooth the reflectivity map to obtain a smoothed reflectivity map; pixel-wise multiplying the enhanced illumination map and the smoothed reflectivity map to obtain a preprocessed visible light image.

[0011] Furthermore, the collected depth image is preprocessed, including: using the internal and external parameters of the infrared depth sensor that have been calibrated in advance to correct the distortion of the depth image, and correcting the three-dimensional coordinates of each pixel in the depth image through radial and tangential distortion coefficients to eliminate the nonlinear distortion introduced by the imaging process, thereby obtaining a distortion-corrected depth image; using the corrected depth image as the input image and the preprocessed visible light image as the guide image, and using the guided filtering algorithm to smooth the depth image to obtain a smoothed depth image; using the smoothed depth image and the preprocessed visible light image as input, and using the joint bilateral filtering algorithm to fill the holes in the depth image, and using the geometric distance between the depth images and the color similarity constraints between the visible light images to fill the hole areas in the depth image to obtain a preprocessed depth image.

[0012] Furthermore, the depth image is smoothed using a guided filtering algorithm to obtain a smoothed depth image, including: using the guided filtering algorithm to fit the linear transformation coefficient between the input depth image and the guided image through the least squares method; for areas where the linear transformation coefficient is greater than a threshold, filtering is performed using a mean filter with a filter window greater than the threshold, and conversely filtering is performed using a mean filter with a filter window less than the threshold to obtain a filtered depth image; calculating the gradient amplitude of the guided image, and setting weights for different areas of the depth image according to the gradient amplitude; and performing weighted fusion based on the set weights and the filtered depth image to obtain a smoothed depth image.

[0013] Furthermore, a three-dimensional model of the low-light-level night vision scene is constructed through a three-dimensional reconstruction algorithm, including: taking the preprocessed depth image as input, dividing the depth image into uniform grids, each grid corresponding to a voxel in the three-dimensional model; calculating the three-dimensional spatial coordinates of each voxel according to the depth value of the depth image pixel corresponding to each voxel, and obtaining an initial three-dimensional model; using a triangular mesh algorithm to simplify the mesh of the initial three-dimensional model; using a Poisson reconstruction algorithm to reconstruct the surface of the three-dimensional model after the mesh simplification, constraining the surface shape of the three-dimensional model through the gradient information of the depth image, and obtaining a reconstructed three-dimensional model; mapping the preprocessed visible light image to the reconstructed three-dimensional model surface, adding the texture information of the three-dimensional model through the visible light image, and obtaining a three-dimensional model of the low-light-level night vision scene.

[0014] Furthermore, a generative adversarial network (GAN) is used to fuse the three-dimensional model with the collected visible light image to generate an enhanced night vision image, including: inputting the three-dimensional model of the low-light night vision scene as a condition into the generator of the generative adversarial network (GAN), and the generator generates an initial night vision image by decoding the geometric structure and texture mapping of the three-dimensional model; inputting the collected visible light image into the discriminator of the generative adversarial network (GAN) as a real sample, and the discriminator obtains the characteristics of the visible light image by encoding the texture and noise distribution of the visible light image; in the adversarial learning of the generator and the discriminator, the generator is constrained by the three-dimensional model, and the discrimination result of the discriminator on the generated image is used as the optimization target, and the generator parameters are updated through back propagation to generate an enhanced night vision image; a reconstruction loss term is set in the loss function of the generator, and the difference in pixel space between the night vision image generated by the generator at the current iteration step and the initial night vision image is constrained. The generator learns the geometric structure and illumination prior information of the three-dimensional model by setting a perceptual loss term in the loss function of the generator, extracts the multi-scale convolution features of the night vision image generated by the generator in the current iteration step and the collected visible light image through a pre-trained convolutional neural network, and calculates the L1 distance of the two image features in the feature space composed of the convolution features, so that the generator learns the texture and noise distribution of the visible light image; the generator sets an adversarial loss term in the loss function of the generator, inputs the night vision image generated by the generator in the current iteration step into the discriminator for true and false discrimination, and uses the discrimination result of the discriminator as the loss of the generator; through the alternating training of the generator and the discriminator, the reconstruction loss, perceptual loss and adversarial loss of the generator are minimized, and at the same time, the binary classification accuracy of the discriminator for the visible light image and the night vision image generated by the generator in the current iteration step is maximized, so that the parameters of the generator and the discriminator reach Nash equilibrium, and the generator after adversarial learning is obtained.

[0015] Furthermore, the reconstruction loss term is expressed as follows: Among them, G represents the generator, z represents the random noise vector, C represents the constructed three-dimensional model of the low-light night vision scene, and I init represents the initial night vision image obtained by decoding C into the generator G, and ||*||1 represents the L1 norm. represents the gradient operator, ▽G(z,C) represents the gradient map of the night vision image generated by the generator, represents the gradient map of the initial night vision image, λ g is the weight coefficient of the gradient loss, which is used to balance the ratio of pixel loss and gradient loss.

[0016] Among them, the constructed three-dimensional model of the low-light night vision scene participates in the process of calculating the reconstruction loss: according to the boundary and internal structure of the three-dimensional model, each voxel is marked as occupied (1) or idle (0). Occupied voxels represent the surface or interior of the model, and idle voxels represent the background. The voxelized three-dimensional model is denoted as V(C). A three-dimensional convolutional neural network is used to extract and encode the voxel model V(C). The network input is a binary voxel occupancygrid, and the high-level semantic features of the model are extracted through multiple layers of three-dimensional convolution and downsampling. Let the encoded feature vector be f(V(C)), which represents the geometric structure information of the entire three-dimensional model in the form of a one-dimensional vector. The voxel code f(V(C)) and the noise vector z are input into the generator G, and the initial night vision image I is generated through the three-dimensional deconvolution network. init The deconvolution network consists of multiple layers of 3D deconvolution and upsampling, which decodes the low-dimensional feature vector into a high-resolution image. During the decoding process, the feature map is gradually expanded and combined with the encoded geometric information f(V(C)) to make the generated image close to the geometric structure of the real scene. The reconstruction loss is calculated based on the generated initial night vision image I_init and the original 3D model C. The reconstruction loss includes pixel loss and gradient loss.

[0017] The generated night vision image I init Compare with the real image in pixel space. The L1 norm can be used to calculate the sum of the absolute differences between the corresponding pixels of the two images, that is, ||G(z,f(V(C)))-I init ||1. This loss term encourages the generated image to be close to the real image at the pixel level. init The gradient map is calculated for the real image and the real image respectively, and then the difference between the two gradient maps is compared. The gradient map can be obtained by pixel finite difference, such as Then use the L1 norm to measure the difference between the two gradient maps, that is, Gradient loss can capture structural information in the image, such as edges and contours, and encourage the generated image to be consistent with the real image in local structure. The pixel loss and gradient loss are weighted and summed to obtain the final reconstruction loss L recon . The reconstruction loss L recon As one of the optimization objectives of the generator G, it guides the training of the generator together with other loss functions (such as perceptual loss and adversarial loss). Through back-propagation and gradient descent, the parameters of the generator are adjusted so that it can generate night vision enhanced images with realistic geometric structures.

[0018] Specifically, based on the pixel-level reconstruction loss, a gradient domain loss term is introduced. The gradient reflects the edge and texture structure information of the image. By constraining the difference between the generated image and the initial image in the gradient domain, the generator can better learn the scene structure and geometric details represented by the 3D model and improve the reconstruction quality. At the same time, the gradient loss is more robust to illumination changes and noise interference, which helps to improve the clarity and realism of the generated image. g The value of can be optimized according to experiments to balance pixel loss and gradient loss, ensuring both the global reconstruction effect and the local detail expression.

[0019] Furthermore, the perceptual loss term is expressed as follows:

[0020] Where F represents a pre-trained convolutional neural network, which is used to extract multi-scale convolutional features of images. i Represents the feature map extracted by the i-th convolutional layer, i∈1,2,....,N, where N is the total number of convolutional layers. real represents the collected visible light image, and G(z,C) represents the night vision image generated by the generator. i is the weight coefficient of the perceptual loss of the i-th layer, which is used to balance the importance of features in different convolutional layers; L style Represents the style loss, which is used to measure the difference between the generated image and the real image in terms of texture, structure and other style features. Its expression is: Among them, G j Represents the Gram matrix calculated from the convolutional feature map, which is used to capture the correlation and statistical information between feature maps. j∈1,2,.....,M, where M is the number of convolutional layers used to calculate the Gram matrix. ||*||1 represents the L1 norm. λ s is the weight coefficient of style loss.

[0021] Among them, the Gram matrix is ​​a matrix used to measure the correlation and statistical information between the feature maps of the convolutional neural network. It characterizes the high-order statistics of the features by calculating the inner product between the feature maps, reflecting the style characteristics of the image in terms of texture and structure. The style loss L_style uses the Gram matrix to measure the difference in style characteristics between the generated image G(z, C) and the real image I_real. By minimizing the style loss, the generated image is encouraged to be similar to the real image in terms of texture, structure, etc., thereby improving the visual realism of the generated image. At the same time, by calculating the Gram matrix on multiple convolutional layers, the style features of different scales and abstract levels are considered to achieve multi-level characterization and matching of image style. It is introduced into the generative adversarial network for night vision image enhancement, and the realism of the generated image is constrained by the style loss, thereby improving the visual quality and perceptual experience of the enhanced result.

[0022] Specifically, based on the multi-scale perceptual loss, a style loss term is introduced. By constraining the difference between the generated image and the real image in the feature space of different convolutional layers, the generator not only learns the content information of the real image, such as objects and scene structures, but also learns the style information of the real image, such as texture, color, and local structure. The multi-scale feature representation helps capture the semantic information of the image at different levels of abstraction, while the style loss helps improve the realism and consistency of the generated image. i and λ s Experimental tuning can be used to balance content loss and style loss, ensuring the perceptual similarity between the generated image and the real image while taking into account the artistic style characteristics of low-light night vision images.

[0023] Furthermore, the adversarial loss term is expressed as follows:

[0024] Where D represents the discriminator, G represents the generator, z represents the random noise vector, and C represents the constructed three-dimensional model of the low-light night vision scene. z represents the expectation of the random noise vector z, D(G(z,C)) represents the authenticity score obtained by inputting the night vision image generated by the generator into the discriminator, and the value range is [0, 1]. The larger the value, the more realistic the discriminator thinks the generated image is.

[0025] The second term is the gradient penalty term, which is used to constrain the gradient norm of the discriminator to be close to 1, thereby improving training stability and generating image quality. Represents the interpolation point between the real image and the generated image, through the formula Calculated, where α is a random interpolation coefficient between [0, 1], I real Represents the acquired visible light image. Indicates that the discriminator is at the interpolation point The gradient at λ, ||*||2 represents the L2 norm. gp is the weight coefficient of the gradient penalty term, which is used to balance the adversarial loss and gradient penalty. Represents the interpolation point expectations.

[0026] Specifically, a gradient penalty term is introduced on the basis of the standard adversarial loss. By minimizing the adversarial loss of the generator, the night vision images generated by the generator are made as close as possible to the real visible light image distribution to deceive the discriminator. At the same time, by minimizing the degree to which the discriminator's gradient norm deviates from 1, the discriminator satisfies the 1-Lipschitz continuity condition, improves the stability and convergence of the discriminator, and prevents problems such as gradient vanishing or gradient exploding during training. gpThe optimization speed of the generator and the discriminator can be balanced according to the experiment, so as to ensure the authenticity of the generated images and take into account the stability of the training process.

[0027] Furthermore, the loss function of the generator is expressed as follows:

[0028] L G =λ recon ×L recon +λ percep ×L percep +λ adv ×L adv , where λ recon ,λ percep ,λ adv Represents the weight coefficient.

[0029] The loss function of the discriminator is expressed as follows:

[0030] in,

[0031] x represents the real visible light image, E x Represents the expectation of the real image distribution.

[0032] Another aspect of the present application also provides an enhancement information generation system for a low-light-level night vision goggle, which is used to execute an enhancement information generation method for a low-light-level night vision goggle of the present application.

[0033] Compared with the prior art, the advantages of this application are:

[0034] By dividing the visible light image collected by the low-light night vision goggles into blocks, extracting local areas, calculating the local noise variance, and adaptively setting the median filter parameters, the high-frequency noise in the image can be effectively removed while retaining the texture details of the image. Retinex decomposition and illumination map enhancement can improve the brightness and contrast of the image, making the image details clearer and more visible, and improving the accuracy of subsequent imaging.

[0035] By performing distortion correction, guided filtering smoothing, and joint bilateral filtering hole filling on the depth image collected by the infrared depth sensor, a geometrically accurate, edge-smoothed, hole-free depth image can be obtained, providing reliable depth prior information for subsequent 3D reconstruction. Guided filtering smoothing uses the gradient information of the visible light image to perform weighted smoothing on the depth image, which can not only smooth the depth image but also retain the edge and structural information of the scene.

[0036] The generative adversarial network is used to fuse the 3D model with the collected visible light image. Through the adversarial learning of the generator and the discriminator, night vision enhanced images with high brightness, strong contrast, clear texture and accurate geometry can be generated. The generator uses the 3D model as a conditional constraint to ensure that the generated image conforms to the geometric structure of the scene; the discriminator's discrimination result is used as the optimization target to make the distribution of the generated image close to the distribution of the real visible light image.

[0037] A reconstruction loss term is introduced into the loss function of the generator. By minimizing the L1 distance between the generated image and the initial night vision image in the pixel domain and gradient domain, the generated image retains the scene geometric structure and lighting prior information provided by the 3D model, avoiding geometric distortion of the generated image.

[0038] A perceptual loss term is introduced into the loss function of the generator. The pre-trained convolutional neural network is used to extract the semantic features of the generated image and the real visible light image at different scales and receptive fields, and the L1 distance in the feature space is minimized, so that the generated image can learn the texture details and noise distribution characteristics of the real image, generating more natural and realistic night vision images. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0040] Figure 1 is an exemplary flow chart of a method for generating enhanced information for a low-light-level night vision goggle according to the present application;

[0041] Figure 2 is an exemplary flow chart of preprocessing visible light images according to the present application;

[0042] Figure 3 is an exemplary flow chart of preprocessing a depth image according to the present application;

[0043] Figure 4 This is an exemplary flow chart for constructing a three-dimensional model of a low-light-level night vision scene as shown in this application. DETAILED DESCRIPTION

[0044] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0045] Figure 1This is an exemplary flow chart of a method for generating enhanced information of a low-light-level night vision goggle according to the present application. A low-light-level night vision goggle is used to collect a visible light image of a low-light-level night vision scene, and an infrared depth sensor is used to collect a depth image of the low-light-level night vision scene; the collected visible light image and depth image are preprocessed; based on the preprocessed depth image, a three-dimensional model of the low-light-level night vision scene is constructed by a three-dimensional reconstruction algorithm; and a generative adversarial network (GAN) is used to fuse the three-dimensional model with the collected visible light image to generate an enhanced night vision image.

[0046] Specifically, the low-light-level night vision goggles and infrared depth sensors synchronously collect visible light images and depth images of low-light-level night vision scenes. For the collection of visible light images, the imaging sensor of the low-light-level night vision goggles usually adopts a high-sensitivity CMOS or EMCCD sensor, and the pixel resolution can reach 1280×1024 or higher. The spectral response range of the sensor covers the visible light band (390-700nm), and the quantum efficiency can reach more than 90% in the 500-600nm band. The objective system of the low-light-level night vision goggles usually adopts a lens with a large aperture (such as 50mm or larger) and a small F number (such as F1.0-F1.4) to obtain more luminous flux. By adjusting the parameters such as the gain and exposure time of the low-light-level night vision goggles, a visible light image with suitable brightness and low noise can be obtained, and the image size can be set to common resolutions such as 1280×1024 or 640×512.

[0047] For the acquisition of depth images, infrared depth sensors use the ToF (Time-of-Flight) principle or the structured light principle. Taking the ToF sensor as an example, it emits infrared pulse light and receives the returned light signal. By calculating the flight time of the emitted and received light signals, the depth value of each pixel in the scene is obtained. The resolution of the ToF sensor can reach 640×480 or 320×240, the depth measurement range can cover 0.5m to 10m, and the depth accuracy can reach within 1cm. The modulation frequency of the sensor can be set to 20-50MHz to obtain a higher depth resolution. By controlling the emission power and pulse width of the infrared light source, a depth image with high signal-to-noise ratio and low distortion can be obtained.

[0048] like Figure 2As shown, the collected visible light image and depth image are preprocessed; when preprocessing the collected visible light image, the visible light image of size W×H is evenly divided into M×N local sub-images of size w×h, where w=W / M, h=H / N. For example, for a 1280×1024 image, it can be divided into 10×8 sub-images of size 128×128. For each local sub-image, a sliding window of size k×k is used, which slides pixel by pixel with a step size of 1, and a k×k neighborhood centered on the current pixel is extracted as the local area. The value of k is usually an odd number, such as 3, 5, 7, etc., to ensure that the pixel position does not shift after filtering. For each k×k local area extracted, the variance σ of all pixel values ​​in the area is calculated. 2 , as the local noise variance of the region. Assume that the pixel value in the region is I(i,j), i,j=1,2,....,k, then the local noise variance is: in, is the mean value of the pixel values ​​in the region. According to the estimated local noise variance σ 2 , adaptively set the window size and threshold parameters of the median filter. For example, the window size can be set to in Indicates rounding down; the threshold is set to 3σ. Use the set parameters to perform median filtering on each local area to remove the high-frequency noise in the area and obtain the denoised local sub-image. All denoised local sub-images are spliced ​​according to their original positions to obtain the denoised complete visible light image I denoised , the size is still W×H.

[0049] to I denoised Perform Retinex decomposition to obtain a lighting map I of size W×H illumination and reflectivity map I reflectance , satisfying: I denoised =I illumnation ×I reflectance , where × represents pixel-level multiplication. Retinex decomposition is usually implemented using algorithms such as multi-scale Gaussian filtering or guided filtering. Specifically, in this embodiment, guided filtering is used to perform Retinex decomposition. denoised Take the input image p as input, convert it to floating point data, and normalize it to the range of [0, 1]. Construct the guidance image g. Usually, the input image p can be used directly as the guidance image, that is, g = p. You can also preprocess p, such as downsampling, smoothing, etc., to obtain a simplified guidance image. Set the window size r and regularization parameter ε of the guided filter. r determines the local range of the filter, usually taking a value of 10 to 30; ε is used to control the smoothness of the filter, usually taking a value of 0.01-0.1. Perform mean filtering on the guidance image g and the input image p to obtain the local mean image μ gand μ p :μ g =f mean (g,r), μ p =f mean (p,r), where f mean represents the mean filter function, r is the window size. Calculate the local covariance σ of the guidance image g and the input image p gp and the local variance σ gg :σ gp =f mean (g×p,r)-μ g ×μ p ; σ gg =f mean (g×g,r)-μ g ×μ p . Calculate the linear coefficients a and b: b=μ p -a×μ g ; where ε is the regularization parameter, which is used to prevent the denominator from being zero. Perform mean filtering on the linear coefficients a and b to obtain the filtered coefficients and Calculate the output image q: The output image q is used as the estimated illumination map I illumnation , and restore it to an integer type in the range of [0, 255]. According to the Retinex theory, the reflectivity map I reflectance By inputting the image I denoised Divide by light map I illumnation get: To avoid division by zero, you can add a small positive value, such as 1e-6, to the denominator. reflectance Restore it to an integer type in the range of [0, 255] to obtain the final reflectivity map.

[0050] Using the linear stretching algorithm, I illumnation Perform dynamic range expansion to increase the pixel value from the original range [I min ,I max ] is mapped to [0, 255] to obtain the enhanced illumination map I illumnation_enhanced :

[0051]

[0052] Using guided filtering algorithm to analyze the reflectivity image I reflectance Smoothing is performed to remove noise and artifacts in the image and obtain a smoothed reflectivity map I illumnation_smoothed Specifically, in this embodiment, the denoised visible light image I denoised As the guide image g, the reflectivity map I reflectanceAs the input image p, and convert both to floating point data. Set the window size r and regularization parameter ε of the guided filter. Usually, the value of r should be larger than the window size used in Retinex decomposition to achieve a wider range of smoothing effects, such as 30-50; the value of ε can be the same as that in Retinex decomposition, such as 0.01 to 0.1. Perform mean filtering on the guided image g and the input image p to obtain the local mean map μ g and μ p :μ g =f mean (g,r);μ p =f mean (p,r); where f mean represents the mean filter function, r is the window size. Calculate the local covariance σ of the guidance image g and the input image p gp and the local variance σ gg :σ gp =f mean (g×p,r)-μ g ×μ p ; σ gg =f mean (g×g,r)-μ g ×μ g ; Calculate the linear coefficients a and b: b=μ p -a×μ g , where ε is the regularization parameter used to prevent the denominator from being zero. Perform mean filtering on the linear coefficients a and b to obtain the filtered coefficients and Calculate the output image q: The output image q is used as the smoothed reflectivity map I illumnation_smoothed , and restore it to an integer type in the range of [0, 255]. In the process of guided filtering, the guide image g provides local structural information of the image, which is used to guide the filtering process of the input image p. By calculating the local mean and variance, the coefficients of the filter can be adaptively adjusted to suppress noise and artifacts while maintaining the edges and textures of the image. Compared with simple mean filtering or Gaussian filtering, guided filtering can better maintain the details and structure of the image. By setting an appropriate window size r and regularization parameter ε, the smoothness of the filter and the degree of detail retention can be balanced.

[0053] like Figure 3 As shown, the collected depth image is preprocessed, including: firstly, the depth image distortion is corrected, and the intrinsic and extrinsic parameters of the infrared depth sensor are obtained through a calibration plate or other methods, such as Zhang Zhengyou calibration method. The intrinsic parameters include focal length (f x ,f y ), principal point coordinates (cx ,c y ) and radial distortion coefficients (k1, k2, k3), tangential distortion coefficients (p1, p2), etc. The external parameters include the rotation matrix R and translation vector t of the depth sensor relative to the world coordinate system. Calculate the three-dimensional coordinates of the pixel. For each pixel (u, v) in the depth image, calculate the three-dimensional coordinates (X, Y, Z) of the pixel in the depth sensor coordinate system according to its corresponding depth value d. Assume that the size of the depth image is (width, height), and the coordinate range of the pixel (u, v) is [0, width-1] and [0, height-1]. First, convert the pixel coordinates (u, v) to normalized coordinates (x, y): Then, according to the depth value d and the normalized coordinates (x, y), the three-dimensional coordinates (X, Y, Z) of the pixel are calculated: Z = d, X = x × Z, Y = y × Z; distortion correction is performed, and the three-dimensional coordinates (X, Y, Z) of each pixel are corrected using the radial distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2). Calculate the radial distortion (d rx ,d ry ): r 2 =X 2 +Y 2 , d rx =X×(1+k1r 2 +k2r 4 +k3r 6 ), d ry =Y×(1+k1r 2 +k2r 4 +k3r 6 ); calculate the tangential distortion (d tx ,d ty ):d tx =2p1XY+p2(r 2 +2X 2 ), d ty =p1(r 2 +2Y 2 )+2p2XY; add the radial distortion and tangential distortion to the original coordinates to obtain the corrected three-dimensional coordinates (X', Y', Z'): X'=X+d rx +d tx ;

[0054] Y'=Y+d ry +d ty ; Z' = Z. Reproject the corrected three-dimensional coordinates (X', Y', Z') onto the two-dimensional plane to obtain the distortion-corrected pixel coordinates (u', v'). Calculate the normalized coordinates (x', y'): x' = X' / Z', y' = Y' / Z', and convert the normalized coordinates to pixel coordinates: u' = x' × fx +c x , v'=y'×f y +c y , for each pixel (u, v), its corresponding corrected coordinates (u', v') are used as coordinates in the distortion-corrected depth image. Since the distortion-corrected coordinates (u', v') may be non-integer values, the depth values ​​need to be interpolated. Common interpolation methods include nearest neighbor interpolation and bilinear interpolation. For pixels whose corrected coordinates exceed the depth image size range, you can choose to discard or fill them with default values ​​(such as 0).

[0055] Smooth the depth image after distortion correction, and record the depth image after distortion correction as D, with a size of (width, height) and a pixel value range of [0, max_depth]. Record the preprocessed visible light image as I, with the same size of (width, height) and a pixel value range of [0, 255]. Fit the linear transformation coefficients a and b between the depth image D and the guidance image I by the least squares method. For each pixel position (i, j), establish the following system of equations: D(i, j) = a(i, j) × I(i, j) + b(i, j), collect the equations of all pixel positions to form an overdetermined system of equations, and use the least squares method to solve the values ​​of a and b. Obtain the transformation coefficient matrices A and B, both of which have a size of (width, height). Set the threshold t and perform threshold processing on the transformation coefficient matrices A and B. For elements in A greater than t, set the filter window size of the corresponding position to r1; otherwise, set it to r2 (r2<r1). For elements in B that are greater than t, set the filter window size at the corresponding position to r1; otherwise, set it to r2 (r2<r1). Get the filter window size matrix R, whose size is (width, height) and whose element value is r1 or r2. According to the filter window size matrix R, perform adaptive mean filtering on the depth image D. For each pixel position (i, j), the size of the filter window is determined based on the value of R(i, j) with the position as the center. Perform arithmetic averaging on the depth values ​​within the filter window to obtain the filtered depth value D'(i, j) Filter all pixel positions to obtain the filtered depth image D'. Calculate the gradient magnitude map G of the guide image I, using the Sobel operator or other edge detection operators. According to the gradient magnitude map G, set the weight matrix W of different regions of the depth image: W(i,j) = exp(-α×G(i,j)), where α is a parameter that controls the weight decay speed and can be adjusted according to actual needs. Perform weighted fusion on the filtered depth image D' and the original depth image D: D smooth (i,j)=W(i,j)×D'(i,j)+(1-W(i,j))×D(i,j), and get the smoothed depth image Dsmooth The filter window sizes r1 and r2 can be adjusted according to the resolution and noise level of the depth image, and are generally in the range of 5 to 15. The selection of the threshold t depends on the distribution of the transform coefficients a and b, which can be determined by histogram analysis or empirical values. The weight attenuation parameter α ranges from 0.1 to 1, and a larger value will make the smoothing effect of the edge area more obvious.

[0056] Fill the holes in the smoothed depth image and record the smoothed depth image as D smooth , size is (width, height), pixel value range is [0, max depth ]. The preprocessed visible light image is recorded as I, the size is also (width, height), and the pixel value range is [0, 255]. Define the hole pixel threshold as hole threshold , the depth value less than or equal to the threshold is considered as a hole. Traverse the depth image D smooth Each pixel D smooth (i,j). If D smooth (i,j) is less than or equal to hole threshold , then mark the pixel as a hole pixel, denoted as hole(i, j) = 1; otherwise, hole(i, j) = 0. Get the hole pixel marking matrix hole, the size is (width, height), and the element value is 0 or 1. For each hole pixel hole(i, j) = 1, select a set of candidate filling pixels in its neighborhood. Define the neighborhood radius as r, then the coordinate range of the candidate filling pixels is p∈[max(0,ir),min(height-1,i+r)] and q∈[max(0,jr),min(width-1,j+r)]. Mark the coordinates of the candidate filling pixels as (p, q), and the depth value is recorded as D smooth (p, q), the color value of the corresponding position in the visible light image is recorded as I(p, q). For each candidate filling pixel (p, q), calculate its geometric distance and color distance with the hole pixel (i, j) in the depth image and visible light image. The geometric distance calculation formula is: Color distance calculation formula (taking RGB color space as an example): Among them, I R ,I G ,I B Represent the red, green, and blue channels of the visible light image respectively.

[0057] According to the geometric distance and color distance, the weight of each candidate filling pixel is calculated. The weight calculation formula is: Among them, σ geo and σ colorThe influence of geometric distance and color distance is controlled respectively. The depth values ​​of candidate filling pixels are weighted averaged to obtain the filling value of the hole pixel: For all hole pixels hole(i, j) = 1, perform the above filling steps to obtain the filled depth value D filled (i, j). Assign the filled depth value to the corresponding position of the original depth image: D smooth (i,j)=D filled (i,j),if hole(i,j)=1; get the depth image D after the hole is filled smooth Hole pixel threshold hole threshold It is set according to the characteristics of the depth image and the noise level, and the general value range is 0 to 100. The selection of the neighborhood radius r depends on the size and distribution of the holes, and the general value range is 1 to 5. geo and σ color The value of controls the influence of geometric distance and color similarity. It can be adjusted according to actual needs. The general value range is 1 to 10.

[0058] like Figure 4 As shown, according to the preprocessed depth image, a three-dimensional model of the low-light night vision scene is constructed by a three-dimensional reconstruction algorithm; including: the preprocessed depth image is recorded as D, the size is (width, height), and the pixel value range is [0, max_depth]. The preprocessed visible light image is recorded as I, the size is also (width, height), and the pixel value range is [0, 255]. The depth image D is divided into uniform grids, and each grid corresponds to a voxel in the three-dimensional model. The size of the voxel is defined as (voxel_size x ,voxel_size y ,voxel_size z ), which represent the size of the voxel in the x, y, and z directions respectively. Calculate the number of voxels (num_voxels x ,num_voxels y ,num_voxels z ): Create a 3D voxel grid V of size (num_voxels x ,num_voxels y ,num_voxels z ), the initial value is 0. For each voxel V(i, j, k), calculate the coordinates (x, y) and depth value d of the corresponding depth image pixel: d = D(x, y). According to the depth value d and the position of the voxel (i, j, k), the three-dimensional spatial coordinates (X, Y, Z) of the voxel are calculated: The three-dimensional spatial coordinates (X, Y, Z) of the voxel are stored in the corresponding voxel position V (i, j, k) to obtain an initial three-dimensional model.

[0059] The initial 3D model is meshed using a triangular mesh algorithm to reduce the number of faces and vertices of the 3D model. Commonly used triangular mesh algorithms include edge collapse and vertex clustering. Set the target number of faces or simplification ratio for mesh simplification, iteratively simplify the 3D model, merge adjacent voxels, and update the voxel coordinates. The 3D model after mesh simplification is obtained, denoted as M simplified The three-dimensional model M after mesh simplification is reconstructed using the Poisson reconstruction algorithm. simplified Perform surface reconstruction. Calculate the gradient field G of the depth image D x and G y , use Sobel operator or other gradient operator. The gradient field G x and G y Interpolate to 3D model M simplified The vertex positions are used as constraints for surface reconstruction. Poisson's equation is constructed to solve the scalar field F on the surface of the three-dimensional model: The scalar field F is extracted by isosurface extraction to obtain the reconstructed 3D model surface, denoted as M reconstructed .

[0060] Map the preprocessed visible light image I to the reconstructed 3D model surface M reconstructed For each vertex on the surface of the 3D model, calculate its corresponding visible light image coordinates (u, v): By interpolation, the color value of the corresponding coordinate (u, v) in the visible light image I is assigned to the vertex of the three-dimensional model surface; commonly used interpolation algorithms are: Nearest neighbor interpolation: select the integer pixel coordinate closest to (u, v) and directly use the color value of the pixel as the color of the vertex. The nearest neighbor interpolation calculation is simple, but it may cause texture jagged and blocky effects. Bilinear interpolation: Consider the four integer pixel coordinates around (u, v), calculate the contribution weights of the color values ​​of these four pixels to the interpolation point respectively, and then obtain the final color value by weighted average. Suppose the four pixel coordinates are (i, j), (i+1, j), (i, j+1), (i+1, j+1), and the corresponding color values ​​are I(i, j), I(i+1, j), I(i, j+1), I(i+1, j+1), then the interpolation formula is: Among them, α=ui, β=vj, which represent the position weights of the interpolation points relative to the four pixels. Bilinear interpolation can produce a smoother texture effect. Assign vertex color: Assign the interpolated color value to the vertex corresponding to the surface of the 3D model as the texture color of the vertex. Repeat for all vertices, and the visible light image I can be completely mapped onto the surface of the 3D model. Get the 3D model M with texture information textured , that is, the three-dimensional model of the low-light night vision scene. x ,voxel_size y ,voxel_size z ) is set according to the resolution of the depth image and the size of the scene, and generally ranges from 1 to 10. The focal length is set according to the intrinsic parameters of the depth camera. The target number of faces or simplification ratio of the mesh simplification is set according to the complexity of the 3D model and the rendering performance requirements, and generally ranges from 10% to 50% of the original number of faces.

[0061] The three-dimensional model is fused with the collected visible light image using the generative adversarial network GAN to generate an enhanced night vision image. The three-dimensional model of the low-light night vision scene is input as a condition into the generator of the generative adversarial network GAN: the three-dimensional model of the low-light night vision scene is denoted as C, and its size is (width, height, depth), indicating the width, height and depth of the three-dimensional model. The generator G receives the random noise vector z and the conditional input C, and maps them to the night vision image space through the decoder network. The output of the generator G is denoted as G(z, C), which represents the generated night vision image, and its size is (width, height, channels), indicating the width, height and number of color channels of the generated image. The conditional input C is input into the generator G to obtain the initial night vision image I init , whose size is the same as G(z, C). The collected visible light image is input as a real sample into the discriminator of the generative adversarial network GAN: The collected visible light image is recorded as I real , whose dimensions are (width, height, channels), representing the width, height, and number of color channels of the image. The discriminator D receives the visible light image I real , extract the characteristics of its texture and noise distribution through the encoder network. The output of the discriminator D is recorded as D(I real ), which indicates the authenticity score of the input image given by the discriminator, and its value range is [0, 1]. The larger the value, the more realistic the discriminator thinks the input image is.

[0062] In the adversarial learning between the generator and the discriminator, the generator is constrained by the 3D model, and the discriminator's judgment result on the generated image is used as the optimization target: Define the input of the generator G: random noise vector z: z is a random noise vector of fixed dimension, usually sampled from the standard normal distribution N(0, 1). Let the dimension of z be d z ,Right now Where R represents the real number space. Conditional input C: C is the three-dimensional model of the low-light night vision scene, which represents the conditional constraints of the generator. The size of C is Represents the width, height, and depth of the 3D model. C can be mapped to the same dimension d as the random noise vector z through a convolutional neural network or a fully connected layer. z . The forward propagation process of generator G: the random noise vector z and the conditional input C are input into generator G. Generator G maps z and C to the night vision image space through a series of transposed convolution layers, upsampling layers and activation functions. The output of generator G is G(z, C), which represents the generated night vision image, and its size is (width G ,height G , channels), which represents the width, height and number of color channels of the generated image. G(z, C) can be expressed as: G(z, C) = G n (G n-1 (...(G1(z,C'))...)), where G i Denotes the i-th layer of the generator, and n is the total number of layers of the generator. Forward propagation process of the discriminator D: The generated night vision image G(z, C) is input into the discriminator D.

[0063] The discriminator D extracts the features of G(z, C) through a series of convolutional layers, downsampling layers, and activation functions, and outputs a scalar value. The output of the discriminator D is D(G(z, C)), which represents the discriminator's score of the authenticity of the generated image G(z, C). The value range is [0, 1]. The larger the value, the closer the discriminator thinks G(z, C) is to the real image. D(G(z, C)) can be expressed as: D(G(z, C)) = D m (D m-1 (...(D1(G(z,C)))...)), where D i represents the i-th layer of the discriminator, and m is the total number of layers of the discriminator. The optimization goal of the generator G is to maximize the authenticity score D(G(z, C) of the generated image G(z, C) by the discriminator D. The optimization goal of the generator G is L G , which can be expressed as: L G= -log(D(G(z,C))), where log represents the natural logarithm function, which is used to map D(G(z,C)) to the logarithmic space to facilitate the calculation of the gradient. The goal of the generator G is to minimize L G , that is, maximize D(G(z, C)). Parameter update of generator G: According to the optimization target L of generator G G , calculate the gradient of the generator parameters where θ G Represents the parameter set of the generator. Using the back-propagation algorithm, the gradient Back propagates to each layer of the generator and updates the parameters θ of the generator according to the gradient G Commonly used optimization algorithms include stochastic gradient descent (SGD), adaptive moment estimation (Adam), etc. The learning rate is α, which represents the step size of each parameter update. The parameter update formula can be expressed as: Among them, ← represents the assignment operation, Represents the dot product operation. The generator G takes the random noise vector z and the conditional input C as input to generate the night vision image G(z, C). G(z, C) is input into the discriminator D to obtain the discriminant result D(G(z, C)). The generator G takes maximizing D(G(z, C)) as the optimization goal, calculates the gradient through the back propagation algorithm, and updates the generator parameter θ G , making the generated night vision images more realistic and difficult for the discriminator to identify as false images. This process is repeated until the preset number of iterations is reached or a certain convergence condition is met. Finally, the generator G is able to generate high-quality, realistic night vision enhanced images given the random noise vector z and the conditional constraint C.

[0064] The reconstruction loss term is expressed as follows: Among them, G represents the generator, z represents the random noise vector, C represents the constructed three-dimensional model of the low-light night vision scene, and I init represents the initial night vision image obtained by decoding C into the generator G, and ||*||1 represents the L1 norm. represents the gradient operator, represents the gradient map of the night vision image generated by the generator, represents the gradient map of the initial night vision image, λ g is the weight coefficient of the gradient loss, which is used to balance the ratio of pixel loss and gradient loss.

[0065] The perceptual loss term is expressed as follows: Where F represents a pre-trained convolutional neural network, which is used to extract multi-scale convolutional features of images. i Represents the feature map extracted by the i-th convolutional layer, i∈{1, 2, ......., N}, N is the total number of convolutional layers. realrepresents the collected visible light image, and G(z, C) represents the night vision image generated by the generator. i is the weight coefficient of the perceptual loss of the i-th layer, which is used to balance the importance of features in different convolutional layers; L style Represents the style loss, which is used to measure the difference between the generated image and the real image in terms of texture, structure and other style features. Its expression is: Among them, G j Represents the Gram matrix calculated from the convolutional feature map, which is used to capture the correlation and statistical information between feature maps. j∈{1, 2, ..., M}, M is the number of convolutional layers used to calculate the Gram matrix. ||*||1 represents the L1 norm. λ s is the weight coefficient of style loss.

[0066] The adversarial loss term is expressed as follows: Where D represents the discriminator, G represents the generator, z represents the random noise vector, and C represents the constructed three-dimensional model of the low-light night vision scene. z represents the expectation of the random noise vector z, log represents the natural logarithm function, and D(G(z, C)) represents the authenticity score obtained by inputting the night vision image generated by the generator into the discriminator. The value range is [0, 1]. The larger the value, the more realistic the discriminator thinks the generated image is. The second term is the gradient penalty term, which is used to constrain the gradient norm of the discriminator to be close to 1, improving the training stability and the quality of the generated image. Represents the interpolation point between the real image and the generated image, through the formula Calculated, where α is a random interpolation coefficient between [0, 1], I real Represents the acquired visible light image. Indicates that the discriminator is at the interpolation point The gradient at λ, ||*||2 represents the L2 norm. gp is the weight coefficient of the gradient penalty term, which is used to balance the adversarial loss and gradient penalty. Represents the interpolation point expectations.

[0067] The loss function of the generator is expressed as follows: L G =λ recon ×L recon +λ percep ×L percep +λ adv ×L adv , where λ recon ,λ percep ,λ adv Represents the weight coefficient. Data processing of the discriminator: The discriminator D receives two types of image data: the real visible light image I real, sampled from the visible light image dataset. The night vision image G(z, C) generated by the generator G is obtained by inputting a random noise vector z and a conditional input C. The task of the discriminator D is to determine whether the input image is a real image or a generated image, that is, to perform a binary classification task. The loss function L of the discriminator D is D It includes three items: Log-likelihood loss of real image: L real = -log(D(I real )), where D(I real ) represents the discriminator's response to the real image I real The log-likelihood loss of the generated image: L fake =-log(1-D(G(z,C))), where D(G(z,C)) represents the discriminator's score of the authenticity of the generated image G(z,C). Gradient penalty term: Among them, I interpolated is the real image I real and the interpolated image between the generated image G(z, C), λ gp is the weight coefficient of the gradient penalty term. The total loss L of the discriminator D D For: L D =L real +L fake +L gp .

[0068] Training of generator G and discriminator D: Generator G and discriminator D are trained alternately, that is, the generator and discriminator are trained in sequence. The goal of generator G is to minimize the total loss L G :L G =L recon +L percep +L adv , where L recon is the reconstruction loss, L percep is the perceptual loss, L adv is the adversarial loss. The goal of the discriminator D is to minimize the total loss L D :L D =L real +L fake +L gp The training process is as follows: Sample a batch of real images I from the visible light image dataset real , batch size is batch size . Sample a batch of noise vectors z from a random noise distribution (such as a Gaussian distribution) with a dimension of noise dim . Input the noise vector z and the conditional input C into the generator G to obtain a batch of generated images G(z, C). real And generate the image G(z, C) and input it into the discriminator D, and calculate the loss L of the discriminator D. The gradient of the discriminator D is calculated through the back propagation algorithm, and the parameters of the discriminator D are updated. The noise vector z and the conditional input C are input into the generator G to obtain a batch of generated images G(z, C). Calculate the reconstruction loss L of the generator recon , Perceptual loss L percep and adversarial loss L adv , and weighted sum to get the total loss L of the generator G The gradient of the generator G is calculated through the back-propagation algorithm, and the parameters of the generator G are updated. Repeat the above steps until the preset number of iterations epoch_num is reached or the quality of the generated image meets the requirements.

[0069] Generate enhanced night vision images: After training, fix the parameters of the generator G. For a given 3D model C of a low-light night vision scene, sample a noise vector z from a random noise distribution. Input the noise vector z and the conditional input C into the generator G, and obtain the enhanced night vision image G(z, C) through the forward propagation process. Output the enhanced night vision image G(z, C) as the final generated result.

Claims

1. A method for generating enhanced information for low-light-level night vision goggles, characterized in that: include: Use low-light-level night vision goggles to collect visible light images of low-light-level night vision scenes, and use infrared depth sensors to collect depth images of low-light-level night vision scenes; Preprocessing the collected visible light images and depth images; According to the preprocessed depth image, a 3D model of the low-light night vision scene is constructed through a 3D reconstruction algorithm; Using the Generative Adversarial Network (GAN), the 3D model is fused with the collected visible light image to generate enhanced night vision images. Preprocess the collected visible light images, including: The visible light image collected by the low-light night vision goggles is divided into blocks to obtain multiple local sub-images; According to each local sub-image, the local area is extracted pixel by pixel on the sub-image by means of a sliding window; For each local area extracted, the variance of the pixel values ​​in the local area is counted as the local noise variance; According to the local noise variance, the parameters of the median filter are set; the corresponding local area is subjected to median filtering using the median filter with the set parameters; all local areas are subjected to median filtering to obtain a visible light image after noise reduction; Perform Retinex decomposition on the denoised visible light image to obtain an illumination map reflecting the image brightness distribution and a reflectivity map reflecting the image. The range of the illumination map is expanded by using a stretching algorithm to enhance the image brightness contrast and obtain an enhanced illumination map; The reflectivity map is smoothed by using a guided filtering algorithm to obtain a smoothed reflectivity map; The enhanced illumination map and the smoothed reflectance map are multiplied at pixel level to obtain the preprocessed visible light image.

2. The method for generating enhanced information for low-light-level night vision goggles according to claim 1, characterized in that: Preprocess the acquired depth image, including: The depth image distortion is corrected using the pre-calibrated internal and external parameters of the infrared depth sensor. The three-dimensional coordinates of each pixel in the depth image are corrected by the radial and tangential distortion coefficients to eliminate the nonlinear distortion introduced by the imaging process and obtain a distortion-corrected depth image. The depth image after distortion correction is used as the input image, the preprocessed visible light image is used as the guide image, and the depth image is smoothed by using the guided filtering algorithm to obtain a smoothed depth image; The smoothed depth image and the preprocessed visible light image are used as input, and the joint bilateral filtering algorithm is used to fill the holes in the depth image. The geometric distance between the depth images and the color similarity constraints between the visible light images are used to fill the hole areas in the depth image to obtain the preprocessed depth image.

3. The method for generating enhanced information for low-light-level night vision goggles according to claim 2, characterized in that: The depth image is smoothed using a guided filtering algorithm to obtain a smoothed depth image, including: Using the guided filtering algorithm, the linear transformation coefficients between the input depth image and the guided image are fitted by the least squares method; For the area where the linear transformation coefficient is greater than the threshold, a mean filter with a filter window greater than the threshold is used for filtering, and vice versa, a mean filter with a filter window less than the threshold is used for filtering to obtain a filtered depth image; Calculate the gradient magnitude of the guide image and set the weights of different regions of the depth image according to the gradient magnitude; According to the set weights and the filtered depth image, weighted fusion is performed to obtain a smoothed depth image.

4. The method for generating enhanced information for low-light-level night vision goggles according to any one of claims 1 to 3, characterized in that: The 3D model of the low-light-level night vision scene is constructed through the 3D reconstruction algorithm, including: Taking the preprocessed depth image as input, the depth image is divided into uniform grids, each grid corresponds to a voxel in the 3D model; According to the depth value of the depth image pixel corresponding to each voxel, the three-dimensional space coordinates of each voxel are calculated to obtain an initial three-dimensional model; Use triangular mesh algorithm to simplify the mesh of the initial three-dimensional model; The surface of the 3D model after mesh simplification is reconstructed using the Poisson reconstruction algorithm, and the surface shape of the 3D model is constrained by the gradient information of the depth image to obtain the reconstructed 3D model. The preprocessed visible light image is mapped to the reconstructed 3D model surface, and the texture information of the 3D model is added through the visible light image to obtain a 3D model of the low-light night vision scene.

5. The method for generating enhanced information for low-light-level night vision goggles according to claim 4, characterized in that: The 3D model is fused with the collected visible light image using the Generative Adversarial Network (GAN), including: The 3D model of the low-light night vision scene is input as a condition into the generator of the Generative Adversarial Network (GAN). The generator generates an initial night vision image by decoding the geometric structure and texture mapping of the 3D model. The collected visible light image is input as a real sample into the discriminator of the generative adversarial network GAN. The discriminator obtains the characteristics of the visible light image by encoding the texture and noise distribution of the visible light image. In the adversarial learning between the generator and the discriminator, the generator is constrained by the 3D model and the discriminator's judgment result on the generated image is used as the optimization target. The generator parameters are updated through back propagation to generate enhanced night vision images. A reconstruction loss term is set in the loss function of the generator, and the difference between the night vision image generated by the generator in the current iteration step and the initial night vision image in the pixel space is constrained, so that the generator can learn the geometric structure and lighting prior information of the 3D model. A perceptual loss term is set in the loss function of the generator, and the multi-scale convolutional features of the night vision image generated by the generator in the current iteration and the collected visible light image are extracted through a pre-trained convolutional neural network. The L1 distance between the features of the two images in the feature space composed of the convolutional features is calculated, so that the generator can learn the texture and noise distribution of the visible light image. An adversarial loss term is set in the generator's loss function, and the night vision image generated by the generator in the current iteration step is input into the discriminator for true or false discrimination, and the discrimination result of the discriminator is used as the generator's loss; Through alternating training of the generator and the discriminator, the reconstruction loss, perceptual loss and adversarial loss of the generator are minimized, while the binary classification accuracy of the discriminator for visible light images and night vision images generated by the generator in the current iteration step is maximized, so that the parameters of the generator and the discriminator reach Nash equilibrium, and the generator after adversarial learning is obtained.

6. The method for generating enhanced information for low-light-level night vision goggles according to claim 5, characterized in that: The reconstruction loss term is expressed as follows: Among them, G represents the generator, z represents the random noise vector, C represents the constructed three-dimensional model of the low-light night vision scene, and I init represents the initial night vision image obtained by decoding C into the generator G, and ||*||1 represents the L1 norm; represents the gradient operator, represents the gradient map of the night vision image generated by the generator, represents the gradient map of the initial night vision image, λ g is the weight coefficient of the gradient loss.

7. The method for generating enhanced information for low-light-level night vision goggles according to claim 6, characterized in that: The perceptual loss term is expressed as follows: Where F represents the pre-trained convolutional neural network; F i Represents the feature map extracted by the i-th convolutional layer; I real represents the collected visible light image, G(z,C) represents the night vision image generated by the generator; λ i is the weight coefficient of the perceptual loss of the i-th layer; L style represents the style loss, which is expressed as: Among them, G j represents the Gram matrix calculated from the convolutional feature map; j∈1,2,.....,M, M is the number of convolutional layers used to calculate the Gram matrix; ||*||1 represents the L1 norm; λ s is the weight coefficient of style loss.

8. The method for generating enhanced information for low-light-level night vision goggles according to claim 7, characterized in that: The adversarial loss term is expressed as follows: Where D represents the discriminator, G represents the generator, z represents the random noise vector, and C represents the constructed three-dimensional model of the low-light night vision scene; E z represents the expectation of the random noise vector z, and D(G(z,C)) represents the authenticity score obtained by inputting the night vision image generated by the generator into the discriminator; Represents the interpolation point between the real image and the generated image; Indicates that the discriminator is at the interpolation point The gradient at, ||*||2 represents the L2 norm; λ gp is the weight coefficient of the gradient penalty term; Represents the interpolation point expectations.

9. The method for generating enhanced information for low-light-level night vision goggles according to claim 8, characterized in that: The loss function of the generator is expressed as follows: L G =λ recon ×L recon +λ percep ×L percep +λ adv ×L adv Among them, λ recon ,λ percep ,λ adv represents the weight coefficient; The loss function of the discriminator is expressed as follows: Among them, x represents the real visible light image, E x Represents the expectation of the real image distribution.

10. An enhanced information generation system for low-light-level night vision goggles, characterized in that: include: At least one processing unit; used to execute instructions to implement the method for generating enhanced information for a low-light-level night vision goggle as described in any one of claims 1 to 9.

Images