A method and system for generating augmented information for a low-light-level night vision device

Abstract

The application discloses a kind of micro-light night vision mirror's enhanced information generation method and system, it is related to image data processing field, including: using micro-light night vision mirror to collect the visible light image of micro-light night vision scene, and using infrared depth sensor to collect the depth image of micro-light night vision scene;The collected visible light image and depth image are preprocessed;According to the depth image after preprocessing, the three-dimensional model of micro-light night vision scene is constructed by three-dimensional reconstruction algorithm;Using generative adversarial network GAN, the visible light image collected is fused with three-dimensional model, and enhanced night vision image is generated;For the scene geometric distortion in micro-light night vision enhanced image in prior art, the depth image of micro-light night vision mirror scene is collected in the application, and the three-dimensional geometric model of scene is constructed by depth image. Enhanced night vision image is generated by using generative adversarial network to fuse three-dimensional model and visible light image, etc., retains scene geometric structure, generates more realistic night vision image.

Description

Technical Field

[0001] The present application relates to the field of image data processing, and in particular to a method and system for generating enhanced information for low-light-level night vision goggles. Background Art

[0002] With the continuous development of night vision imaging technology, low-light-level night vision goggles have gained widespread application in military reconnaissance, security surveillance, and other fields. By enhancing weak visible and near-infrared light, these goggles enable imaging in low-light conditions, extending the human eye's observation capabilities. However, due to the weak signal and low signal-to-noise ratio of low-light imaging, direct imaging quality is poor, necessitating enhanced processing of low-light images to improve visual perception.

[0003] Traditional low-light-level night vision enhancement methods primarily optimize image contrast, brightness, and clarity. Among these, histogram equalization improves image contrast by stretching the pixel value distribution; Retinex theory achieves image dynamic range compression and color constancy correction by estimating illumination and reflectance components; and multi-scale fusion methods enhance image texture detail by fusing features such as multi-scale gradients. While these methods improve the visual quality of low-light images to some extent, they fail to fully utilize prior information about the imaged scene, making it difficult to restore scene geometry and resulting in a lack of realism in the enhanced images.

[0004] In recent years, the rise of deep learning technology has brought new opportunities for low-light-level night vision image enhancement. Convolutional neural networks can directly model image quality enhancement through end-to-end training, and generative adversarial networks can learn the probability distribution of real images to generate realistic enhanced images. However, most existing deep learning-based methods treat low-light-level image enhancement as a two-dimensional image-to-image mapping, ignoring the three-dimensional structure of the scene. The low-light imaging process is prone to optical distortion and noise, resulting in inaccurate depth estimation and geometric distortion in the enhanced image. Furthermore, the network structure and loss function design of existing methods are not sophisticated enough, making it difficult to effectively constrain the local detail and global consistency of the enhanced image. Summary of the Invention

[0005] In response to the problem of scene geometric distortion in low-light-level night vision enhanced images in the prior art, the present application provides a method and system for generating enhanced information for low-light-level night vision goggles, which fully utilizes the depth information of the scene and generative adversarial networks, retains the scene geometric structure, and generates more realistic night vision images.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] One aspect of the present application provides a method for generating enhanced information for a low-light-level night vision goggle, comprising: using a low-light-level night vision goggle to collect a visible light image of a low-light-level night vision scene, and using an infrared depth sensor to collect a depth image of the low-light-level night vision scene; preprocessing the collected visible light image and depth image; constructing a three-dimensional model of the low-light-level night vision scene through a three-dimensional reconstruction algorithm based on the preprocessed depth image; and using a generative adversarial network (GAN) to fuse the three-dimensional model with the collected visible light image to generate an enhanced night vision image.

[0008] Among them, low-light-level night vision goggles are a photoelectric device used for imaging in low-light conditions. They can amplify weak visible light and near-infrared light to enable nighttime observation. Its core component is the low-light-level image intensifier, which mainly includes a photocathode, a microchannel plate, a fluorescent screen, etc. In this application, low-light-level night vision goggles can acquire scene images at night and in low-light environments, but the image quality is usually not high. This application scheme uses low-light-level night vision goggles to collect visible light images as input for night vision enhancement, and at the same time introduces an infrared depth sensor to collect depth images of the scene to provide depth information for subsequent three-dimensional reconstruction. Post-processing and enhancement of low-light-level night vision goggles through algorithms can further improve their imaging performance and user experience.

[0009] Low-light-level night vision scenes refer to scenes observed at night or in low-light environments using night vision goggles. These scenes typically have extremely dim lighting, difficult to see with the naked eye, but contain a small amount of visible and near-infrared light. Specifically, the visible light illumination in the scene is typically less than 0.1 lux, which is considered a low-light environment.

[0010] The collected visible light image is preprocessed, including: dividing the visible light image collected by the low-light night vision goggles into blocks to obtain multiple local sub-images; according to each local sub-image, extracting a local area on the sub-image pixel by pixel by a sliding window; for each extracted local area, counting the variance of the pixel values ​​in the local area as the local noise variance; setting the parameters of the median filter according to the local noise variance; performing median filtering on the corresponding local area using the median filter with the set parameters; performing median filtering on all local areas to obtain a denoised visible light image; performing Retinex decomposition on the denoised visible light image to obtain an illumination map reflecting the brightness distribution of the image and a reflectance map reflecting the image; using a stretching algorithm to expand the range of the illumination map to enhance the brightness contrast of the image to obtain an enhanced illumination map; using a guided filtering algorithm to smooth the reflectance map to obtain a smoothed reflectance map; and performing pixel-level multiplication on the enhanced illumination map and the smoothed reflectance map to obtain a preprocessed visible light image.

[0011] Further, the collected depth image is preprocessed, including: correcting the depth image distortion by using the internal and external parameters of the infrared depth sensor calibrated in advance, correcting the three-dimensional coordinates of each pixel in the depth image through the radial and tangential distortion coefficients to eliminate the nonlinear distortion introduced in the imaging process, and obtaining the depth image after distortion correction; using the corrected depth image as an input image, using the preprocessed visible light image as a guide image, and using a guided filtering algorithm to smooth the depth image to obtain a smoothed depth image; using the smoothed depth image and the preprocessed visible light image as inputs, using a joint bilateral filtering algorithm to fill the holes in the depth image, and using the geometric distance between the depth images and the color similarity between the visible light images to constrain the filling of the hole region in the depth image, to obtain the preprocessed depth image.

[0012] Further, the depth image is smoothed using a guided filtering algorithm to obtain a smoothed depth image, including: using a guided filtering algorithm to fit the linear transformation coefficients between the input depth image and the guide image by a least squares method; for the region where the linear transformation coefficient is greater than a threshold, using a mean filter with a filter window greater than the threshold to filter, and otherwise using a mean filter with a filter window smaller than the threshold to filter, to obtain a filtered depth image; calculating the gradient amplitude of the guide image, and setting the weight of different regions of the depth image according to the gradient amplitude; performing weighted fusion according to the set weight and the filtered depth image to obtain the smoothed depth image.

[0013] Further, a three-dimensional model of the low-light night vision scene is constructed by a three-dimensional reconstruction algorithm, including: taking the preprocessed depth image as an input, dividing the depth image into uniform grids, and each grid corresponding to a voxel in the three-dimensional model; calculating the three-dimensional space coordinates of each voxel according to the depth value of the depth image pixel point corresponding to each voxel to obtain an initial three-dimensional model; using a triangular mesh algorithm to simplify the initial three-dimensional model; using a Poisson reconstruction algorithm to reconstruct the surface of the three-dimensional model after grid simplification, and using the gradient information of the depth image to constrain the surface shape of the three-dimensional model to obtain a reconstructed three-dimensional model; mapping the preprocessed visible light image to the surface of the reconstructed three-dimensional model, adding texture information of the three-dimensional model through the visible light image, and obtaining the three-dimensional model of the low-light night vision scene.

[0014] Furthermore, a generative adversarial network (GAN) is used to fuse the three-dimensional model with the collected visible light image to generate an enhanced night vision image, including: inputting the three-dimensional model of the low-light night vision scene as a condition into the generator of the generative adversarial network (GAN), and the generator generates an initial night vision image by decoding the geometric structure and texture mapping of the three-dimensional model; inputting the collected visible light image as a real sample into the discriminator of the generative adversarial network (GAN), and the discriminator obtains the characteristics of the visible light image by encoding the texture and noise distribution of the visible light image; in the adversarial learning between the generator and the discriminator, the generator is constrained by the three-dimensional model and the discrimination result of the discriminator on the generated image is used as the optimization target, and the generator parameters are updated through back propagation to generate an enhanced night vision image; a reconstruction loss term is set in the loss function of the generator, and the difference in pixel space between the night vision image generated by the generator in the current iteration step and the initial night vision image is constrained. The generator learns the geometric structure and lighting prior information of the three-dimensional model; a perceptual loss term is set in the generator's loss function, and the multi-scale convolution features of the night vision image generated by the generator in the current iteration step and the collected visible light image are extracted through a pre-trained convolutional neural network, and the L1 distance of the two image features in the feature space composed of the convolution features is calculated, so that the generator learns the texture and noise distribution of the visible light image; an adversarial loss term is set in the generator's loss function, and the night vision image generated by the generator in the current iteration step is input into the discriminator for true or false discrimination, and the discrimination result of the discriminator is used as the generator's loss; through alternating training of the generator and the discriminator, the reconstruction loss, perceptual loss and adversarial loss of the generator are minimized, and the binary classification accuracy of the discriminator for the visible light image and the night vision image generated by the generator in the current iteration step is maximized, so that the parameters of the generator and the discriminator reach Nash equilibrium, and the generator after adversarial learning is obtained.

[0015] Furthermore, the reconstruction loss term is expressed as follows: Among them, G represents the generator, z represents the random noise vector, C represents the constructed three-dimensional model of the low-light night vision scene, and I init represents the initial night vision image obtained by decoding C into the generator G, and ||*||1 represents the L1 norm. represents the gradient operator, ▽G(z,C) represents the gradient map of the night vision image generated by the generator, represents the gradient map of the initial night vision image, λ g is the weight coefficient of gradient loss, which is used to balance the ratio of pixel loss and gradient loss.

[0016] Among them, the constructed three-dimensional model of the low-light night vision scene participates in the process of calculating the reconstruction loss: according to the boundary and internal structure of the three-dimensional model, each voxel is marked as occupied (1) or idle (0). Occupied voxels represent the surface or interior of the model, and idle voxels represent the background. The voxelized three-dimensional model is denoted as V(C). A three-dimensional convolutional neural network is used to extract and encode the voxel model V(C). The network input is a binary voxel occupancy grid, and the high-level semantic features of the model are extracted through multi-layer three-dimensional convolution and downsampling. Let the encoded feature vector be f(V(C)), which represents the geometric structure information of the entire three-dimensional model in the form of a one-dimensional vector. The voxel code f(V(C)) and the noise vector z are input into the generator G, and the initial night vision image I is generated through the three-dimensional deconvolution network. init The deconvolution network consists of multiple layers of 3D deconvolution and upsampling, decoding low-dimensional feature vectors into high-resolution images. During the decoding process, the feature map is gradually expanded and combined with the encoded geometric information f(V(C)), so that the generated image approximates the geometry of the real scene. The reconstruction loss is calculated based on the generated initial night vision image I_init and the original 3D model C. The reconstruction loss consists of pixel loss and gradient loss.

[0017] The generated night vision image I init Compare with the real image in pixel space. The L1 norm can be used to calculate the sum of the absolute differences between the corresponding pixels of the two images, that is, ||G(z,f(V(C)))-I init || 1. This loss term encourages the generated image to be close to the real image at the pixel level. init Calculate the gradient map of the real image and the real image respectively, and then compare the difference between the two gradient maps. The gradient map can be obtained by pixel finite difference, such as Then use the L1 norm to measure the difference between the two gradient maps, that is, Gradient loss can capture the structural information in the image, such as edges, contours, etc., and encourage the generated image to be consistent with the real image in local structure. The pixel loss and gradient loss are weighted and summed to obtain the final reconstruction loss L recon . The reconstruction loss L recon As one of the optimization objectives of the generator G, it guides the training of the generator together with other loss functions (such as perceptual loss and adversarial loss). Through backpropagation and gradient descent, the parameters of the generator are adjusted so that it can generate geometrically realistic night vision enhanced images.

[0018] Specifically, a gradient domain loss term is introduced based on the pixel-level reconstruction loss. The gradient reflects the edge and texture structure information of the image. By constraining the difference between the generated image and the initial image in the gradient domain, the generator can better learn the scene structure and geometric details represented by the 3D model, improving the reconstruction quality. At the same time, the gradient loss is more robust to lighting changes and noise interference, helping to improve the clarity and realism of the generated image. g The value of can be tuned according to experiments to balance pixel loss and gradient loss, ensuring both the global reconstruction effect and the local detail expression.

[0019] Furthermore, the perceptual loss term is expressed as follows:

[0020] Where F represents a pre-trained convolutional neural network, which is used to extract multi-scale convolution features of the image. i Represents the feature map extracted by the i-th convolutional layer, i∈1,2,....,N, where N is the total number of convolutional layers. real Represents the collected visible light image, and G(z,C) represents the night vision image generated by the generator. i is the weight coefficient of the perceptual loss of the i-th layer, which is used to balance the importance of features in different convolutional layers; L style Represents the style loss, which is used to measure the difference between the generated image and the real image in terms of texture, structure and other style features. Its expression is: Among them, G j Represents the Gram matrix calculated from the convolutional feature map, which is used to capture the correlation and statistical information between feature maps. j∈1,2,.....,M, where M is the number of convolutional layers used to calculate the Gram matrix. ||*||1 represents the L1 norm. λ s is the weight coefficient of style loss.

[0021] Among them, the Gram matrix is ​​a matrix used to measure the correlation and statistical information between the feature maps of the convolutional neural network. It characterizes the high-order statistics of the features by calculating the inner product between the feature maps, reflecting the style characteristics of the image in terms of texture, structure, etc. The style loss L_style uses the Gram matrix to measure the difference in style characteristics between the generated image G(z, C) and the real image I_real. By minimizing the style loss, the generated image is encouraged to be similar to the real image in terms of texture, structure, etc., thereby improving the visual realism of the generated image. At the same time, by calculating the Gram matrix on multiple convolutional layers, the style features of different scales and abstraction levels are considered to achieve multi-level characterization and matching of image style. It is introduced into the generative adversarial network for night vision image enhancement, and the realism of the generated image is constrained by the style loss, thereby improving the visual quality and perceptual experience of the enhanced result.

[0022] Specifically, a style loss term is introduced based on the multi-scale perceptual loss. By constraining the differences between the generated image and the real image in the feature space of different convolutional layers, the generator not only learns the content information of the real image, such as objects and scene structure, but also learns the style information of the real image, such as texture, color, and local structure. Multi-scale feature representation helps capture the semantic information of the image at different levels of abstraction, while the style loss helps improve the realism and consistency of the generated image. i and λ s Experimental tuning can be used to balance content loss and style loss, ensuring the perceptual similarity between the generated image and the real image while taking into account the artistic style characteristics of low-light night vision images.

[0023] Furthermore, the adversarial loss term is expressed as follows:

[0024] Where D represents the discriminator, G represents the generator, z represents the random noise vector, and C represents the constructed three-dimensional model of the low-light night vision scene. z represents the expectation of the random noise vector z, and D(G(z,C)) represents the authenticity score obtained by inputting the night vision image generated by the generator into the discriminator. The value range is [0, 1]. The larger the value, the more realistic the discriminator thinks the generated image is.

[0025] The second term is the gradient penalty term, which is used to constrain the gradient norm of the discriminator to be close to 1, thereby improving training stability and generating image quality. Represents the interpolation point between the real image and the generated image, through the formula Calculated, where α is a random interpolation coefficient between [0, 1], I real Represents the collected visible light image. Indicates that the discriminator is at the interpolation point The gradient at λ, ||*||2 represents the L2 norm. gp is the weight coefficient of the gradient penalty term, which is used to balance the adversarial loss and gradient penalty. Represents the interpolation point expectations.

[0026] Specifically, a gradient penalty term is introduced based on the standard adversarial loss. By minimizing the adversarial loss of the generator, the night vision images generated by the generator are made as close as possible to the distribution of real visible light images, thereby deceiving the discriminator. At the same time, by minimizing the degree to which the discriminator's gradient norm deviates from 1, the discriminator satisfies the 1-Lipschitz continuity condition, improving the stability and convergence of the discriminator and preventing problems such as gradient vanishing or gradient exploding during training. gpIt can be tuned according to experiments to balance the optimization speed of the generator and discriminator, ensuring the authenticity of the generated images while taking into account the stability of the training process.

[0027] Furthermore, the loss function of the generator is expressed as follows:

[0028] L G =λ recon ×L recon +λ percep ×L percep +λ adv ×L adv , where λ recon ,λ percep ,λ adv Represents the weight coefficient.

[0029] The loss function of the discriminator is expressed as follows:

[0030] in,

[0031] x represents the real visible light image, E x represents the expectation of the true image distribution.

[0032] Another aspect of the present application also provides an enhanced information generation system for low-light-level night vision goggles, which is used to execute the enhanced information generation method for low-light-level night vision goggles of the present application.

[0033] Compared with the prior art, the advantages of this application are:

[0034] By segmenting visible light images captured by low-light-level night vision goggles, extracting local regions, calculating local noise variance, and adaptively setting median filter parameters, high-frequency noise can be effectively removed while preserving texture detail. Using Retinex decomposition and illumination map enhancement, image brightness and contrast can be improved, making image details more visible and enhancing subsequent imaging accuracy.

[0035] By performing distortion correction, guided filtering smoothing, and joint bilateral filtering hole filling on the depth image captured by the infrared depth sensor, a geometrically accurate, edge-smoothed, hole-free depth image can be obtained, providing reliable depth prior information for subsequent 3D reconstruction. Guided filtering smoothing utilizes the gradient information of the visible light image to perform weighted smoothing on the depth image, smoothing the depth image while preserving the scene's edges and structural information.

[0036] The three-dimensional model is fused with the collected visible light image by using a generative adversarial network. Through the adversarial learning of the generator and the discriminator, a night vision enhanced image with high brightness, strong contrast, clear texture and accurate geometry can be generated. The generator is constrained by the three-dimensional model to ensure that the generated image conforms to the geometric structure of the scene. The discrimination result of the discriminator is used as the optimization target to make the distribution of the generated image close to that of the real visible light image.

[0037] A reconstruction loss term is introduced into the loss function of the generator. By minimizing the L1 distance between the generated image and the initial night vision image in the pixel domain and the gradient domain, the generated image retains the scene geometry and lighting prior information provided by the three-dimensional model, avoiding geometric distortion of the generated image.

[0038] A perceptual loss term is introduced into the loss function of the generator. A pre-trained convolutional neural network is used to extract semantic features of the generated image and the real visible light image at different scales and different receptive fields, and minimize the L1 distance in the feature space, so that the generated image learns the texture details and noise distribution characteristics of the real image, and generates a more natural and realistic night vision image. BRIEF DESCRIPTION OF DRAWINGS

[0039] The present application will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same numbers represent the same structures, wherein:

[0040] Figure 1 is an exemplary flowchart of a method for generating enhanced information of a low-light night vision mirror according to the present application;

[0041] Figure 2 is an exemplary flowchart of preprocessing a visible light image according to the present application;

[0042] Figure 3 is an exemplary flowchart of preprocessing a depth image according to the present application;

[0043] Figure 4 is an exemplary flowchart of constructing a three-dimensional model of a low-light night vision scene according to the present application. DETAILED DESCRIPTION

[0044] The method and system provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0045] Figure 1This is an exemplary flowchart of a method for generating enhanced information for a low-light-level night vision goggle according to the present application. The method comprises collecting visible light images of a low-light-level night vision scene using a low-light-level night vision goggle and collecting depth images of the low-light-level night vision scene using an infrared depth sensor; preprocessing the collected visible light image and depth image; constructing a three-dimensional model of the low-light-level night vision scene using a three-dimensional reconstruction algorithm based on the preprocessed depth image; and using a generative adversarial network (GAN) to fuse the three-dimensional model with the collected visible light image to generate an enhanced night vision image.

[0046] Specifically, the low-light-level night vision goggles and infrared depth sensors synchronously collect visible light images and depth images of the low-light-level night vision scene. For the collection of visible light images, the imaging sensor of the low-light-level night vision goggles usually adopts a high-sensitivity CMOS or EMCCD sensor, and the pixel resolution can reach 1280×1024 or higher. The spectral response range of the sensor covers the visible light band (390-700nm), and the quantum efficiency can reach more than 90% in the 500-600nm band. The objective system of the low-light-level night vision goggles usually adopts a lens with a large aperture (such as 50mm or larger) and a small F number (such as F1.0-F1.4) to obtain more light flux. By adjusting the parameters such as the gain and exposure time of the low-light-level night vision goggles, a visible light image with appropriate brightness and low noise can be obtained. The image size can be set to common resolutions such as 1280×1024 or 640×512.

[0047] For the acquisition of depth images, infrared depth sensors use the ToF (Time-of-Flight) principle or the structured light principle. Taking the ToF sensor as an example, it emits infrared pulse light and receives the returned light signal. By calculating the flight time of the transmitted and received light signals, the depth value of each pixel in the scene is obtained. The resolution of the ToF sensor can reach 640×480 or 320×240, the depth measurement range can cover 0.5m to 10m, and the depth accuracy can reach within 1cm. The modulation frequency of the sensor can be set to 20-50MHz to obtain a higher depth resolution. By controlling the emission power and pulse width of the infrared light source, a depth image with a high signal-to-noise ratio and low distortion can be obtained.

[0048] like Figure 2As shown, the collected visible light image and depth image are preprocessed; when the collected visible light image is preprocessed, the visible light image with a size of WxH is uniformly divided into MxN local sub-images with a size of wxh, wherein w=W / M and h=H / N. For example, for an image with a size of 1280x1024, 10x8 sub-images with a size of 128x128 can be divided. For each local sub-image, a sliding window with a size of kxk is used to slide pixel by pixel with a step of 1, and a kxk neighborhood centered on the current pixel is extracted as a local region. The value of k is usually an odd number, such as 3, 5, 7, etc., to ensure that the pixel position does not shift after filtering. For each kxk local region extracted, the variance σ 2 of all pixel values in the region is calculated as the local noise variance of the region. Let the pixel value in the region be I(i,j), i,j=1,2,....,k, then the local noise variance is: wherein is the mean value of the pixel value in the region. According to the estimated local noise variance σ 2 , the window size and threshold parameters of the median filter are adaptively set. For example, the window size can be set to wherein represents rounding down; and the threshold is set to 3σ. The parameters set are used to perform median filtering on each local region to remove high-frequency noise in the region, and a denoised local sub-image is obtained. All denoised local sub-images are spliced according to the original position to obtain a denoised complete visible light image I denoised , which still has a size of WxH.

[0049] I denoised is decomposed by Retinex to obtain an illumination map I illumination and a reflectance map I reflectance with a size of WxH, which satisfy: I denoised =I illumnation xI reflectance , wherein x represents pixel-level multiplication. Retinex decomposition is usually implemented by algorithms such as multi-scale Gaussian filtering or guided filtering. Specifically, in this embodiment, guided filtering is used for Retinex decomposition, I denoised is used as an input image p, which is converted to floating-point data and normalized to the range [0, 1]. A guide image g is constructed, which can usually directly use the input image p as the guide image, i.e., g=p. The guide image p can also be preprocessed, such as downsampling, smoothing, etc., to obtain a simplified guide image. The window size r and the regularization parameter ε of the guided filtering are set. r determines the local range of the filtering, and is usually valued at 10-30; ε is used to control the smoothing degree of the filtering, and is usually valued at 0.01-0.1. The mean filtering is performed on the guide image g and the input image p to obtain a local mean map μ gand μ p :μ g =f mean (g,r), μ p =f mean (p,r), where f mean Represents the mean filter function, r is the window size. Calculate the local covariance σ of the guidance image g and the input image p gp and local variance σ gg :σ gp =f mean (g×p,r)-μ g ×μ p ; σ gg =f mean (g×g,r)-μ g ×μ p . Calculate the linear coefficients a and b: b=μ p -a×μ g ; Among them, ε is the regularization parameter used to prevent the denominator from being zero. The linear coefficients a and b are mean filtered to obtain the filtered coefficients and Calculate the output image q: The output image q is used as the estimated illumination map I illumnation , and restore it to an integer type in the range of [0, 255]. According to Retinex theory, the reflectivity map I reflectance The input image I denoised Divide by light map I illumnation get: To avoid division by zero errors, you can add a small positive value, such as 1e-6, to the denominator. reflectance Restore it to an integer type in the range of [0, 255] to obtain the final reflectance map.

[0050] Using linear stretching algorithm to illumnation Perform dynamic range expansion to expand the pixel values ​​from the original range [I min ,I max ] is mapped to [0, 255] to obtain the enhanced illumination map I illumnation_enhanced :

[0051]

[0052] Using guided filtering algorithm to analyze the reflectivity image I reflectance Smoothing is performed to remove noise and artifacts in the image and obtain the smoothed reflectivity image I illumnation_smoothed Specifically, in this embodiment, the denoised visible light image I denoised As the guide image g, the reflectivity map I reflectanceAs input image p, and convert both to floating-point data. Set the window size r of the guided filter and the regularization parameter ε. Generally, the value of r is larger than the window size used in Retinex decomposition to achieve a larger range of smoothing effect, such as 30-50; the value of ε can be the same as that in Retinex decomposition, such as 0.01 to 0.1. Mean filtering is performed on the guide image g and the input image p to obtain the local mean image μ g and μ p : μ g = f mean (g, r); μ p = f mean (p, r); where f mean represents the mean filtering function, and r is the window size. Calculate the local covariance σ gp and the local variance σ gg of the guide image g and the input image p: gp = f mean (g × p, r) - μ g × μ p ; σ gg = f mean (g × g, r) - μ g × μ g ; Calculate the linear coefficients a and b: b = μ p - a × μ g , where ε is the regularization parameter to prevent the denominator from being zero. Mean filtering is performed on the linear coefficients a and b to obtain the filtered coefficients and Calculate the output image q: Take the output image q as the smoothed reflectance image I illumnation_smoothed , and restore it to an integer type in the range [0, 255]. In the process of guided filtering, the guide image g provides the local structure information of the image to guide the filtering process of the input image p. By calculating the local mean and variance, the coefficients of the filter can be adaptively adjusted to suppress noise and artifacts while preserving the edges and textures of the image. Compared with simple mean filtering or Gaussian filtering, guided filtering can better preserve the details and structure of the image, and by setting appropriate window size r and regularization parameter ε, the smoothing degree and detail preservation degree of the filter can be balanced.

[0053] As shown in Figure 3 , the collected depth image is preprocessed, including: first, the depth image is distorted, and the internal and external parameters of the infrared depth sensor are obtained through a calibration board or other methods, such as Zhang Zhengyou calibration method. The internal parameters include focal length (f x , f y ), principal point coordinates (cx ,c y ) and radial distortion coefficients (k1, k2, k3), tangential distortion coefficients (p1, p2), etc. The external parameters include the rotation matrix R and translation vector t of the depth sensor relative to the world coordinate system. Calculate the three-dimensional coordinates of the pixel. For each pixel (u, v) in the depth image, calculate the three-dimensional coordinates (X, Y, Z) of the pixel in the depth sensor coordinate system according to its corresponding depth value d. Assume that the size of the depth image is (width, height), and the coordinate range of the pixel (u, v) is [0, width-1] and [0, height-1]. First, convert the pixel coordinates (u, v) to normalized coordinates (x, y): Then, according to the depth value d and the normalized coordinates (x, y), the three-dimensional coordinates (X, Y, Z) of the pixel are calculated: Z = d, X = x × Z, Y = y × Z; distortion correction is performed, and the three-dimensional coordinates (X, Y, Z) of each pixel are corrected using the radial distortion coefficients (k1, k2, k3) and the tangential distortion coefficients (p1, p2). Calculate the radial distortion (d rx ,d ry ):r 2 =X 2 +Y 2 , d rx =X×(1+k1r 2 +k2r 4 +k3r 6 ), d ry =Y×(1+k1r 2 +k2r 4 +k3r 6 ); calculate the tangential distortion (d tx ,d ty ):d tx =2p1XY+p2(r 2 +2X 2 ), d ty =p1(r 2 +2Y 2 )+2p2XY; add the radial distortion and tangential distortion to the original coordinates to obtain the corrected three-dimensional coordinates (X', Y', Z'): X'=X+d rx +d tx ;

[0054] Y'=Y+d ry +d ty ; Z'=Z. Reproject the corrected three-dimensional coordinates (X', Y', Z') onto the two-dimensional plane to obtain the distortion-corrected pixel coordinates (u', v'). Calculate the normalized coordinates (x', y'): x'=X' / Z', y'=Y' / Z', and convert the normalized coordinates to pixel coordinates: u'=x'×fx +c x , v'=y'×f y +c y For each pixel (u, v), its corresponding corrected coordinates (u', v') are used as the coordinates in the distortion-corrected depth image. Because the distortion-corrected coordinates (u', v') may be non-integer values, the depth values ​​need to be interpolated. Common interpolation methods include nearest neighbor interpolation and bilinear interpolation. Pixels whose corrected coordinates exceed the depth image size range can be discarded or filled with a default value (such as 0).

[0055] Smooth the distortion-corrected depth image. Denote the distortion-corrected depth image D, with a size of (width, height) and a pixel value range of [0, max_depth]. Denote the preprocessed visible light image I, also with a size of (width, height) and a pixel value range of [0, 255]. Fit the linear transformation coefficients a and b between the depth image D and the guidance image I using the least squares method. For each pixel position (i, j), establish the following system of equations: D(i, j) = a(i, j) × I(i, j) + b(i, j). Collect the equations for all pixel positions to form an overdetermined system, and use the least squares method to solve for the values ​​of a and b. Obtain the transformation coefficient matrices A and B, both of size (width, height). Set a threshold t and perform thresholding on the transformation coefficient matrices A and B. For elements in A greater than t, set the filter window size at the corresponding position to r1; otherwise, set it to r2 (r2 < r1). For elements in B that are greater than t, the filter window size at the corresponding position is set to r1; otherwise, it is set to r2 (r2<r1). Obtain the filter window size matrix R, which is of size (width, height) and element values ​​are r1 or r2. According to the filter window size matrix R, perform adaptive mean filtering on the depth image D. For each pixel position (i, j), the size of the filter window is determined based on the value of R(i, j) with the position as the center. Perform arithmetic averaging on the depth values ​​within the filter window to obtain the filtered depth value D'(i, j) Filter all pixel positions to obtain the filtered depth image D'. Calculate the gradient magnitude map G of the guide image I and use the Sobel operator or other edge detection operators. According to the gradient magnitude map G, set the weight matrix W of different regions of the depth image: W(i,j) = exp(-α×G(i,j)), where α is a parameter that controls the weight decay speed and can be adjusted according to actual needs. Perform weighted fusion on the filtered depth image D' and the original depth image D: D smooth (i, j) = W(i, j) × D'(i, j) + (1-W(i, j)) × D(i, j), and get the smoothed depth image Dsmooth The filter window sizes r1 and r2 can be adjusted based on the depth image's resolution and noise level, typically ranging from 5 to 15. The threshold t is determined by the distribution of the transform coefficients a and b and can be determined through histogram analysis or empirical analysis. The weight decay parameter α ranges from 0.1 to 1, with larger values ​​resulting in more pronounced smoothing of edge regions.

[0056] Fill the holes in the smoothed depth image and record the smoothed depth image as D smooth , size is (width, height), pixel value range is [0,max depth ]. The pre-processed visible light image is recorded as I, the size is also (width, height), and the pixel value range is [0, 255]. Define the hole pixel threshold as hole threshold , the depth value less than or equal to the threshold is considered as a hole. Traverse the depth image D smooth Each pixel D smooth (i,j). If D smooth (i,j) is less than or equal to hole threshold , then mark the pixel as a hole pixel, denoted as hole(i, j) = 1; otherwise, hole(i, j) = 0. Get the hole pixel marking matrix hole, the size is (width, height), and the element value is 0 or 1. For each hole pixel hole(i, j) = 1, select a set of candidate filling pixels in its neighborhood. Define the neighborhood radius as r, then the coordinate range of the candidate filling pixels is p∈[max(0,ir),min(height-1,i+r)] and q∈[max(0,jr),min(width-1,j+r)]. Mark the coordinates of the candidate filling pixels as (p, q), and the depth value is recorded as D smooth (p, q), the color value of the corresponding position in the visible light image is recorded as I(p, q). For each candidate filling pixel (p, q), calculate its geometric distance and color distance with the hole pixel (i, j) in the depth image and visible light image. The geometric distance calculation formula is: Color distance calculation formula (taking RGB color space as an example): Among them, I R ,I G ,I B Represent the red, green, and blue channels of the visible light image respectively.

[0057] Calculate the weight of each candidate filling pixel based on the geometric distance and color distance. Weight calculation formula: Among them, σ geo and σ colorThe influence of geometric distance and color distance is controlled respectively. The depth values ​​of the candidate filling pixels are weighted averaged to obtain the filling value of the hole pixel: For all hole pixels hole(i, j)=1, perform the above filling steps to obtain the filled depth value D filled (i, j). Assign the filled depth value to the corresponding position of the original depth image: D smooth (i,j)=D filled (i, j), if hole(i, j) = 1; get the depth image D after the hole is filled smooth Hole pixel threshold hole threshold It is set according to the characteristics of the depth image and the noise level, and the value range is generally 0 to 100. The selection of the neighborhood radius r depends on the size and distribution of the holes, and the value range is generally 1 to 5. geo and σ color The value of controls the influence of geometric distance and color similarity, which can be adjusted according to actual needs. The general value range is 1 to 10.

[0058] like Figure 4 As shown, based on the preprocessed depth image, a three-dimensional model of the low-light night vision scene is constructed by a three-dimensional reconstruction algorithm; including: the preprocessed depth image is recorded as D, the size is (width, height), and the pixel value range is [0, max_depth]. The preprocessed visible light image is recorded as I, the size is also (width, height), and the pixel value range is [0, 255]. The depth image D is divided into uniform grids, and each grid corresponds to a voxel in the three-dimensional model. The size of the voxel is defined as (voxel_size x ,voxel_size y ,voxel_size z ), which represent the size of the voxel in the x, y, and z directions respectively. Calculate the number of voxels (num_voxels x ,num_voxels y ,num_voxels z ): Create a 3D voxel grid V of size (num_voxels x ,num_voxels y ,num_voxels z ), the initial value is 0. For each voxel V(i, j, k), calculate the coordinates (x, y) and depth value d of its corresponding depth image pixel: d = D(x, y). Calculate the three-dimensional coordinates (X, Y, Z) of the voxel based on the depth value d and the voxel's position (i, j, k): The three-dimensional spatial coordinates (X, Y, Z) of the voxels are stored in the corresponding voxel positions V(i, j, k) to obtain an initial three-dimensional model.

[0059] The initial three-dimensional model is meshed using a triangular mesh algorithm to reduce the number of surfaces and vertices of the three-dimensional model. Common triangular mesh algorithms include edge collapse (Edge Collapse) and vertex clustering (Vertex Clustering). Set the target number of surfaces or the simplification ratio of mesh simplification, and iteratively simplify the three-dimensional model, merge adjacent voxels, and update the coordinates of the voxels. Obtain the three-dimensional model after mesh simplification, denoted as M simplified . The three-dimensional model M simplified after mesh simplification is surface reconstructed using a Poisson reconstruction algorithm. The gradient fields G x and G y of the depth image D are calculated using a Sobel operator or other gradient operator. The gradient fields G x and G y are interpolated to the vertex positions of the three-dimensional model M simplified as the constraint condition for surface reconstruction. The Poisson equation is constructed, and the scalar field F of the three-dimensional model surface is solved: The scalar field F is isosurface extracted to obtain the reconstructed three-dimensional model surface, denoted as M reconstructed .

[0060] The preprocessed visible light image I is mapped to the reconstructed three-dimensional model surface M reconstructed . For each vertex on the three-dimensional model surface, the corresponding visible light image coordinates (u, v) are calculated: The color value of the corresponding coordinates (u, v) in the visible light image I is assigned to the vertex of the three-dimensional model surface by interpolation; common interpolation algorithms include: nearest neighbor interpolation: select the integer pixel coordinates closest to (u, v), and directly use the color value of the pixel as the color of the vertex. Nearest neighbor interpolation is simple to calculate, but may cause texture sawtooth and block effects. Bilinear interpolation: consider the four integer pixel coordinates around (u, v), calculate the contribution weight of the color values of the four pixels to the interpolation point, and then weight average to obtain the final color value. Let the four pixel coordinates be (i, j), (i+1, j), (i, j+1), (i+1, j+1), and the corresponding color values be I(i, j), I(i+1, j), I(i, j+1), I(i+1, j+1), then the interpolation formula is: Where α = ui, β = vj, represents the position weight of the interpolation point relative to the four pixels. Bilinear interpolation can produce a smoother texture effect. Assign vertex color: Assign the interpolated color value to the vertex corresponding to the 3D model surface as the texture color of the vertex. Repeat for all vertices to completely map the visible light image I onto the 3D model surface. The 3D model M with texture information is obtained. textured , that is, the three-dimensional model of the low-light night vision scene. x ,voxel_size y ,voxel_size z ) is set based on the depth image resolution and scene size, typically ranging from 1 to 10. The focal length is set based on the depth camera's intrinsic parameters. The target number of faces or simplification ratio for mesh simplification is set based on the complexity of the 3D model and rendering performance requirements, typically ranging from 10% to 50% of the original number of faces.

[0061] Using the generative adversarial network GAN, the three-dimensional model is fused with the collected visible light image to generate an enhanced night vision image. The three-dimensional model of the low-light night vision scene is input as a condition into the generator of the generative adversarial network GAN: the three-dimensional model of the low-light night vision scene is denoted as C, and its size is (width, height, depth), which represents the width, height and depth of the three-dimensional model. The generator G receives the random noise vector z and the conditional input C, and maps them to the night vision image space through the decoder network. The output of the generator G is denoted as G(z, C), which represents the generated night vision image, and its size is (width, height, channels), which represents the width, height and number of color channels of the generated image. The conditional input C is input into the generator G to obtain the initial night vision image I init , whose size is the same as G(z, C). The collected visible light image is input as a real sample into the discriminator of the Generative Adversarial Network GAN: The collected visible light image is recorded as I real , whose dimensions are (width, height, channels), representing the width, height and number of color channels of the image. The discriminator D receives the visible light image I real , extract the features of its texture and noise distribution through the encoder network. The output of the discriminator D is recorded as D(I real ), which represents the discriminator’s score on the authenticity of the input image. Its value range is [0, 1]. The larger the value, the more realistic the discriminator thinks the input image is.

[0062] In the adversarial learning between the generator and the discriminator, the generator is constrained by the 3D model, and the discriminator's judgment of the generated image is used as the optimization goal: Define the input of the generator G: random noise vector z: z is a random noise vector of fixed dimension, usually sampled from the standard normal distribution N(0, 1). Let the dimension of z be d z ,Right now Where R represents the real space. Conditional input C: C is the three-dimensional model of the low-light night vision scene, which represents the conditional constraints of the generator. The size of C is Represents the width, height, and depth of the 3D model. C can be mapped to the same dimension d as the random noise vector z through a convolutional neural network or a fully connected layer. z The forward propagation process of the generator G: The random noise vector z and the conditional input C are input to the generator G. The generator G maps z and C to the night vision image space through a series of transposed convolution layers, upsampling layers and activation functions. The output of the generator G is G(z, C), which represents the generated night vision image, and its size is (width G ,height G , channels), which represents the width, height and number of color channels of the generated image. G(z, C) can be expressed as: G(z, C) = G n (G n-1 (...(G1(z,C'))...)), where G i Denotes the i-th layer of the generator, and n is the total number of layers of the generator. Forward propagation process of the discriminator D: The generated night vision image G(z, C) is input into the discriminator D.

[0063] The discriminator D extracts the features of G(z, C) through a series of convolutional layers, downsampling layers, and activation functions, and outputs a scalar value. The output of the discriminator D is denoted as D(G(z, C)), which represents the discriminator's score on the authenticity of the generated image G(z, C). The value range is [0, 1]. The larger the value, the closer the discriminator thinks G(z, C) is to the real image. D(G(z, C)) can be expressed as: D(G(z, C)) = D m (D m-1 (...(D1(G(z,C)))...)), where D i Represents the i-th layer of the discriminator, and m is the total number of layers of the discriminator. The optimization goal of the generator G: The optimization goal of the generator G is to maximize the authenticity score D(G(z, C)) of the generated image G(z, C) by the discriminator D. Let the optimization goal of the generator G be L G , which can be expressed as: L G= -log(D(G(z,C))), where log represents the natural logarithm function, which is used to map D(G(z,C)) to the logarithmic space to facilitate the calculation of the gradient. The goal of the generator G is to minimize L G , that is, maximize D(G(z, C)). Parameter update of generator G: According to the optimization target L of generator G G , calculate the gradient of the generator parameters where θ G Represents the parameter set of the generator. Using the back propagation algorithm, the gradient Backpropagate to each layer of the generator and update the parameters θ of the generator according to the gradient G Common optimization algorithms include stochastic gradient descent (SGD) and adaptive moment estimation (Adam). The learning rate is α, which represents the step size of each parameter update. The parameter update formula can be expressed as: Among them, ← represents the assignment operation, Represents the dot product operation. The generator G takes the random noise vector z and the conditional input C as input and generates the night vision image G(z, C). G(z, C) is input into the discriminator D to obtain the discriminant result D(G(z, C)). The generator G uses the maximization of D(G(z, C)) as the optimization goal, calculates the gradient through the backpropagation algorithm, and updates the generator parameter θ G , making the generated night vision images more realistic and difficult for the discriminator to identify as fake images. This process is repeated until the preset number of iterations is reached or a certain convergence condition is met. Ultimately, the generator G is able to generate high-quality, realistic night vision-enhanced images given a random noise vector z and the conditional constraint C.

[0064] The reconstruction loss term is expressed as follows: Among them, G represents the generator, z represents the random noise vector, C represents the constructed three-dimensional model of the low-light night vision scene, and I init represents the initial night vision image obtained by decoding C into the generator G, and ||*||1 represents the L1 norm. represents the gradient operator, represents the gradient map of the night vision image generated by the generator, represents the gradient map of the initial night vision image, λ g is the weight coefficient of gradient loss, which is used to balance the ratio of pixel loss and gradient loss.

[0065] The perceptual loss term is expressed as follows: Where F represents a pre-trained convolutional neural network, which is used to extract multi-scale convolution features of the image. i Represents the feature map extracted by the i-th convolutional layer, i∈{1, 2, ..., N}, N is the total number of convolutional layers. realRepresents the collected visible light image, and G(z, C) represents the night vision image generated by the generator. i is the weight coefficient of the perceptual loss of the i-th layer, which is used to balance the importance of features in different convolutional layers; L style Represents the style loss, which is used to measure the difference between the generated image and the real image in terms of texture, structure and other style features. Its expression is: Among them, G j Represents the Gram matrix calculated from the convolutional feature map, which is used to capture the correlation and statistical information between feature maps. j∈{1, 2, ...., M}, where M is the number of convolutional layers used to calculate the Gram matrix. ||*||1 represents the L1 norm. λ s is the weight coefficient of style loss.

[0066] The adversarial loss term is expressed as follows: Where D represents the discriminator, G represents the generator, z represents the random noise vector, and C represents the constructed three-dimensional model of the low-light night vision scene. z represents the expectation of the random noise vector z, log represents the natural logarithm function, and D(G(z, C)) represents the realism score obtained by inputting the night vision image generated by the generator into the discriminator. The value range is [0, 1], with larger values ​​indicating that the discriminator considers the generated image more realistic. The second term is the gradient penalty term, which is used to constrain the gradient norm of the discriminator to be close to 1, improving training stability and generated image quality. Represents the interpolation point between the real image and the generated image, through the formula Calculated, where α is a random interpolation coefficient between [0, 1], I real Represents the collected visible light image. Indicates that the discriminator is at the interpolation point The gradient at λ, ||*||2 represents the L2 norm. gp is the weight coefficient of the gradient penalty term, which is used to balance the adversarial loss and gradient penalty. Represents the interpolation point expectations.

[0067] The loss function of the generator is expressed as follows: L G =λ recon ×L recon +λ percep ×L percep +λ adv ×L adv , where λ recon ,λ percep ,λ adv Represents the weight coefficient. Discriminator data processing: The discriminator D receives two types of image data: the real visible light image I real, sampled from the visible light image dataset. The night vision image G(z, C) generated by the generator G is obtained by inputting the random noise vector z and the conditional input C. The task of the discriminator D is to determine whether the input image is a real image or a generated image, that is, to perform a binary classification task. The loss function L of the discriminator D is D It includes three items: Log-likelihood loss of real image: L real = -log(D(I real )), where D(I real ) represents the discriminator's response to the real image I real The log-likelihood loss of the generated image: L fake = -log(1-D(G(z,C))), where D(G(z,C)) represents the discriminator's score on the authenticity of the generated image G(z,C). Gradient penalty term: Among them, I interpolated is the real image I real and the interpolated image between the generated image G(z, C), λ gp is the weight coefficient of the gradient penalty term. The total loss L of the discriminator D D For: L D =L real +L fake +L gp .

[0068] Training of generator G and discriminator D: Generator G and discriminator D are trained alternately, that is, the generator and discriminator are trained in sequence. The goal of generator G is to minimize the total loss L G :L G =L recon +L percep +L adv , where L recon To reconstruct the loss, L percep is the perceptual loss, L adv is the adversarial loss. The goal of the discriminator D is to minimize the total loss L D :L D =L real +L fake +L gp The training process is as follows: Sample a batch of real images I from the visible light image dataset real , batch size is batch size . Sample a batch of noise vectors z from a random noise distribution (such as Gaussian distribution), with dimension noise dim . Input the noise vector z and the conditional input C into the generator G to get a batch of generated images G(z, C). real And generate the image G(z, C) and input it into the discriminator D, and calculate the loss L of the discriminator D. Calculate the gradient of the discriminator D through the back propagation algorithm and update the parameters of the discriminator D. Input the noise vector z and the conditional input C into the generator G to obtain a batch of generated images G(z, C). Calculate the reconstruction loss L of the generator recon , Perceptual Loss L percep and adversarial loss L adv , and weighted sum to get the total loss L of the generator G The gradient of the generator G is calculated through the back-propagation algorithm, and the parameters of the generator G are updated. Repeat the above steps until the preset number of iterations epoch_num is reached or the generated image quality meets the requirements.

[0069] Generate enhanced night vision images: After training, fix the parameters of the generator G. For a given 3D model C of a low-light-level night vision scene, sample a noise vector z from a random noise distribution. Feed the noise vector z and the conditional input C into the generator G. Through forward propagation, generate the enhanced night vision image G(z, C). Output the enhanced night vision image G(z, C) as the final generated result.

Claims

1. A method for generating enhanced information for low-light-level night vision goggles, characterized in that: include: Use low-light-level night vision goggles to collect visible light images of low-light-level night vision scenes, and use infrared depth sensors to collect depth images of low-light-level night vision scenes; Preprocessing the collected visible light images and depth images; Based on the pre-processed depth image, a 3D model of the low-light-level night vision scene is constructed using a 3D reconstruction algorithm. Using generative adversarial networks (GANs), the 3D model is fused with the collected visible light images to generate enhanced night vision images. Preprocess the collected visible light images, including: The visible light image collected by the low-light-level night vision goggles is divided into blocks to obtain multiple local sub-images; According to each local sub-image, the local area is extracted pixel by pixel on the sub-image by sliding the window; For each extracted local area, the variance of the pixel values ​​in the local area is counted as the local noise variance; According to the local noise variance, the parameters of the median filter are set; the corresponding local area is subjected to median filtering using the median filter with the set parameters; all local areas are subjected to median filtering to obtain a denoised visible light image; Perform Retinex decomposition on the denoised visible light image to obtain an illumination map reflecting the image brightness distribution and a reflectivity map reflecting the image; The range of the illumination map is expanded using a stretching algorithm to enhance the image brightness contrast and obtain an enhanced illumination map; The reflectivity map is smoothed using a guided filtering algorithm to obtain a smoothed reflectivity map; Multiply the enhanced illumination map and the smoothed reflectance map at the pixel level to obtain the preprocessed visible light image; Among them, the generative adversarial network (GAN) is used to fuse the 3D model with the collected visible light image, including: The 3D model of the low-light-level night vision scene is input as a condition into the generator of the Generative Adversarial Network (GAN). The generator generates an initial night vision image by decoding the geometric structure and texture mapping of the 3D model. The collected visible light image is input as a real sample into the discriminator of the Generative Adversarial Network (GAN). The discriminator obtains the features of the visible light image by encoding the texture and noise distribution of the visible light image. In the adversarial learning between the generator and the discriminator, the generator is constrained by the 3D model and the discriminator's judgment of the generated image is used as the optimization target. The generator parameters are updated through backpropagation to generate enhanced night vision images. A reconstruction loss term is set in the generator's loss function. By constraining the difference in pixel space between the night vision image generated by the generator at the current iteration and the initial night vision image, the generator learns the geometric structure and lighting prior information of the 3D model. A perceptual loss term is set in the generator's loss function. A pre-trained convolutional neural network is used to extract the multi-scale convolutional features of the night vision image generated by the generator at the current iteration and the collected visible light image. The L1 distance between the two image features in the feature space composed of the convolutional features is calculated, allowing the generator to learn the texture and noise distribution of the visible light image. An adversarial loss term is set in the generator's loss function. The night vision image generated by the generator in the current iteration step is input into the discriminator for true or false discrimination, and the discrimination result of the discriminator is used as the generator's loss. By alternately training the generator and discriminator, the reconstruction loss, perception loss, and adversarial loss of the generator are minimized, while the discriminator's binary classification accuracy for visible light images and night vision images generated by the generator in the current iteration step is maximized, so that the parameters of the generator and discriminator reach Nash equilibrium, and the generator after adversarial learning is obtained.

2. The method for generating enhanced information for low-light-level night vision goggles according to claim 1, characterized in that: Preprocess the acquired depth image, including: The depth image distortion is corrected using the pre-calibrated internal and external parameters of the infrared depth sensor. The three-dimensional coordinates of each pixel in the depth image are corrected using the radial and tangential distortion coefficients to eliminate the nonlinear distortion introduced by the imaging process and obtain a distortion-corrected depth image. The depth image after distortion correction is used as the input image, the preprocessed visible light image is used as the guide image, and the depth image is smoothed using the guided filtering algorithm to obtain a smoothed depth image; The smoothed depth image and the preprocessed visible light image are used as input, and the joint bilateral filtering algorithm is used to fill the holes in the depth image. The geometric distance between the depth images and the color similarity constraints between the visible light images are used to fill the hole areas in the depth image to obtain the preprocessed depth image.

3. The method for generating enhanced information for low-light-level night vision goggles according to claim 2, characterized in that: The depth image is smoothed using a guided filtering algorithm to obtain a smoothed depth image, including: Using the guided filtering algorithm, the linear transformation coefficients between the input depth image and the guided image are fitted by the least squares method; For areas where the linear transformation coefficient is greater than the threshold, a mean filter with a filter window greater than the threshold is used for filtering, whereas a mean filter with a filter window less than the threshold is used for filtering to obtain a filtered depth image. Calculate the gradient magnitude of the guide image and set the weights of different regions of the depth image based on the gradient magnitude; According to the set weights and the filtered depth image, weighted fusion is performed to obtain a smoothed depth image.

4. The method for generating enhanced information for low-light-level night vision goggles according to any one of claims 1 to 3, characterized in that: The 3D model of the low-light-level night vision scene is constructed using a 3D reconstruction algorithm, including: Taking the preprocessed depth image as input, the depth image is divided into uniform grids, each grid corresponds to a voxel in the 3D model; According to the depth value of the depth image pixel corresponding to each voxel, the three-dimensional space coordinates of each voxel are calculated to obtain the initial three-dimensional model; Use triangular mesh algorithm to simplify the mesh of the initial three-dimensional model; The surface of the 3D model after mesh simplification is reconstructed using the Poisson reconstruction algorithm. The surface shape of the 3D model is constrained by the gradient information of the depth image to obtain the reconstructed 3D model. The preprocessed visible light image is mapped to the reconstructed 3D model surface, and the texture information of the 3D model is added through the visible light image to obtain a 3D model of the low-light-level night vision scene.

5. The method for generating enhanced information for low-light-level night vision goggles according to claim 1, characterized in that: The reconstruction loss term is expressed as follows: Among them, G represents the generator, z represents the random noise vector, and C represents the constructed three-dimensional model of the low-light night vision scene. represents the initial night vision image obtained by decoding C into the generator G, represents the L1 norm; represents the gradient operator, represents the gradient map of the night vision image generated by the generator, represents the gradient map of the initial night vision image, is the weight coefficient of the gradient loss.

6. The method for generating enhanced information for low-light-level night vision goggles according to claim 5, characterized in that: The perceptual loss term is expressed as follows: Where F represents the pre-trained convolutional neural network; Represents the feature map extracted by the i-th convolutional layer; represents the collected visible light image, represents the night vision image generated by the generator; is the weight coefficient of the perceptual loss of the i-th layer; represents the style loss, which is expressed as: in, Represents the Gram matrix calculated from the convolutional feature map; , M is the number of convolutional layers used to calculate the Gram matrix; represents the L1 norm; is the weight coefficient of style loss.

7. The method for generating enhanced information for low-light-level night vision goggles according to claim 6, characterized in that: The adversarial loss term is expressed as follows: Where D represents the discriminator, G represents the generator, z represents the random noise vector, and C represents the constructed three-dimensional model of the low-light night vision scene; represents the expectation of the random noise vector z, represents the authenticity score obtained by inputting the night vision image generated by the generator into the discriminator; Represents the interpolation point between the real image and the generated image; Indicates that the discriminator is at the interpolation point The gradient at represents the L2 norm; is the weight coefficient of the gradient penalty term; Represents the interpolation point expectations.

8. The method for generating enhanced information for low-light-level night vision goggles according to claim 7, characterized in that: The loss function of the generator is expressed as follows: in, represents the weight coefficient; The loss function of the discriminator is expressed as follows: Among them, x represents the real visible light image, represents the expectation of the true image distribution.

9. An enhanced information generation system for low-light-level night vision goggles, characterized in that: include: At least one processing unit; used to execute instructions to implement the enhanced information generation method for low-light-level night vision goggles according to any one of claims 1 to 8.

Images