Passive night vision full-color video image enhancement method and system based on AI learning

Abstract

The invention relates to the technical field of computer vision and video image processing, and discloses a passive night vision full-color video image enhancement method and system based on AI learning, and the method comprises the steps: constructing a Poisson to Gaussian mixed noise model through sensor metadata, generating a signal-to-noise ratio confidence map, and quantifying the physical reliability of pixels; constructing an optical flow energy functional through a signal-to-noise ratio weighted data item and a semantic boundary constraint regular item in combination with a semantic label graph, and calculating a high-robustness optical flow field; in a time-space fusion stage, taking the signal-to-noise ratio confidence map as a gating condition, and dynamically adjusting a time domain recursion fusion proportion of historical features and current features; and finally, driving an adaptive normalization unit by using the semantic tag graph, and retrieving a semantic chrominance priori library to restore the inherent physical color of the object. According to the method, physical noise and scene semantic priori are utilized, the problems of random noise interference, motion ghosting and color loss under low illumination can be solved, and clear passive full-color night vision imaging with real color is achieved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and video image processing technology, specifically to a passive night vision full-color video image enhancement method and system based on AI learning. Background Technology

[0002] Low-light night vision technology has crucial application value in security monitoring, driver assistance, and all-weather observation. While traditional infrared imaging technology can detect targets in dark environments, the resulting images lack color information and have blurred texture details, failing to meet the human eye's need for full-color, high-definition video. Low-light video enhancement technology based on digital signal processing and deep learning is gradually becoming mainstream, reconstructing scene details under low-light conditions through algorithms.

[0003] In extremely low-light environments without auxiliary light sources, video enhancement technology faces multiple physical bottlenecks and algorithmic defects. Image sensors generate high-intensity photon shot noise when photons are scarce, and the noise intensity exhibits a complex statistical correlation with the physical signal. Existing video enhancement algorithms often ignore the physical characteristics of sensor noise, making it difficult to distinguish between weak texture signals and random noise during the feature extraction stage. This leads to the temporal recursive network erroneously accumulating noise features when fusing historical frame information, resulting in flickering and artifacts in the output video.

[0004] Meanwhile, high-intensity noise masks the gradient texture of the image, disrupting the motion correspondence between adjacent video frames. Optical flow estimation algorithms struggle to calculate accurate motion vectors without clear texture guidance, especially in the edge regions of dynamic objects. Incorrect motion compensation leads to severe motion blur and ghosting in the synthesized video. Furthermore, due to insufficient photon count, sensors struggle to capture effective chromaticity components, causing the original video data to degenerate almost entirely into grayscale images. Existing image restoration methods lack an understanding of the scene's semantic attributes, relying solely on data-driven methods for blind coloring, resulting in severe color distortion and saturation deviations in the enhanced video, failing to accurately reproduce the scene's natural colors.

[0005] Therefore, this invention proposes a passive night vision full-color video image enhancement method and system based on AI learning to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a passive night vision full-color video image enhancement method and system based on AI learning. It solves the problems of video noise flickering caused by high-intensity random noise interference in extremely low illumination environments, motion ghosting caused by optical flow matching errors due to missing texture details, and color loss and distortion caused by the light sensitivity limitations of physical sensors.

[0007] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of this invention provides a passive night vision full-color video image enhancement method based on AI learning, comprising the following steps: S1. Acquire continuous low-light video frames and synchronized sensor metadata, and generate a signal-to-noise ratio confidence map based on a physical noise model using the sensor metadata; the generation process of the signal-to-noise ratio confidence map includes reading the analog gain parameter and readout noise variance parameter from the sensor metadata, substituting the analog gain parameter and the readout noise variance parameter into a Poisson-Gaussian mixed noise model, and quantifying the physical signal confidence of each pixel by calculating the ratio of the expected signal value to the noise standard deviation of each pixel in the low-light video frame.

[0008] S2. Semantic segmentation is performed on the low-light video frames to generate a semantic label map. The signal-to-noise ratio confidence map and the semantic label map are used as joint constraints to calculate the optical flow field between adjacent video frames. The optical flow field calculation process constructs an optical flow energy functional that includes weighted data terms and semantic regularization terms. The signal-to-noise ratio confidence map is used to perform pixel-by-pixel weighting on the weighted data terms to reduce the weight of low signal-to-noise ratio regions in the motion matching process.

[0009] The semantic label map is used to set the boundary conditions of the semantic regularization term. When pixels belong to the same semantic category, a positive smoothing control coefficient is set. When pixels belong to different semantic categories, the smoothing control coefficient is set to zero. This prevents motion ghosting artifacts by blocking smooth diffusion across semantic categories.

[0010] S3. Based on the optical flow field, the historical frame features are spatially aligned, and the fusion gate coefficient is calculated in combination with the signal-to-noise ratio confidence map. The spatially aligned historical features and the current frame features are recursively fused to generate temporal fusion features. The recursive fusion process uses a convolutional neural network encoder to generate the current frame feature tensor, and uses the optical flow field to spatially resample the hidden layer feature states of the previous frame to generate an aligned historical feature tensor. The signal-to-noise ratio confidence map is used as an independent feature channel and concatenated with the current frame feature tensor and the aligned historical feature tensor. A time-update gating map is calculated through convolutional neural network branches. The time-update gating map is used to dynamically determine the proportion of historical information retained and current frame information accepted at different spatial locations, and the historical information is used to suppress random noise.

[0011] S4. The temporal fusion features are modulated using the semantic tag map, and the image is reconstructed by combining semantic chromaticity prior knowledge, outputting a full-color enhanced video frame. The image reconstruction process includes a luminance reconstruction branch and a chromaticity reconstruction branch. The luminance reconstruction branch restores the luminance map of a single channel. The chromaticity reconstruction branch introduces the semantic tag map as a conditional control variable, and performs affine transformation modulation on the feature layer through a semantic adaptive normalization unit to generate a chromaticity map that conforms to the physical properties of the object.

[0012] Further, a pre-set semantic-to-chroma prior library is retrieved. Correction is performed by minimizing the difference between the average chroma of the object region in the chroma map and the mean chroma center value under standard illumination in the prior library. Finally, the luminance map and the corrected chroma map are merged, and a full-color video is output through inverse color space conversion.

[0013] A second aspect of the present invention provides a passive night vision full-color video image enhancement system based on AI learning, comprising: The signal-to-noise ratio (SNR) feature extraction module is configured to generate an SNR confidence map using sensor metadata. The SNR feature extraction module analyzes the analog gain parameter and readout noise variance parameter in the sensor metadata, calculates the expected signal value and noise standard deviation of each pixel using a Poisson-Gaussian mixed noise model, and generates the SNR confidence map that reflects the physical noise distribution.

[0014] The semantic-assisted optical flow calculation module is configured to calculate the optical flow field between adjacent video frames using the signal-to-noise ratio confidence map and the semantic label map. The semantic-assisted optical flow calculation module constructs an optical flow energy functional that includes weighted data terms and semantic regularization terms. It uses the signal-to-noise ratio confidence map to weight the data term errors in the motion matching process and uses the semantic label map generated by semantic segmentation to control the smooth boundary of the regularization terms, ensuring that motion vectors are smooth within the same semantic region but break at the semantic boundary.

[0015] The spatiotemporal feature recursive fusion module is configured to recursively fuse historical frame features and current frame features using the optical flow field and the signal-to-noise ratio confidence map; the spatiotemporal feature recursive fusion module uses the optical flow field to spatially resample the hidden layer feature state of the previous frame to achieve feature alignment; at the same time, it uses the signal-to-noise ratio confidence map as the input parameter of the gating network to calculate the time-updated gating map used to adjust the cumulative ratio of historical features, and generates high signal-to-noise ratio time fusion features through weighted fusion; The chroma consistency reconstruction module is configured to reconstruct full-color enhanced video frames using the semantic label map and semantic chroma prior knowledge. The chroma consistency reconstruction module includes a luminance decoding path and a chroma decoding path. The chroma decoding path uses the semantic label map to drive a semantic adaptive normalization unit to restore the chroma distribution of the feature layer, and combines the statistical mean in the semantic to chroma prior library to perform global correction on the generated chroma, and finally outputs a color-accurate image.

[0016] This invention provides a passive night vision full-color video image enhancement method and system based on AI learning. It has the following beneficial effects: 1. This invention utilizes sensor metadata to construct a physical noise model and generate a signal-to-noise ratio (SNR) confidence map. This SNR confidence map is then introduced into the optical flow calculation and spatiotemporal feature fusion process. By reducing the weight of data items in low SNR regions within the optical flow energy functional and calculating the fusion gating coefficients using the SNR confidence map in the recursive fusion network to suppress the accumulation of noise features, this invention addresses the problem of motion estimation failure and temporal artifact spread caused by random noise interference in low-illuminance environments, thereby improving the SNR and temporal stability of night vision videos.

[0017] 2. This invention introduces a semantic label map as a regularization constraint in the optical flow estimation stage, and dynamically adjusts the boundary conditions of the smoothing term according to the semantic category to which the pixel belongs. By forcibly maintaining the smoothness consistency of motion vectors within the same semantic category, and blocking motion smoothing constraints at the intersection of different semantic categories, it can overcome the problem of optical flow matching blurring caused by the loss of texture details in low-light scenes, eliminate the motion ghosting phenomenon generated by the edges of dynamic objects, and ensure accurate spatial alignment between video frames.

[0018] 3. This invention utilizes a semantic label graph to drive an adaptive normalization unit to modulate the feature layer, and combines a pre-set semantic chromaticity prior library to perform chromaticity correction on the generated image. By mapping the semantic category information of objects to the corresponding chromaticity distribution features and minimizing the difference from the statistical chromaticity center in the Lab color space, it can overcome the limitation of insufficient photosensitivity of physical sensors under low light conditions and robustly recover the natural colors of the scene in a completely passive dark environment. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] See attached document Figure 2 This invention provides a passive night vision full-color video image enhancement system based on AI learning. The low-light video enhancement system, based on physical perception and semantic guidance, operates within an electronic device equipped with image acquisition and tensor computation capabilities. The electronic device includes an image sensor, an image signal processor, and a neural network computing unit, all physically connected in sequence.

[0022] The image sensor is used to acquire light signals from the external environment and convert these signals into raw image data in Bayer array format via photoelectric conversion. The raw image data is transmitted to the image signal processor via a high-speed serial interface. Simultaneously, the image sensor outputs metadata corresponding to each frame of the raw image data. This metadata explicitly records the analog gain value, digital gain value, and the sensor's inherent readout noise standard deviation parameter at the time of acquisition for the current frame.

[0023] The image signal processor receives the raw image data and performs black level subtraction and de-mosaic interpolation on it to generate a low-light video frame sequence in red-green-blue format. The low-light video frame sequence preserves the brightness and noise distribution characteristics of the original low-light environment.

[0024] The low-light video frame sequence generated by the image signal processor and the metadata output by the image sensor are transmitted to the neural network computing unit. The neural network computing unit is a graphics processing unit or a dedicated neural network accelerator. The neural network computing unit internally stores and executes a deep learning model. The deep learning model is logically divided into four cooperating processing modules: a signal-to-noise ratio feature extraction module, a semantic-assisted optical flow calculation module, a spatiotemporal feature recursive fusion module, and a chromaticity consistency reconstruction module.

[0025] The signal-to-noise ratio (SNR) feature extraction module is configured to receive gain values ​​and noise parameters from metadata and generate an SNR confidence map using a physical model. The semantic-assisted optical flow calculation module, connected to the SNR feature extraction module, receives the SNR confidence map as a constraint term to calculate the motion vector field between two frames. The spatiotemporal feature recursive fusion module uses the motion vector field to spatially align the features of historical frames and outputs the fused features of the current frame. The chroma consistency reconstruction module receives the fused features of the current frame and outputs the final full-color enhanced video frame.

[0026] All processing modules interact with each other via the internal data bus of the electronic device, ensuring that the pipelined processing of low-light video frames from input to enhanced full-color video frame output is completed within a predetermined time window.

[0027] See attached document Figure 1 This invention provides a passive night vision full-color video image enhancement method based on AI learning. It includes four sequentially executed and tightly coupled processing steps for converting an input low-light video frame sequence into a high signal-to-noise ratio full-color video frame sequence.

[0028] Step S1 executes the confidence generation step based on the physical model. The electronic device receives the low-light video frame at the current moment and synchronized sensor metadata. The method reads the analog gain parameter and readout noise variance parameter from the sensor metadata. Substituting the analog gain parameter and readout noise variance parameter into the Poisson-Gaussian noise distribution model, the expected signal value and noise standard deviation of each pixel in the low-light video frame are calculated. Based on the calculation results, a signal-to-noise ratio confidence map with the same resolution as the low-light video frame is generated. Each value in the signal-to-noise ratio confidence map quantitatively represents the confidence level of the pixel data at the corresponding location.

[0029] Step S2 involves performing a semantically constrained optical flow estimation step. The method performs semantic segmentation on low-light video frames, generating a semantic label map containing object category information. It then constructs an optical flow energy function that includes data and smoothing terms. During construction, the data terms are weighted pixel-by-pixel using a signal-to-noise ratio confidence map to reduce the influence of low signal-to-noise ratio regions on optical flow calculation. Simultaneously, the semantic label map is used to set boundary conditions for the smoothing term, preventing optical flow smoothing operations from crossing the boundaries of different semantic categories. By minimizing the energy function, the optical flow field describing pixel motion trajectories is solved.

[0030] Step S3 involves the method performing a physically-perceived recursive feature fusion step. The method uses the solved optical flow field to perform a spatial transformation operation on the hidden layer features of the previous frame stored in the buffer, obtaining spatially aligned historical features. The method calculates the fusion gating coefficients for the current frame based on the signal-to-noise ratio confidence map. Using these gating coefficients, the spatially aligned historical features and the convolutional features extracted from the current frame are weighted and summed to generate the temporal fusion features for the current time step. These temporal fusion features aggregate effective information from the temporal dimension and suppress irrelevant noise.

[0031] Step S4 involves performing a chromaticity consistency reconstruction step. The method inputs temporal fusion features into an image reconstruction network. The image reconstruction network maps these features to the Lab color space. Within the Lab color space, the method retrieves a pre-defined semantic chromaticity mean library and calculates the difference between the current pixel's chromaticity value and the statistical mean of its corresponding semantic category. By minimizing this difference, the generated image undergoes chromaticity correction. The method ultimately outputs a full-color enhanced video frame in red-green-blue format and updates the current temporal fusion features in a buffer memory as the basis for processing the next frame.

[0032] See attached document Figure 1 The signal-to-noise ratio (SNR) confidence map construction step is performed by the SNR feature extraction module. This module aims to provide pixel-level reliability weights for subsequent optical flow calculation and feature fusion steps. By introducing a physical noise model of photoelectric conversion, the SNR feature extraction module quantifies the signal quality of each pixel in low-light environments.

[0033] The signal-to-noise ratio (SNR) feature extraction module first receives the raw RAW domain image data output by the image signal processor. Each pixel value in the raw RAW domain image data is directly proportional to the number of photons accumulated by the sensor's photodiode during the exposure time. Simultaneously, the SNR feature extraction module reads metadata synchronized with the raw RAW domain image data frame. This metadata includes the system total gain parameter for the current frame. With reading noise variance parameter System total gain parameter It is the product of analog gain and digital gain.

[0034] The signal-to-noise ratio (SNR) feature extraction module models the original RAW domain image data based on a Poisson-Gaussian mixture noise model. In low-light imaging, photon shot noise follows a Poisson distribution, with its variance linearly related to signal intensity; readout noise follows a Gaussian distribution, with its variance independent of signal intensity. The SNR feature extraction module extracts the signal-to-noise ratio feature from any pixel in the original RAW domain image data. The total noise variance estimate of each pixel is calculated using the following formula. : ; In the formula, Represents pixels Intensity values ​​in the original RAW domain image data; This represents the total system gain parameter; This indicates that the noise variance parameter has been read. This represents an unbiased estimate of the number of incident photoelectrons.

[0035] After calculating the total noise variance estimate, the signal-to-noise ratio (SNR) feature extraction module constructs an SNR confidence map. The module uses the ratio between pixel intensity values ​​and the total noise variance estimate to map the confidence level of each pixel to the [0,1] interval. The SNR feature extraction module calculates the confidence level of each pixel using the following formula. The corresponding signal-to-noise ratio confidence value : ; In the formula, Represents pixels Intensity values ​​in the original RAW domain image data; This represents the sensitivity adjustment hyperparameter, which is used to linearly scale the noise tolerance according to the application scenario.

[0036] The signal-to-noise ratio (SNR) feature extraction module iterates through all pixels in the original RAW domain image data to complete the above calculations. After calculation, the SNR feature extraction module outputs an SNR confidence map with the exact same resolution as the original RAW domain image data. In the SNR confidence map, regions with values ​​close to 1 correspond to bright areas with low noise, while regions with values ​​close to 0 correspond to extremely low illumination areas where noise overwhelms the signal. The SNR confidence map is then transmitted to the semantic-assisted optical flow calculation module and the spatiotemporal feature recursive fusion module.

[0037] The semantic-assisted optical flow calculation module aims to solve the problems of photometric matching failure caused by noise and motion boundary blurring caused by low contrast in low-light environments. By introducing prior physical signal-to-noise ratio and semantic category information, it calculates dense optical flow fields for inter-frame alignment.

[0038] The semantic-assisted optical flow calculation module receives the previous frame image data, the current frame image data, and the signal-to-noise ratio confidence map generated in step one from the buffer memory. Simultaneously, the semantic-assisted optical flow calculation module performs semantic segmentation on the current frame image data, generating a semantic label matrix consistent with the image resolution. The semantic label matrix stores the object category index of each pixel.

[0039] See attached document Figure 1 The semantic-assisted optical flow calculation module constructs an optical flow energy functional that includes weighted data terms and semantic regularization terms. This module then uses a multi-scale iterative algorithm to minimize the optical flow energy functional, thereby solving for the optical flow field describing the pixel's motion trajectory.

[0040] For the weighted data items, the semantic-assisted optical flow calculation module uses the signal-to-noise ratio confidence map to weight the pixel matching error point by point. The semantic-assisted optical flow calculation module constructs the weighted data items according to the following formula. : ; In the formula, Represents the two-dimensional pixel coordinates of the image plane; This represents the optical flow field to be solved; Represents two-dimensional pixel coordinates The two-dimensional motion vector at the location; Representing coordinates The signal-to-noise ratio confidence value at the location; Represents the pixel grayscale value of the current frame image; This represents the grayscale value of the corresponding pixel in the previous frame after displacement by the motion vector. This represents the robust penalty function, used to measure the brightness difference of corresponding pixels between two frames.

[0041] For the semantic regularization term, the semantic-assisted optical flow calculation module constructs discontinuous smoothing constraints using the semantic label matrix. The semantic-assisted optical flow calculation module constructs the semantic regularization term according to the following formula. : ; In the formula, Represents the weight coefficient of the regularization term; Represents two-dimensional pixel coordinates The set of four neighboring regions; Indicates the coordinates of adjacent pixels within the neighborhood; It represents the square of the Euclidean distance between motion vectors of adjacent pixels; This represents the smoothing control coefficient determined by semantic relationships.

[0042] The semantic-assisted optical flow calculation module determines the smoothing control coefficients based on the semantic tag matrix. The value: when the two-dimensional pixel coordinates With coordinates When they belong to the same semantic category, the semantic-assisted optical flow calculation module will Set to a preset positive value to force the motion vectors of adjacent pixels to remain smooth and consistent; when the coordinates With coordinates When belonging to different semantic categories, the semantic-assisted optical flow calculation module will Setting it to zero removes the smoothing constraint on the motion vectors of adjacent pixels, allowing the optical flow field to exhibit discontinuous abrupt changes at the object's edge.

[0043] The semantic-assisted optical flow calculation module calculates the total energy. The motion vector distribution reaches its minimum value, and this distribution is output as the final optical flow field. The optical flow field is then transmitted to the subsequent spatiotemporal feature recursive fusion module.

[0044] The spatiotemporal feature recursive fusion module uses a recurrent neural network architecture to align historical features with optical flow and dynamically adjusts the temporal fusion weights in combination with physical signal-to-noise ratio confidence, thereby effectively accumulating video temporal information while suppressing motion blur and ghosting artifacts.

[0045] The spatiotemporal feature recursive fusion module receives the current low-light video frame, the signal-to-noise ratio confidence map generated in step one, the corrected optical flow field output in step two, and the hidden layer feature state of the previous frame stored in the buffer memory.

[0046] First, the spatiotemporal feature recursive fusion module uses the built-in convolutional neural network encoder to extract features from the current low-light video frame, generating the current frame feature tensor. Current frame feature tensor It is a high-dimensional feature map containing multiple channels, whose spatial resolution is consistent with or proportionally reduced from that of the input video frame.

[0047] See attached document Figure 1 Secondly, the spatiotemporal feature recursive fusion module performs feature space alignment. The spatiotemporal feature recursive fusion module reads the hidden layer feature state of the previous frame from the buffer memory. Using the corrected optical flow field right Spatial resampling (warping) is performed to generate aligned historical feature tensors. Spatial alignment operations eliminate feature misalignment caused by camera or object motion.

[0048] See attached document Figure 1 Furthermore, the spatiotemporal feature recursive fusion module performs a physically-aware gated fusion operation. To address the issue that traditional recursive networks cannot perceive physical noise intensity, the spatiotemporal feature recursive fusion module integrates the signal-to-noise ratio confidence map... As an independent feature channel, it is related to the current frame feature tensor. and aligned historical feature tensor Perform channel-dimensional splicing.

[0049] The spatiotemporal feature recursive fusion module computes the time-updated gating graph through a convolutional neural network branch. The calculation formula is as follows: ; In the formula, Indicates a channel splicing operation; Indicates the convolution operation; and These represent the kernel weights and bias parameters of the gated network, respectively, which are obtained through AI deep learning training; This represents the Sigmoid activation function, used to constrain the output value to the interval (0,1).

[0050] Time-updated gating graph Each value in the input determines how much historical information the system should retain and how much current frame information it should accept at the corresponding spatial location. Specifically, this is because the input includes a physical signal-to-noise ratio confidence map. When the signal-to-noise ratio confidence level is low (i.e., the noise is extremely high), the neural network will adaptively increase the weight of historical information and use the accumulated signal of historical frames to suppress the sudden noise of the current frame.

[0051] See attached document Figure 1 Finally, the spatiotemporal feature recursive fusion module updates the gating graph based on time. The spatiotemporal fusion feature tensor for the current moment is generated according to the following formula. : ; In the formula, This indicates element-wise multiplication.

[0052] After the calculation is completed, the spatiotemporal feature recursive fusion module performs a dual-output operation: the first output will... Write to the buffer memory, overwriting the state data of the previous frame, as the basis for recursive processing of the next frame; the second path will... The image is then transferred to the chromaticity consistency reconstruction module for final image restoration.

[0053] The chromaticity consistency reconstruction module aims to map the recursively fused high-dimensional depth features back to visualized full-color video frames, and utilize semantic category color prior knowledge to correct the color drift and grayscale distortion problems that are common in extremely low-light environments.

[0054] The chromaticity consistency reconstruction module receives the spatiotemporal fusion feature tensor. And synchronously read the semantic tag matrix .

[0055] The chromaticity consistency reconstruction module incorporates a dual-branch decoding architecture based on a deep convolutional neural network, including a luminance reconstruction branch and a chromaticity reconstruction branch. The module uses the CIELAB color space as the intermediate reconstruction target to realize luminance information. With chromaticity information , Decoupling process.

[0056] First, the brightness reconstruction branch applies to the spatiotemporal fusion feature tensor. A series of convolution and upsampling operations are performed to focus on restoring high-frequency texture details and overall contrast of the image, outputting a single-channel luminance map. Since the luminance channel is insensitive to color noise, the luminance reconstruction branch is mainly constrained by the gradient loss function.

[0057] Secondly, the chroma reconstruction branch performs semantically guided feature modulation operations. To ensure that the generated colors conform to the physical attributes of object categories (e.g., vegetation should be green, and the sky should be dark blue), the chroma reconstruction branch introduces a semantic label matrix. As a conditional control variable, each convolutional layer in the chroma reconstruction branch contains a semantically adaptive normalization unit.

[0058] The semantic adaptive normalization unit performs pixel-by-pixel affine transformation modulation on the chroma feature layer according to the following formula: ; In the formula, Indicates the pixel position index; and These represent the characteristic values ​​of the input and output of the normalized unit, respectively; and These represent the mean and standard deviation statistics of the input feature channels, respectively. Represents pixels Semantic category index at the location; and These are semantic mapping functions learned by a convolutional neural network, used to generate scale scaling and bias translation coefficients corresponding to semantic categories. The network can dynamically adjust the feature distribution of each pixel based on its object category, thereby driving the chromaticity information. , Generate color values ​​that conform to semantic priors.

[0059] In generating the initial chromaticity map Subsequently, the chromaticity consistency reconstruction module performs semantic chromaticity constraint correction. The module retrieves a pre-set semantic-to-chromaticity prior library, which stores the mean chromaticity center values ​​for different semantic categories under normal lighting. The module calculates the average chromaticity of various object regions in the current frame and fine-tunes it in the direction of minimizing chromaticity deviation to ensure the color reproduction accuracy of the night view image.

[0060] Finally, the chromaticity consistency reconstruction module will generate a luminance map. Compared with the corrected chromaticity diagram At the channel level, the merged data is converted from CIELAB color space to standard RGB color space using an inverse color space conversion algorithm, outputting a final high signal-to-noise ratio, color-accurate full-color enhanced video frame. Simultaneously, the system outputs the full-color enhanced video frame to the display terminal and updates the current processing status to the system cache.

[0061] The signal-to-noise ratio (SNR) estimation module in the AI-based passive night vision full-color video image enhancement system of this invention is a core interface component connecting the physical layer of the image sensor and the back-end artificial intelligence processing layer. The SNR estimation module is configured to execute the logic related to physical modeling and noise variance calculation in the aforementioned method steps. The SNR estimation module is typically implemented by a dedicated digital signal processing circuit (DSP) or a computational kernel in a graphics processing unit (GPU).

[0062] The signal-to-noise ratio estimation module mainly includes a data receiving and parsing unit, a noise variance calculation unit, and a confidence mapping unit.

[0063] The data receiving and parsing unit is directly connected to the image sensor or raw data buffer via a high-speed bus interface (such as a MIPI interface or DRAM bus). The data receiving and parsing unit is configured to read the raw RAW domain image data stream in real time. While reading the image pixel data, the data receiving and parsing unit synchronously parses the metadata accompanying each frame of the image. The metadata contains the analog gain parameters, digital gain parameters, black level correction value, and pre-calibrated sensor readout noise reference parameters for the current exposure frame. The data receiving and parsing unit standardizes these parameters and calculates the overall system gain coefficient. .

[0064] The noise variance calculation unit is connected to the data receiving and parsing unit. The noise variance calculation unit incorporates a physical noise model algorithm based on a Poisson-Gaussian mixture distribution. For each pixel coordinate in the input image, the noise variance calculation unit performs a linear operation based on the pixel intensity value and the parsed total system gain coefficient to estimate the signal-dependent noise component (i.e., photon shot noise) of that pixel. Simultaneously, it calculates the summed total noise variance estimate by combining the read noise baseline parameters. The noise variance calculation unit adopts a parallel single-instruction multiple-data (SIMD) architecture to ensure that the pixel-by-pixel variance calculation for high-resolution video frames meets real-time requirements.

[0065] The confidence mapping unit receives the total noise variance map and the original pixel intensity map output by the noise variance calculation unit. The confidence mapping unit primarily performs nonlinear mapping operations. It compares the pixel energy (signal power) with the total noise energy (noise variance) to generate a normalized signal-to-noise ratio confidence value. Internally, the confidence mapping unit includes a programmable sensitivity adjustment register for storing adjustment hyperparameters. This allows the system to dynamically adjust the sensitivity curve generated by the confidence level based on changes in external ambient illuminance.

[0066] The output of the signal-to-noise ratio (SNR) estimation module is connected to the semantic-assisted optical flow calculation module and the spatiotemporal feature recursive fusion module. Its output data product is a SNR confidence map that matches the original video frame resolution and bit depth. In the SNR confidence map, high-level values ​​on the data bus represent high SNR for the pixel at that location, indicating to the backend AI network that the features in this region are reliable; low-level values ​​represent noise at that location, indicating to the backend AI network that the weight of this region needs to be reduced.

[0067] In terms of physical implementation, the semantic optical flow correction module is deployed in the high-performance core of the neural network computing unit (NPU) and uses parallel floating-point computing power to process high-dimensional tensor data.

[0068] The semantic optical flow correction module includes a multi-scale feature extraction unit, a semantic perception unit, a correlation calculation unit, and a variational optimization unit in its logical architecture.

[0069] The multi-scale feature extraction unit receives the current frame image with confidence weights, output by the signal-to-noise ratio estimation module, and the previous frame image. The multi-scale feature extraction unit contains a Siamese CNN, which shares weight parameters. For each frame image, the multi-scale feature extraction unit constructs an image feature pyramid through successive convolution and downsampling operations. The feature pyramid typically contains 3 to 5 levels with different resolutions; the lower levels retain high-frequency texture information, while the higher levels encode abstract structural information, thus providing multi-receptive field support for handling large displacement motions.

[0070] The semantic awareness unit (SAU) is primarily responsible for scene parsing. It incorporates a lightweight semantic segmentation network (such as a simplified variant of UNet or DeepLab). It processes the current frame image in parallel and outputs a semantic label matrix through forward inference. Each element in the semantic label matrix identifies the object category (e.g., vehicle, pedestrian, road, sky) to which the corresponding pixel belongs. The semantic perception unit also extracts the semantic edge map, which is used as the blocking boundary signal for subsequent optical flow smoothing and transmitted to the variational optimization unit.

[0071] The correlation calculation unit is used to calculate the matching cost between feature maps of two frames. For each layer in the feature pyramid, the correlation calculation unit constructs an all-to-all or local correlation cost volume by calculating the dot product of feature vectors. The correlation cost volume quantifies the similarity between pixels in the current frame and pixels in the search region of the previous frame, and is the basic data structure for optical flow estimation.

[0072] The variational optimization unit receives the correlation cost volume, signal-to-noise ratio confidence map, and semantic boundary map. Instead of relying solely on pixel grayscale matching, the variational optimization unit iteratively solves an optimization problem that minimizes the energy functional.

[0073] Specifically, the variational optimization unit contains a differentiable iterative optimization layer. The iterative optimization layer performs the following operations: Data item weighting: Read the signal-to-noise ratio confidence map, reduce the weight of low signal-to-noise ratio regions (noisy regions) in relevance matching, and rely only on high confidence regions (such as strong edge structures) for anchoring.

[0074] Smoothing constraint: The semantic boundary map is read, and during the smoothing diffusion operation of the optical flow field, the diffusion coefficient of the Laplacian operator is dynamically adjusted according to the semantic boundary. When a semantic category jump is detected, the variational optimization unit sets the diffusion coefficient to zero to prevent the foreground optical flow vector from contaminating the background, thereby eliminating ghosting.

[0075] Finally, the semantic optical flow correction module fuses the coarse optical flow from the top layer of the pyramid to the bottom layer through upsampling operations, outputting a high-precision, corrected dense optical flow field. The data is transmitted in real time to the subsequent spatiotemporal feature recursive fusion module for feature alignment.

[0076] The spatiotemporal feature fusion module in this invention is the core processing unit for achieving video sequence denoising and temporal information accumulation. The spatiotemporal feature fusion module is configured to execute physically-aware recursive fusion logic. It is implemented as an improved recurrent neural network or gated recurrent unit hardware accelerator, possessing high-bandwidth on-chip storage read / write capabilities to meet the high-speed access requirements for historical state data.

[0077] The spatiotemporal feature fusion module is mainly composed of an input feature encoding unit, a historical state alignment unit, a physical gating calculation unit, and a state update unit in its internal logical architecture.

[0078] The input feature encoding unit is directly connected to the image sensor data stream interface and is responsible for performing feature space mapping on the low-light video frame input at the current moment. The input feature encoding unit consists of a set of cascaded convolutional layers and non-linear activation function layers. It transforms single-channel or three-channel raw image data into a current frame feature tensor with tens to hundreds of channels. This process not only extracts the spatial texture features of the image, but also expands the receptive field of the pixels through convolution operations, providing rich contextual information for subsequent temporal fusion.

[0079] The history state alignment unit is connected to the system's high-speed cache and is used to manage the transfer of information in the time dimension. The history state alignment unit first reads the hidden layer feature state of the previous frame from the cache. Subsequently, the history state alignment unit receives the corrected optical flow field input from the semantic optical flow correction module. Through a built-in hardware interpolator, the historical state alignment unit aligns the optical flow vectors... Perform pixel-level spatial resampling to generate a historical feature tensor aligned with the spatial location of the current frame. This operation ensures that, during time-series fusion, the features of the same physical object at different points in time coincide in spatial coordinates.

[0080] The physical gating computation unit is a core innovative component that distinguishes this module from traditional deep learning networks. The physical gating computation unit is responsible for computing time-varying gating graph updates. This determines the ratio of information forgotten to retained. The physical gating computation unit receives input in three dimensions: the current frame feature tensor. Aligned historical feature tensor And the physical signal-to-noise ratio confidence map from the signal-to-noise ratio estimation module. .

[0081] The physical gated computation unit uses the physical signal-to-noise ratio confidence map as an explicit attention channel in the convolution operation. When the signal-to-noise ratio (SNR) of a certain area is extremely low (high noise), the updated gating map calculated by the physical gating computing unit is indicated. It tends to retain more historical information, thereby using the accumulated energy of historical frames to smooth the current random noise; conversely, when the signal-to-noise ratio is high (such as in bright areas or moving edges), the gate value tends to be updated to the information of the current frame in order to avoid overwriting the true image details.

[0082] The state update unit performs the final weighted fusion operation. The state update unit utilizes the update gating graph output by the physical gating computation unit. For the current frame feature tensor and aligned historical feature tensor Perform element-wise linear interpolation.

[0083] The newly generated spatiotemporal fusion feature tensor for the current moment The output is split into two paths: one path writes back to the buffer memory via a direct memory access channel, overwriting the old state data for use in the next frame; the other path transmits to the chromaticity consistency reconstruction module for the generation of the final video frame. This recursive mechanism ensures that the video frames maintain temporal continuity and stability even under extremely low illumination, effectively eliminating flicker noise.

[0084] The color reconstruction and constraint module in this invention's AI-based passive night vision full-color video image enhancement system is the final output stage component of the entire image enhancement pipeline. The color reconstruction and constraint module is configured to execute semantic-guided generation and chromaticity correction logic. Integrated into the decoding end of the neural network accelerator, the color reconstruction and constraint module possesses high-precision floating-point arithmetic capabilities to ensure the accuracy of color calculations.

[0085] The color reconstruction and constraint module adopts a dual-stream parallel processing structure in its logical architecture, mainly including a luminance decoding unit, a semantic chrominance modulation unit, a priori library matching unit, and a color space inverse transformation unit.

[0086] The luminance decoding unit focuses on recovering the spatial geometry, texture details, and contrast information of the video image. The luminance decoding unit receives the spatiotemporal fusion feature tensor of the current moment from the spatiotemporal feature fusion module. The luminance decoding unit consists of a series of deconvolutional layers or upsampling interpolation layers. It progressively restores high-dimensional features from low resolution to the original video resolution. The output of the luminance decoding unit is only valid for colors in the CIELAB color space. The luminance channel is optimized, which allows the luminance decoding unit to ignore the interference of color noise and focus on improving the image's clarity and edge sharpness.

[0087] The semantic chroma modulation unit is a key component for restoring true colors, responsible for generating colors in the CIELAB color space. (Red / Green) and (Yellow / Blue) chroma channels. The semantic chroma modulation unit receives not only the spatiotemporal fusion feature tensor... It also receives the semantic tag matrix generated by the semantic awareness unit through a bypass interface. .

[0088] The semantic chroma modulation unit integrates a hardware-based semantic adaptive normalization operator. After each convolution operation, the semantic adaptive normalization operator dynamically queries the corresponding scaling factor based on the input semantic label index. With bias coefficient The semantic chroma modulation unit then performs an affine transformation on the current feature layer. Through this mechanism, the semantic chroma modulation unit forces the statistical distribution of the feature map to be pulled toward the color distribution range that the semantic category (such as vegetation, roads, and buildings) should have in the physical world, thereby synthesizing the lost chroma information under extremely low illumination.

[0089] The prior library matching unit is connected to a pre-configured read-only memory that stores a semantic-to-chroma prior library. This semantic-to-chroma prior library contains the statistically averaged chroma values ​​of hundreds of common objects under a standard D65 light source. The prior library matching unit uses this data to generate a preliminary chroma map from the semantic-to-chroma modulation unit. A global verification is performed. It calculates the average chroma of each semantic region in the current frame and compares it with the reference value in the database. If the mean deviation is found to exceed a preset threshold, the prior library matching unit calculates a global chroma offset vector and applies the global chroma offset vector in reverse to the chroma channel to eliminate the color cast phenomenon common in night view images.

[0090] The color space inverse transform unit is the final output stage of the system. It receives the optimized luminance map output by the luminance decoding unit. and the chromaticity map corrected by the prior library matching unit The inverse color space conversion unit performs standard color space conversion matrix operations, converting CIELAB data format to the RGB data format commonly used by display devices. The inverse color space conversion unit typically also includes a gamma correction circuit to adjust the gamma curve of the final output video frame according to the characteristics of human vision. The processed full-color video frame is then sent to the HDMI interface or display driver module for real-time display.

Claims

1. A passive night vision full-color video image enhancement method based on AI learning, characterized in that, Includes the following steps: S1. Acquire continuous low-light video frames and synchronized sensor metadata, and use the sensor metadata to generate a signal-to-noise ratio confidence map based on a physical noise model; S2. Perform semantic segmentation on the low-light video frames to generate semantic label maps, and use the signal-to-noise ratio confidence map and the semantic label maps as joint constraints to calculate the optical flow field between adjacent video frames. S3. Based on the optical flow field, the historical frame features are spatially aligned, and the fusion gate coefficient is calculated in combination with the signal-to-noise ratio confidence map. The spatially aligned historical features and the current frame features are recursively fused to generate temporal fusion features. S4. Use the semantic tag map to modulate the temporal fusion features, and combine the semantic color prior knowledge to reconstruct the image, outputting a full-color enhanced video frame.

2. The passive night vision full-color video image enhancement method based on AI learning according to claim 1, characterized in that, In step S1, the specific steps for generating the signal-to-noise ratio confidence map based on the physical noise model using the sensor metadata include the following steps: Read the analog gain parameter and readout noise variance parameter from the sensor metadata; Substitute the analog gain parameter and the readout noise variance parameter into the Poisson-Gaussian mixed noise model to calculate the signal expectation value and noise standard deviation of each pixel in the low-light video frame. The signal-to-noise ratio confidence map is generated based on the ratio of the expected signal value to the standard deviation of the noise.

3. The passive night vision full-color video image enhancement method based on AI learning according to claim 1, characterized in that, In step S2, the specific steps for calculating the optical flow field between adjacent video frames include the following: Construct an optical flow energy functional that includes weighted data terms and semantic regularization terms; The signal-to-noise ratio confidence map is used to perform pixel-by-pixel weighting on the weighted data terms in the optical flow energy functional; The semantic label graph is used to set the boundary conditions for the semantic regularization term in the optical flow energy functional; The optical flow field is obtained by minimizing the optical flow energy functional.

4. The passive night vision full-color video image enhancement method based on AI learning according to claim 3, characterized in that, The specific steps of setting the boundary conditions for the semantic regularization term in the optical flow energy functional using the semantic tag map include: When pixels belong to the same semantic category, the smoothing control coefficient is set to a positive value; When pixels belong to different semantic categories, the smoothing control coefficient is set to zero, thereby releasing the smoothing constraint on the motion vectors of adjacent pixels in adjacent video frames.

5. The passive night vision full-color video image enhancement method based on AI learning according to claim 1, characterized in that, In step S3, the specific steps for spatially aligning historical frame features based on the optical flow field include the following: A convolutional neural network encoder is used to extract features from the low-light video frames to generate the current frame feature tensor; Read the hidden layer feature state of the previous frame; The optical flow field is used to spatially resample the hidden layer feature state of the previous frame to generate an aligned historical feature tensor.

6. The passive night vision full-color video image enhancement method based on AI learning according to claim 5, characterized in that, In step S3, the specific steps for recursively fusing spatially aligned historical features with current frame features by calculating the fusion gating coefficients in conjunction with the signal-to-noise ratio confidence map include the following steps: The signal-to-noise ratio confidence map is used as an independent feature channel and concatenated with the current frame feature tensor and the aligned historical feature tensor. The gated graph is updated over time by calculating branches of a convolutional neural network; The time-fusion feature is generated by weighted summation of the current frame feature tensor and the aligned historical feature tensor using the time-update gating graph.

7. The passive night vision full-color video image enhancement method based on AI learning according to claim 1, characterized in that, In step S4, the specific steps for modulating the temporal fusion features using the semantic label map include the following: The temporal fusion features are input into an image reconstruction network that includes a luminance reconstruction branch and a chrominance reconstruction branch; The brightness reconstruction branch performs convolution and upsampling operations on the temporal fusion features to output a single-channel brightness map; The chroma reconstruction branch introduces the semantic label map as a conditional control variable, and performs affine transformation modulation on the feature layer through a semantic adaptive normalization unit to generate a chroma map.

8. The passive night vision full-color video image enhancement method based on AI learning according to claim 7, characterized in that, In step S4, the specific steps for image reconstruction based on semantic colorimetric prior knowledge include the following: Retrieve a pre-set semantic-to-chromatic prior library, which stores the mean chromaticity center values ​​of different semantic categories under normal illumination; Calculate the average chromaticity of each object region in the chromaticity diagram; The chromaticity map is chromaticity corrected by minimizing the difference between the average chromaticity and the mean chromaticity center.

9. The passive night vision full-color video image enhancement method based on AI learning according to claim 8, characterized in that, The specific steps for outputting the full-color enhanced video frame in step S4 include the following: Merge the luminance map with the chrominance-corrected chrominance map in the channel dimension; The merged data is converted from CIELAB space to standard RGB space using a color space inverse conversion algorithm to generate the full-color enhanced video frame.

10. A passive night vision full-color video image enhancement system based on AI learning, characterized in that, The method for performing the AI-based passive night vision full-color video image enhancement method according to any one of claims 1-9 includes: The signal-to-noise ratio feature extraction module is configured to generate a signal-to-noise ratio confidence map using sensor metadata. The semantic-assisted optical flow calculation module is configured to calculate the optical flow field between adjacent video frames using the signal-to-noise ratio confidence map and the semantic label map; The spatiotemporal feature recursive fusion module is configured to recursively fuse historical frame features and current frame features using the optical flow field and the signal-to-noise ratio confidence map; The chromaticity consistency reconstruction module is configured to reconstruct full-color enhanced video frames using the semantic label map and semantic chromaticity prior knowledge.

Images