Machine learning based network monitoring video frame enhancement system

Abstract

The application discloses a network monitoring video picture enhancement system based on machine learning and belongs to the technical system of video picture enhancement. A photon counting degradation model is established for a night long-exposure frame output by a monitoring camera. Through variational Bayesian inference, a single long-exposure image is inversely decomposed into a plurality of virtual short-exposure subframes, and a pixel-by-pixel motion displacement field is estimated. A virtual short-exposure frame sequence and a corresponding motion displacement field are output. In a Riemannian manifold space, dictionary learning and sparse coding are performed through geodesic distance measurement. Meanwhile, the output motion displacement field is utilized to construct a pixel-by-pixel motion blur kernel. In the process of sparse coding, a small target saliency map is introduced as a spatial adaptive regularization constraint. A clean frame after denoising and deblurring and a small target protection mask are output. The transmission mapping reconstruction result and the clean frame are adaptively weighted and fused, and a final monitoring picture of evidence-level enhancement is output.

Description

Technical Field

[0001] This invention relates to video enhancement technology systems, specifically to a network surveillance video enhancement system based on machine learning. Background Technology

[0002] Surveillance cameras often use long exposures at night to achieve brightness, but the movement of pedestrians and vehicles will produce severe blur and motion blur. Traditional low-light enhancement, such as Zero-DCE, will greatly amplify the noise after brightening. Forced noise reduction will erase small targets such as pedestrians in the distance. It has the defects of low light, high noise and motion blur, and cannot retain small physical evidence. Summary of the Invention

[0003] The purpose of this invention is to provide a network surveillance video enhancement system based on machine learning to solve the technical problems mentioned in the background art.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] A machine learning-based network surveillance video enhancement system includes:

[0006] Virtual short exposure frame processing module: Based on the Poisson statistical characteristics of photon shot noise of CMOS sensor, a photon counting degradation model is established for the night long exposure frame output by the surveillance camera. Through variational Bayes inference, the single frame long exposure image is decomposed into multiple virtual short exposure sub-frames. At the same time, the pixel-by-pixel motion displacement field is estimated, and the virtual short exposure frame sequence and the corresponding motion displacement field are output.

[0007] Image Local Region Processing Optimization Module: The local regions of the images in the obtained virtual short exposure frame sequence are represented as symmetric positive definite covariance matrices. On the Riemann manifold space, dictionary learning and sparse coding are performed through geodesic distance metric to achieve geometric decoupling between noise components and clean signal components. At the same time, the output motion displacement field is used to construct a pixel-by-pixel motion blur kernel. In the sparse coding process, a small target saliency map is introduced as a spatial adaptive regularization constraint. The output is a denoised and deblurred clean frame and a small target protection mask.

[0008] The monitoring image enhancement processing module uses the output clean frame as the target distribution anchor point and the original long-exposure degraded frame as the source distribution to construct the Wasserstein optimal transmission mapping with the generated small target protection mask as the transmission cost weight modulation factor. During the mapping solution process, a structural similarity hard constraint is applied to the evidence area. The transmission mapping reconstruction result and the clean frame are adaptively weighted and fused to output the final monitoring image with evidence-level enhancement.

[0009] Furthermore, when establishing a photon counting degradation model based on Poisson statistical properties, for coordinates in a long exposure frame... The pixels, in the exposure time Internally accumulated received photon count , obeys the parameter The Poisson distribution;

[0010] Long exposure time Divide the data into N equal virtual short exposure time slices, with each time slice having a duration of: ;in, The duration of a single virtual short exposure time slice; N is the total number of virtual short exposure subframes;

[0011] In the Within a virtual short exposure time frame, calculate pixels The expected number of photons arriving at a given location is equal to the sum of the number of photons arriving at all N virtual short exposure timepieces, satisfying the conservation constraint.

[0012] Furthermore, when constructing the Poisson likelihood function based on motion compensation, the first... Pixels within a time slice The displacement vector of motion is Then the pixel is in the th position. Within a given time slice, the photons actually received originate from locations within the scene. ;

[0013] Based on motion compensation, the first A virtual short-exposure subframe, in pixels The expected number of photons arriving at the location is corrected to: ;in, To take motion compensation into account, the first The expected number of photons arriving at (x,y); Let be the actual photon arrival rate distribution function of the scene; For the first Within a time slice, pixels The horizontal displacement component; For the first Within a time slice, pixels The vertical displacement component.

[0014] Furthermore, when updating parameters alternately through coordinate ascending variational inference, the coordinate ascending variational inference method is adopted to alternately fix one set of variables and update another set of variables, including fixing the motion displacement field and updating the variational parameters of the photon arrival rate; and fixing the photon arrival rate and updating the variational parameters of the motion displacement field.

[0015] Furthermore, iterative convergence determination is implemented. After convergence, N virtual short exposure subframes are output, along with a pixel-by-pixel motion displacement field.

[0016] Furthermore, when introducing adaptive sparse coding for the small target saliency map, after dictionary learning, sparse coding is performed on each data point. The temporal median is calculated pixel-by-pixel for N subframes to obtain the temporal median frame. The absolute deviation of each pixel from the temporal median in its neighborhood is calculated, and then Gaussian smoothing is applied to obtain the saliency map. Based on the small target saliency map, a spatial adaptive sparse coding optimization problem is defined to normalize the small target saliency map to... After the interval, it is output as a small target protection mask.

[0017] Furthermore, a pixel-by-pixel motion blur kernel is constructed using the motion displacement field and embedded during deblurring. The average displacement vectors of all N time slices are then used to obtain the comprehensive displacement. Based on the comprehensive displacement, a linear motion blur kernel is generated.

[0018] In the sparse coding reconstruction stage, when embedding the motion blur kernel into the reconstruction process, the reconstruction covariance matrix obtained by sparse coding is mapped back to image patches through the manifold exponent, and then non-blind deconvolution is performed on each image patch.

[0019] All the deblurred image blocks are stitched together according to their original spatial positions, and the overlapping areas are weighted and averaged to obtain a clean frame after denoising and deblurring.

[0020] Furthermore, the original long-exposure degraded frame and the output clean frame are defined as the source probability measure and the target probability measure in the optimal transmission problem, respectively.

[0021] Furthermore, when solving the Wasserstein optimal transport map with hard constraints, an optimal transport optimization problem with hard constraints of the structural similarity of the evidence region is constructed based on the defined transport cost function.

[0022] By using structural similarity hard constraints, from the perspective of optimizing the feasible domain, it is required that the structural similarity between the reconstructed evidence region and the clean frame is not lower than a preset value, ensuring that the edges, textures and contrasts of the evidence are not destroyed after transmission mapping.

[0023] Furthermore, based on the optimal transmission plan, the transmission mapping reconstruction result is calculated through centroid mapping; the transmission mapping reconstruction result is then adaptively weighted and fused with clean frames to generate the final enhanced output.

[0024] Compared to existing solutions, the beneficial effects achieved by this invention are:

[0025] This invention establishes a degradation model based on the Poisson statistical characteristics of photon shot noise in CMOS sensors, rather than using an empirical Gaussian noise assumption. This ensures that the noise modeling is consistent with the actual physical process and avoids enhancement artifacts caused by the failure of the Gaussian assumption under extremely low illumination. Through variational Bayesian inference, the true brightness and motion displacement field are estimated simultaneously, explicitly separating motion blur from the degraded image. In the output virtual short-exposure subframe, motion blur has been significantly reduced, providing high-quality input for further denoising and deblurring processing. The image patch is modeled as a symmetric positive definite covariance matrix and processed on a Riemannian manifold. Utilizing the nonlinear geometric structure of the manifold, noise disturbances and signal variations are naturally separable under geodesic distance metrics. Compared with traditional denoising methods in Euclidean space, this method has superior noise reduction performance under extremely low signal-to-noise ratio conditions. It possesses strong suppression and structure preservation capabilities. Through spatial adaptive sparse regularization driven by the temporal median saliency map, it differentiates between small target regions and background regions at the algorithm optimization level. This avoids the problem of erasing small evidence such as distant pedestrians and vehicles caused by the global uniform denoising intensity of traditional methods, significantly improving the evidence detection rate. Wasserstein optimal transmission redistributes global brightness at the probability distribution level. Compared with pixel-by-pixel gamma correction or histogram equalization, it can maintain the spatial coherence and naturalness of the image, avoiding local over-enhancement or under-enhancement, so that the details in dark areas are fully enhanced while the bright areas are not overexposed. Through spatial modulation of transmission costs and hard constraints on structural similarity, a dual protection mechanism is formed to ensure that the structure of the evidence region is not destroyed by transmission mapping from both the optimization target level and the feasible region level. Attached Figure Description

[0026] The invention will now be further described with reference to the accompanying drawings.

[0027] Figure 1 This is a flowchart illustrating the operation of the machine learning-based network monitoring video enhancement system of the present invention. Detailed Implementation

[0028] The technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] like Figure 1 As shown, the present invention is a network surveillance video enhancement system based on machine learning, comprising:

[0030] Virtual short exposure frame processing module: Based on the Poisson statistical characteristics of photon shot noise of CMOS sensor, a photon counting degradation model is established for the night long exposure frame output by the surveillance camera. Through variational Bayes inference, the single frame long exposure image is decomposed into multiple virtual short exposure sub-frames. At the same time, the pixel-by-pixel motion displacement field is estimated, and the virtual short exposure frame sequence and the corresponding motion displacement field are output.

[0031] To address the issue of severe motion blur and trailing of moving targets such as pedestrians and vehicles caused by long exposure modes when surveillance cameras acquire images at night, and the significant photon shot noise of CMOS sensors under low-light conditions, a photon counting degradation model conforming to the physical imaging mechanism is established. Through variational Bayesian inference, a single-frame long-exposure degraded image is inverted and decomposed into multiple virtual short-exposure subframes, while simultaneously estimating the pixel-by-pixel motion displacement field. This module provides a clean, motion-decoupled subframe sequence and accurate motion information for subsequent Riemannian manifold denoising and deblurring processing, forming the foundation for the entire system to achieve joint recovery from the triple degradation of low light, high noise, and motion blur. Specifically, it includes the following sub-steps:

[0032] When establishing a photon counting degradation model based on Poisson statistical properties, in low-light conditions at night, the number of photons received by the CMOS image sensor of the surveillance camera is extremely small, and the process of photons reaching the sensor pixels follows a Poisson distribution; specifically, for a long-exposure frame with coordinates... The pixels, in the exposure time Internally accumulated received photon count , obeys the parameter The probability mass function is a Poisson distribution with the following characteristics: ;in, pixel coordinates in a long exposure frame The observed gray value at that point, i.e., the CMOS sensor at the exposure time. The photon count value accumulated within the internal record is a non-negative integer. For pixels Throughout the long exposure time The expected total number of photons arriving within the range, i.e., the intensity parameter of the Poisson distribution. ; For pixels The actual photon arrival rate per unit time, i.e., photon flux, reflects the actual brightness information of the scene point corresponding to the pixel, and the unit is photons / second; The long exposure time set for surveillance cameras, in seconds, typically ranging from 1 / 15 of a second to 2 seconds; is a natural constant, with a value of approximately 2.71828; factorial.

[0033] The physical meaning of the above formula is: given a real photon arrival rate and exposure time Under these conditions, the actual photon count value recorded by the sensor It is a random variable whose variance (i.e., shot noise) is equal to its mean. This statistical property is the theoretical basis for constructing the likelihood function in subsequent variational inference;

[0034] It should be noted that in this embodiment, the formulas or expressions involved in the calculations are all standardized before the calculations are performed. The standardization process includes, but is not limited to, normalization and extraction of numerical values ​​from the calculation data. These are all existing conventional technical solutions, and the specific implementation steps will not be elaborated here.

[0035] Furthermore, the long exposure time Divide the data into N equal virtual short exposure time slices, with each time slice having a duration of: ;in, The duration of a single virtual short exposure time slice is in seconds; N is the total number of virtual short exposure subframes, which is a positive integer ranging from 4 to 16, preferably 8.

[0036] The above formula discretizes a continuous long exposure process into N equally spaced short exposure segments in the time dimension, with each segment corresponding to a virtual short exposure subframe. This time discretization is a prerequisite for subsequent variational inference of motion information: if there is a moving target in the scene, the photons received at the same pixel position in different time slices come from different spatial positions in the scene, thus producing motion blur.

[0037] In the Within a virtual short exposure time frame For virtual short exposure time slice index, Pixels The expected number of photons arriving at the location is: ;in, For the first Within a virtual short exposure time frame, pixels The expected number of photons arriving at the location; For the first Within a virtual short exposure time frame, pixels The corresponding actual photon arrival rate, due to the presence of the moving target, It may differ in different time slices.

[0038] The sum of the photon arrivals of all N virtual short-exposure timepieces equals the total photon arrivals of the long exposure, satisfying the following conservation constraint: ;

[0039] The above formula establishes a mathematical constraint relationship between long exposure observations and each virtual short exposure subframe. It is the core equation for constructing the likelihood function in variational Bayesian inference. Its physical meaning is: the total photon count of each pixel in the long exposure frame is equal to the sum of the photon counts in all virtual short exposure time slices. This conservation relationship ensures the physical consistency of the inversion decomposition.

[0040] When constructing the Poisson likelihood function based on motion compensation, since there are moving targets in the scene, the th... Within a virtual short exposure time frame, the space originally located at... The scene point moved to its current position at the end of the exposure. ; Define the first Pixels within a time slice The displacement vector of motion is Then the pixel is in the th position. Within a given time slice, the photons actually received originate from locations within the scene. .

[0041] Based on motion compensation, the first A virtual short-exposure subframe, in pixels The expected number of photons arriving at the location is corrected to: ;in, To take motion compensation into account, the first The expected number of photons arriving at (x,y); Let be the actual photon arrival rate distribution function of the scene; For the first Within a time slice, pixels The horizontal displacement component, in pixels, is the horizontal direction, i.e., the x-axis direction. For the first Within a time slice, pixels The vertical displacement component, measured in pixels, is located in the y-axis direction.

[0042] It needs to be explained that due to the target's movement, pixels In the The photons received within a time slice do not originate from the scene point directly above them, but rather from points that have undergone displacement. The offset scene position; by introducing a motion displacement field, the motion blur effect is explicitly modeled into the degradation model.

[0043] Pixels in long exposure frames Observations The complete likelihood function is: ;in, Given a true photon arrival rate distribution and the motion displacement field of all time slices Under the conditions, the gray value was observed. The probability of; For the first The expected number of photons arriving per time slice due to motion compensation. For the set of motion displacement fields of all N virtual short exposure time slices, .

[0044] The above formula is the core likelihood function of variational Bayesian inference. It unifies photon shot noise (Poisson statistics) and motion blur (displacement field) within a probabilistic framework, enabling subsequent variational inference to simultaneously invert the true brightness and motion information of the scene.

[0045] When constructing the variational posterior distribution and setting the prior, due to the posterior distribution... Since this involves high-dimensional integrals that cannot be solved analytically directly, variational Bayesian inference is used for approximation to construct a variational posterior distribution. Assuming that each pixel is conditionally independent, it can be decomposed into: ;in, For pixels Real photon arrival rate The variational posterior distribution is parameterized using the Gamma distribution, i.e. ,in For shape parameters, For rate parameters; For pixels In the Motion displacement in each time slice The variational posterior distribution is parameterized using a two-dimensional Gaussian distribution, i.e. ,in, It is the mean vector. It is the covariance matrix; It is the coordinates of all pixels in the image The product of consecutive products; It is the product of all N virtual short exposure time slices.

[0046] The mean field of the above formula approximates the decomposition of the high-dimensional joint posterior into the product of the independent edge distributions of each pixel and each time slice, which greatly reduces the computational complexity of inference and makes real-time processing of large-scale surveillance images, such as 1920×1080 resolution, possible.

[0047] For true photon arrival rate Let the conjugate prior be a Gamma distribution: ;in, The shape parameter of the Gamma prior is set to 1, indicating no strong prior preference for photon arrival rate. The rate parameter for the Gamma prior is 0.01, which corresponds to a wide prior distribution and allows the photon arrival rate to vary over a large range.

[0048] For motion displacement Let the prior be a zero-mean Gaussian distribution: ;in, The zero vector is a two-dimensional vector, representing the prior assumption that there is no motion; To determine the prior variance of displacement, the maximum allowable motion amplitude is controlled, with a value ranging from 4 to 64 pixels², preferably 16 pixels². It is a 2×2 identity matrix.

[0049] It needs to be explained that: when there is no observation information, the prior assumption is that the scene is stationary, that is, the displacement is zero, but motion is allowed within a certain range; The value of determines the system's tolerance for motion amplitude: the larger the value, the wider the range of motion that can be detected.

[0050] When updating parameters alternately through coordinate ascending variational inference, the goal of variational inference is to maximize the lower bound of evidence (ELBO), which is expressed as: ;in, The lower bound of evidence is ELBO, which is the true logarithm marginal likelihood. The lower bound, maximize Equivalent to minimizing variational posterior KL divergence between the true posterior and the actual posterior; It is in the variational posterior The expected value of the logarithm of the likelihood function is used to measure the goodness of fit of the variational posterior to the observed data. For variational posterior with prior The Kullback-Leibler divergence between the two measures the degree to which the variational posterior deviates from the prior, and plays a regularization role to prevent overfitting. Let be the Poisson likelihood function; It is a joint prior distribution.

[0051] The coordinate ascending variational inference method is used, which alternately fixes one set of variables and updates another set of variables. The specific iterative process is as follows:

[0052] (a) Fixed motion displacement field Update photon arrival rate Variational parameters:

[0053] Utilizing the conjugate property of the Gamma distribution, the updated variational parameters are: ; ;in, For the first After the next iteration, the pixel The shape parameters of the variational posterior of the photon arrival rate; For the first After the next iteration, the pixel The rate parameter of the variational posterior of the photon arrival rate; These are prior parameters; Pixels in a long exposure frame The observed grayscale value; It is in the The variational posterior of motion displacement in the next iteration subscript expectations; superscript and They represent the first Second and third The value of the next iteration.

[0054] What needs to be explained is: the observed photon count The increase in shape parameters directly contributes to the higher the observed value, the higher the estimated photon arrival rate.

[0055] The rate parameter, determined by the prior and the expected photon contributions of all virtual short exposure time slices, embodies the specific realization of the long exposure conservation constraint in variational inference.

[0056] After the update, the variational posterior expectation of the photon arrival rate is: ;

[0057] The above formula gives the optimal estimate of the actual photon arrival rate at the pixel in the current iteration, which is the actual brightness of the scene point corresponding to that pixel.

[0058] (b) Fixed photon arrival rate Update the motion displacement field Variational parameters:

[0059] Variational posterior of motion displacement By maximizing ELBO about and The gradient is used to obtain the update formula: ; ;in, For the first After the next iteration, the pixel In the The mean displacement estimate for each time slice is a two-dimensional vector. For the first After the next iteration, the pixel In the The motion displacement covariance matrix of each time slice is a 2×2 symmetric positive definite matrix. It is about the displacement vector The gradient operator; It is about the displacement vector The Hessian matrix is ​​a second-order derivative matrix; It is in the The variational a posteriori of photon arrival rate in the next iteration lower expectations; For the prior variance of displacement; It is a 2×2 identity matrix.

[0060] It needs to be explained that the estimated direction of the motion displacement is determined by the gradient of the likelihood function with respect to the displacement: the gradient direction indicates the displacement direction that maximizes the probability of the observed data, that is, the most likely direction of motion.

[0061] Furthermore, the uncertainty (covariance) of displacement estimation is determined by the curvature (Hessian matrix) of the likelihood function: the greater the curvature (the sharper the likelihood function), the more certain the displacement estimation.

[0062] When performing iterative convergence determination and virtualizing short-exposure subframes, repeat steps (a) and (b) from the preceding steps until the following convergence condition is met: ;in, and The first Second and third ELBO value after the next iteration; To prevent extremely small constants with a denominator of zero, the value is taken as... ; The convergence threshold is set to a value of [value to be filled in]. When the relative change in ELBO is less than this threshold, convergence is determined.

[0063] It should be noted that the convergence state of variational inference is monitored as follows: when the relative change of ELBO in two consecutive iterations is sufficiently small, it is considered that the variational posterior has sufficiently approximated the true posterior, and the iteration is stopped.

[0064] After convergence, output N virtual short-exposure subframes. , of which The pixel values ​​of each virtual short-exposure sub-frame are calculated as follows: ;in, For the first A virtual short exposure subframe at pixels The grayscale value at that location; For estimating the actual photon arrival rate after convergence; For the converged first... The mean displacement estimate for each time slice is a two-dimensional vector. .

[0065] It needs to be explained that each virtual short-exposure subframe is obtained by multiplying the converged true brightness estimate by the short exposure time after inverse compensation based on the motion displacement of the corresponding time slice; since the exposure time corresponding to each subframe is extremely short, Motion blur is significantly reduced, and the variance of Poisson noise is also reduced accordingly because the expected number of photons is reduced.

[0066] Simultaneously outputs pixel-by-pixel motion displacement field ,in: ;

[0067] The final output motion displacement field is the converged mean displacement estimate, which will be used in subsequent steps to construct a pixel-by-pixel motion blur kernel to guide the deblurring process.

[0068] It should be noted that this module, as the first-level processing link of the entire network surveillance video enhancement system, undertakes the core task of degradation decoupling. It separates the three degradation factors coupled together in a single frame of long exposure degradation image—low light, shot noise, and motion blur—at the physical model level. Through the Poisson degradation model, it explicitly models the statistical characteristics of shot noise; through the motion displacement field, it explicitly models motion blur; and through virtual short exposure decomposition, it transforms the low light problem into a problem of processing multiple short exposure subframes with higher signal-to-noise ratios.

[0069] In this embodiment, a degradation model is established based on the Poisson statistical characteristics of photon shot noise from a CMOS sensor, rather than using an empirical Gaussian noise assumption. This ensures that the noise modeling is consistent with the actual physical process and avoids enhanced artifacts caused by the failure of the Gaussian assumption under extremely low illumination. Through variational Bayesian inference, the true brightness and motion displacement field are estimated simultaneously, explicitly separating motion blur from the degraded image. In the output virtual short-exposure sub-frames, motion blur has been significantly reduced, providing high-quality input for further denoising and deblurring processing in subsequent steps. Multiple virtual short-exposure sub-frames can be derived from a single long-exposure image, without... It requires multi-frame acquisition or high-speed sensors in hardware, reducing system deployment costs and making it suitable for existing ordinary surveillance cameras. The characteristic that the variance of the Poisson distribution equals the mean allows variational inference to naturally distinguish between signal components (high photon count regions) and noise components (low photon count regions) without the need for additional noise estimation steps. The output motion displacement field provides accurate motion parameters for constructing a pixel-by-pixel motion blur kernel in subsequent steps, and the output virtual short-exposure subframe sequence provides noise-reduced input data for dictionary learning on the Riemannian manifold in subsequent steps. Both of these together ensure the accuracy and effectiveness of subsequent processing steps.

[0070] Image Local Region Processing Optimization Module: The local regions of the images in the obtained virtual short exposure frame sequence are represented as symmetric positive definite covariance matrices. On the Riemann manifold space, dictionary learning and sparse coding are performed through geodesic distance metric to achieve geometric decoupling between noise components and clean signal components. At the same time, the output motion displacement field is used to construct a pixel-by-pixel motion blur kernel. In the sparse coding process, a small target saliency map is introduced as a spatial adaptive regularization constraint. The output is a denoised and deblurred clean frame and a small target protection mask.

[0071] This module receives the output virtual short-exposure frame sequence. and pixel-by-pixel motion displacement field To address the Poisson shot noise still present in the virtual short exposure subframes, as well as the motion blur remaining due to imperfect inversion, a Riemannian manifold geometric framework is adopted for joint denoising and deblurring.

[0072] The core innovation of this module lies in: modeling local image regions as symmetric positive definite covariance matrices, and utilizing the nonlinear geometric structure of Riemannian manifolds to achieve intrinsic separation of noise and signal; simultaneously transforming the estimated motion displacement field into a physical motion blur kernel, and embedding sparse coding regularization terms into the saliency map of small targets, thereby removing noise and blur while protecting distant pedestrians, vehicles, and other minute evidence from excessive smoothing; this module outputs a clean frame after denoising and deblurring. and small target protection mask This provides high-quality target distribution anchor points and physical evidence region constraints for subsequent Wasserstein optimal transport mapping, specifically including the following sub-steps:

[0073] When constructing a symmetric positive definite covariance matrix representation of a local region of an image, for each virtual short-exposure subframe in the output... In pixels Centered on, extract size is A local image patch, wherein The value is an odd number, taking the values ​​5 or 7, preferably 5; for each pixel within the image patch, a 5-dimensional feature descriptor vector is extracted. ,in, This represents the pixel index within the image block. The meanings of each channel are as follows: It is the pixel grayscale value, that is The normalized intensity value at this pixel, with a range of values. ; For the horizontal gradient, use the Sobel operator. Calculated; For the vertical gradient, using the Sobel operator Calculated; For the Laplace response, via the Laplacian operator The calculations reflect local second-order texture information; For local variance, centered on this pixel The variance of gray values ​​within a neighborhood reflects local contrast.

[0074] Based on the above 5-dimensional features, calculate the image patch's... Sample covariance matrix: ;

[0075] in, For the first Each image patch corresponds to Symmetric positive definite covariance matrix The index for traversing an image patch throughout the entire image; The total number of pixels within an image block, when hour, ; For the first image block A 5-dimensional feature column vector of pixels; This is the mean of the feature vectors of all pixels within the image patch. ; This is a matrix transpose operation; Here is the regularization constant, and its value is [value]. This is used to ensure the positive definiteness of the covariance matrix and prevent matrix singularity due to similar feature dimensions or insufficient samples. Identity matrix.

[0076] It needs to be explained that traditional image processing uses pixel grayscale vectors as the basic units in Euclidean space. However, grayscale vectors cannot simultaneously encode multi-dimensional structural information such as texture, gradient, and contrast. Furthermore, Euclidean distance is sensitive to noise. The covariance matrix, on the other hand, naturally incorporates the statistical correlation of multi-channel features and is a symmetric positive definite matrix, forming a Riemannian manifold with rich geometric structure. The geodesic distance defined on this manifold is inherently robust to noise: the covariance matrix caused by noise is distributed along the tangent space of the manifold, while the signal variation is distributed along the curvature of the manifold; the two are geometrically separable. The regularization term... This ensures that all covariance matrices are strictly positive definite, located inside the manifold rather than on its boundary, thus guaranteeing the numerical stability of subsequent Riemannian operations.

[0077] For all virtual short exposure subframes Performing the above operations yields the covariance matrix dataset. ,in, The total number of image patches in all subframes. , and These represent the height and width of the subframe, respectively.

[0078] When implementing dictionary learning on a Riemannian manifold, in a symmetric positive definite matrix manifold Dictionary learning is conducted on the above to construct a dictionary containing dictionary atoms A manifold dictionary, where each dictionary atom For one Symmetric positive definite matrix The value is the dictionary size, ranging from 64 to 256, with 128 being the preferred value.

[0079] Define two points on a manifold and The affine invariant Riemann distance (AIRM) between them: ;in, Covariance matrix With dictionary atoms In the Riemannian manifold The geodesic distance on the surface is a non-negative real number. For matrix The inverse square root matrix, through the... Perform eigenvalue decomposition Then, calculate get; Let be the matrix logarithm function, for a symmetric positive definite matrix. , ,in Take the natural logarithm of the diagonal elements; It is the Frobenius norm. , This is for matrix trace operations.

[0080] It needs to be explained that the Frobenius distance in Euclidean space... It lacks affine invariance, meaning that the distance changes after a linear transformation of the data, causing the dictionary learning result to depend on the feature scaling method; affine invariant Riemann distance satisfy For any invertible matrix This means that the distance metric is independent of the coordinate system chosen for the features, fundamentally guaranteeing the geometric consistency of dictionary learning; more importantly, The measurement on the manifold is the geodesic length, not the chord length, which can accurately capture the nonlinear geometric relationship between the covariance matrices. This makes the noise disturbance (small offset along the tangent space) and the signal change (large offset along the curvature direction) significantly different in distance measurement, laying the foundation for subsequent noise-signal geometric decoupling.

[0081] Based on Riemannian distance, the optimization objective for manifold dictionary learning is: ;in, For the first Data points In the dictionary atoms The weighting of the distribution, ,and ; The square of the Riemann distance is used as a measure of the reconstruction error from the data point to the dictionary atom. This is the dictionary regularization coefficient, with a value of 0.01, to prevent dictionary atoms from degenerating or becoming overly concentrated; The Riemann mean, or Fréchet mean, of all current dictionary atoms is defined as follows: The solution is obtained iteratively on the manifold using gradient descent. The size of the dataset is the covariance matrix.

[0082] It needs to be explained that: the first term is the weighted Riemann reconstruction error, which drives dictionary atoms to cluster towards the geometric center of the data points; the second term is the dispersion penalty of dictionary atoms relative to the Riemann mean, which prevents dictionary atoms from collapsing to the same position on the manifold, ensuring the diversity of the dictionary; this optimization problem is solved by an alternating minimization strategy: fixed renew At that time, on a Riemannian manifold, gradient descent is used to move dictionary atoms along geodesics; fixed renew At that time, through soft allocation rules Calculation, where The temperature parameter is initially set to 1, and then gradually annealed to 0.1.

[0083] When introducing adaptive sparse coding of small target saliency maps, after dictionary learning is completed, for each data point... Sparse coding is performed; first, the output virtual short-exposure frame sequence is used. Generate a saliency map of small targets. The specific method is as follows:

[0084] Calculate the temporal median for each pixel of N subframes to obtain the temporal median frame. ;

[0085] Calculate each pixel and its The absolute deviation of the time series median within the neighborhood. The value was set to 15, and then Gaussian smoothing was applied to obtain the significance plot: ;in, For pixels The significance value of the small target at the location, and its range. The larger the value, the more likely the pixel is to be a small target; For the temporal median frame in pixels The grayscale value at that location; Therefore Centered Inside the window The local mean; The standard deviation is Two-dimensional Gaussian kernel, The value is 3. Represents a two-dimensional convolution operation; This is for absolute value operations.

[0086] It needs to be explained that small targets, such as pedestrians and vehicles in the distance, occupy only a few pixels in the monitoring image. Their brightness has a local difference from the surrounding background, but they are easily submerged by noise in a single frame. Temporal median filtering can effectively suppress random noise while retaining the continuous small target signal. Absolute bias captures the contrast difference between small targets and the background. Gaussian smoothing eliminates isolated noise spikes and makes the saliency map spatially continuous. This saliency map will be used as a spatial adaptive weight in sparse coding to guide the algorithm to retain more high-frequency details in the small target area.

[0087] Based on the small target saliency map, we define a spatial adaptive sparse coding optimization problem: ;in, , for the first A sparse coefficient vector of data points ; Riemann reconstruction error;

[0088] This is the basic parameter for sparsity regularization, with a value of 0.1, which controls the overall sparsity. For the first The center pixel of each image block The significance value of the small target at the location; The smoothing constant is set to 0.01 to prevent the denominator from being zero and to avoid the regularization weights of background regions with zero significance from tending to infinity.

[0089] It needs to be explained that traditional sparse coding applies the same pressure to all spatial locations. The regularization strength causes high-frequency details in small target regions to be suppressed along with noise; by designing the regularization weights as... Inverse proportional function: when When the target area is large, the denominator increases, and the effective regularization strength decreases. Reduced size allows for finer reconstruction of the region using more dictionary atoms, preserving edge and texture details; when When the background area is small, the regularization intensity is large, which forces sparse representation and effectively suppresses noise. This spatial adaptive mechanism realizes a differentiated processing strategy that preserves evidence and suppresses noise.

[0090] The optimization problem is a convex optimization problem under non-negative constraints, which is solved by the projective gradient descent method: after each gradient update, the negative components are truncated to zero, and the iteration continues until convergence or the maximum number of iterations is reached. The maximum number of iterations is set to 100.

[0091] A pixel-by-pixel motion blur kernel is constructed using the motion displacement field and embedded during deblurring, utilizing the output pixel-by-pixel motion displacement field. For each pixel Constructing motion fuzzy kernel Specifically, the average displacement is obtained by averaging the displacement vectors of all N time slices: ;in, : pixel The combined motion displacement vector at the point is a two-dimensional vector. ; For the output of the first Each time slice motion displacement vector; This represents the number of virtual short exposure subframes.

[0092] Based on the comprehensive displacement, a linear motion fuzzy kernel is generated: ;in, For pixels At this point, the motion blur kernel is offset. The value at; This represents the length of the motion blur kernel, corresponding to the number of pixels. For the comprehensive displacement amplitude, The duration of the virtual short exposure film; These are the offset coordinates within the nuclear plane. It is an integer.

[0093] It needs to be explained that the physical essence of motion blur is the integral effect of the continuous displacement trajectory of the target on the sensor during the exposure period. By transforming the estimated pixel-by-pixel displacement field into a standard linear motion blur kernel model, the kernel length is proportional to the displacement amplitude and inversely proportional to the subframe exposure time, and the kernel direction is consistent with the displacement direction. This pixel-by-pixel adaptive blur kernel will be used to compensate for residual motion blur in subsequent reconstruction.

[0094] In the sparse coding reconstruction stage, when embedding the motion blur kernel into the reconstruction process, let the reconstruction covariance matrix obtained by sparse coding be... It is then mapped back to image patches using the manifold index. Then, perform non-blind deconvolution on each image patch: ;in, The image block after deblurring; For the first Motion blur kernel corresponding to the center pixel of each image patch; Image patches reconstructed from sparse coding; The total variational regularization parameter is set to 0.05 to suppress ringing artifacts generated by deconvolution. For image blocks Gradient magnitude plot; Norms promote gradient sparsity.

[0095] It needs to be explained that sparse coding, based on the Riemannian manifold, achieves geometric decoupling between noise and signal, but the reconstruction result... It may still contain residual motion blur due to imperfections in the inversion; For observation, with Given a fuzzy kernel, residual fuzziness is further eliminated through non-blind deconvolution with total variation regularization. The total variation term... While removing blur, it maintains edge sharpness and avoids edge ringing caused by traditional Wiener filtering.

[0096] All deblurred image blocks The frames are stitched together according to their original spatial positions, and a weighted average of the overlapping areas is performed to obtain a clean frame after denoising and blurring. At the same time, the saliency map of small targets will be used. Normalization to After the interval, it is output as a small target protection mask.

[0097] It should be noted that this module undertakes the dual tasks of geometric decoupling and evidence protection. On the one hand, through dictionary learning and sparse coding on Riemannian manifolds, it achieves intrinsic separation of noise components and clean signal components in nonlinear geometric space, overcoming the fundamental contradiction between denoising and detail preservation in Euclidean space. On the other hand, by transforming the motion displacement field into a physical blur kernel and embedding it into the reconstruction process, residual motion blur is eliminated. Simultaneously, by modulating the sparse regularization intensity through small target saliency maps, it achieves proactive protection of minute evidence at the algorithmic level; the output is a clean frame. and small target protection mask It directly serves as the target distribution anchor point and cost weight modulation factor for Wasserstein optimal transmission in the steps, and is a key intermediate link for the entire system to achieve evidence-level enhancement.

[0098] In this embodiment, image patches are modeled as symmetric positive definite covariance matrices and processed on Riemannian manifolds. Utilizing the nonlinear geometric structure of the manifolds, noise disturbances and signal variations are naturally separable under geodesic distance metrics. Compared to traditional denoising methods in Euclidean space, such as BM3D and NLM, this method exhibits superior noise suppression and structure preservation capabilities under extremely low signal-to-noise ratio conditions. Through spatial adaptive sparse regularization driven by the temporal median saliency map, small target regions and background regions are treated differently at the algorithm optimization level. This avoids the problem of erasing distant pedestrians, vehicles, and other minute evidence caused by the globally uniform denoising intensity of traditional methods, significantly improving the evidence detection rate. Finally, by directly utilizing the estimated physical motion displacement field, a pixel-by-pixel self-... By adapting to the blur kernel rather than blindly estimating it, the algorithm avoids the artifact problem caused by inaccurate kernel estimation in traditional blind deconvolution, resulting in more reliable deblurring results. Using affine-invariant Riemann distance as the metric basis ensures that dictionary learning and sparse coding results are unaffected by linear transformations such as feature scaling and rotation, enhancing the algorithm's generalization ability to different monitoring scenarios and camera parameters. The clean output frames serve as target distribution anchors for subsequent Wasserstein transmission, and their quality directly determines the upper limit of the final enhanced image. Small target protection masks, as transmission cost weight modulation factors, ensure that subsequent steps do not damage the protected evidence structure during optimal transmission. The two outputs and subsequent steps form a tight logical loop.

[0099] The monitoring image enhancement processing module uses the output clean frame as the target distribution anchor point and the original long-exposure degraded frame as the source distribution to construct the Wasserstein optimal transmission mapping with the generated small target protection mask as the transmission cost weight modulation factor. During the mapping solution process, a structural similarity hard constraint is applied to the evidence area. The transmission mapping reconstruction result and the clean frame are adaptively weighted and fused to output the final monitoring image with evidence-level enhancement.

[0100] This module receives the clean frames after denoising and deblurring. and small target protection mask Original long exposure degraded frame For source distribution, with clean frames For the target distribution anchor points, construct an image enhancement map based on Wasserstein's optimal transmission theory;

[0101] The core innovation of this module lies in: using a small target protection mask. The spatial weighting modulation factor, acting as the transmission cost, maps the optimal transmission to the evidence region, incurring a higher transmission cost to maintain structural integrity. Simultaneously, during the mapping solution process, a hard constraint of structural similarity is applied to the evidence region to prevent structural damage from an optimization perspective. Finally, the transmission mapping reconstruction result is adaptively weighted and fused with clean frames, combining the complementary advantages of the two reconstruction paths to output an evidence-enhanced final monitoring image. This module is the final output stage of the entire system, and its quality directly determines the identifiability of distant pedestrians, vehicles, and other minute evidence in the monitoring image. It includes the following sub-steps:

[0102] When defining the source probability measure and the target probability measure, the original long-exposure degradation frame is used. and clean frames output , which are defined as the source probability measure and the target probability measure in the optimal transmission problem, respectively;

[0103] For the original long exposure degraded frame This refers to long-exposure frames at night directly output by surveillance cameras, with a size of [missing information]. After normalizing its pixel intensity, a source probability measure is constructed. : ;in, Source probability measure in pixels The probability mass at the location satisfies and ; For the original long exposure degraded frame in pixels The grayscale value at that location, and its range. ; Let be the probability mass smoothing constant, and let its value be . To prevent pure black pixels with a grayscale value of 0 from resulting in zero probability quality, and to ensure that the source metric has full support; This is the set of indices for all pixels in the image. ; The pixel coordinates are dummy variables used for summation and traversal.

[0104] For clean frames of output Construct a target probability measure : ;in, For the target probability measure in pixels The probability mass at the location satisfies and ; The clean frame after denoising and deblurring is output in pixels The grayscale value at that location; It is a set of indices for all pixels in the image.

[0105] It needs to be explained that Wasserstein's optimal transport theory requires that both the source and target distributions be probability measures defined in the same space, i.e., non-negative and summing to 1. After normalizing the image pixel intensity into a probability distribution, the optimal transport problem is equivalent to finding a transport scheme that redistributes the source distribution (the brightness distribution of the degraded frame) to the target distribution (the brightness distribution of the clean frame) at the lowest cost. This probabilistic representation allows the transport mapping to globally redistribute image brightness, rather than processing it pixel by pixel, thus enhancing dark areas while maintaining the naturalness of the overall brightness distribution; the smoothing constant... The introduction of ensures the full support of the measure, which is a mathematical premise for the finiteness of Wasserstein distance and the existence of transport maps.

[0106] When constructing the transmission cost function of small target protection mask modulation, the generated small target protection mask is used. Spatially modulate the standard secondary transmission cost to construct the transmission cost function for physical evidence perception: ;in, pixel positions from source distribution pixel positions in the target distribution The cost of transmission unit probability quality; The strength coefficient for physical evidence protection ranges from 5 to 20, with 10 being preferred, and controls the modulation amplitude of the small target protection mask on the transmission cost. To generate a small target protection mask in the source pixel The value at the location, the range of values The larger the value, the more likely the pixel is to belong to the area of ​​small target evidence; These are the pixel coordinates in the source distribution; These are the pixel coordinates in the target distribution; It is the square of the Euclidean distance between the source pixel and the target pixel, reflecting the spatial displacement cost.

[0107] It should be noted that in the above formula, the standard Wasserstein optimal transmission uses pure spatial distance. As a transmission cost, this means that the cost of transporting all pixels depends only on spatial distance, without distinguishing between the evidence area and the background area; this is achieved by multiplying by the spatial modulation factor. This amplifies the transmission cost in the evidence area. times: when (In areas with strong physical evidence), the transmission cost is amplified to its original value. times; when In the (pure background region), the transmission cost degenerates into the standard quadratic cost. This cost modulation forces the optimal transmission solver to tend to stay in place in the evidence region, i.e., keep the pixel values ​​unchanged, because any transmission that deviates from its original position will incur an extremely high cost. In the background region, the transmission map can freely redistribute brightness to achieve a global enhancement effect.

[0108] When solving the Wasserstein optimal transport map with hard constraints, an optimal transport optimization problem with hard constraints on the structural similarity of the evidence region is constructed based on the defined transport cost function: ; ;in, For the elements of the transmission plan matrix, representing the source pixel To target pixel The probabilistic quality of transmission satisfies ; Let the coupling set be defined as all sets that satisfy the marginal constraints. ,and Transmission plan A set; The defined cost of transmitting physical evidence; For transmission mapping on pixels The reconstruction results at the location are determined by the transmission plan. It is obtained through centroid mapping calculation; It is a structural similarity exponential function; The structural fidelity threshold is set to 0.92, requiring that the reconstructed evidence area has a structural similarity to the clean frame that is not lower than this threshold. The set of pixels in the evidence area is defined as follows: ,in, The threshold value for mask binarization is 0.3.

[0109] It should be explained that this formula is the standard form of the discrete Wasserstein-2 optimal transmission problem, and the objective function is to minimize the total transmission cost; transmission plan It describes how to transfer the probability mass of the source distribution to the target distribution, and the optimal transfer plan. The transportation scheme that corresponds to the minimum total cost; due to the cost function It has been modulated by a small target protection mask, the optimal solution In the area of ​​physical evidence, there is a natural tendency for transmission to be diagonal, that is... This maintains the structure of the physical evidence unchanged.

[0110] Furthermore, cost modulation alone is a soft constraint. In extreme cases, such as when the brightness difference between the evidence area and the background is extremely large, the transport mapping may still violate the intent of evidence protection. By using structural similarity as a hard constraint, the reconstruction result of the evidence area is forced to meet certain requirements at the level of the feasible optimization domain. With clean frames The structural similarity is no less than that of the structural similarity of the two. This ensures that the edges, textures, and contrasts of the evidence are not destroyed after transmission mapping; this hard constraint, together with cost modulation, forms a dual protection mechanism combining hard and soft elements.

[0111] Based on the optimal transmission plan The reconstruction results of the transfer mapping are calculated through the centroid mapping: ;in, For transmission mapping on source pixels The reconstructed grayscale value at the location; These are elements of the optimal transmission plan matrix; For clean frames at target pixels The grayscale value at the given location; the denominator is the source pixel. The total output probability mass is equal to ;

[0112] It needs to be explained that the center of gravity mapping will affect the transmission plan. This is transformed into a deterministic pixel mapping function. For each pixel in the source distribution Its reconstruction value This is a weighted average of the grayscale values ​​of all pixels in the target distribution, with the weights determined by the optimal transmission plan. The decision is made in the area of ​​physical evidence, due to the combined effect of cost modulation and hard constraints. It is approximately a diagonal matrix, therefore The physical evidence structure was preserved; in the background area, Allows off-diagonal transmission. It achieves global redistribution and enhancement of brightness by weighting multiple pixels in the target distribution.

[0113] The above formula is solved using the Sinkhorn algorithm with entropy regularization, combined with the projected gradient method to handle hard constraints; specifically, an entropy regularization term is introduced. ,in, The entropy regularization coefficient, with a value of 0.01, transforms the original problem into a strictly convex problem. Through Sinkhorn iterations, the row and column normalization factors are updated alternately. After each Sinkhorn iteration, the structural similarity constraint of the evidence region is checked. If it is violated, the transport plan is projected onto the feasible region using a projection operation. The iteration terminates when the marginal constraint residual is less than [value missing]. And all physical evidence area constraints are satisfied, or the maximum number of iterations of 500 is reached;

[0114] In addition, the structural similarity exponential function involved The definition is as follows: ;in, It is based on pixels Within a local window centered on the center, the structural similarity index between the transport mapping reconstruction result T and the clean frame J ranges from 1 to 1. The closer the value is to 1, the more similar the structures are; Therefore Centered The mean of T within a local window. The value is 11; The mean of J within the same window; The variance of T within the same window; The variance of J within the same window; Let T and J be the covariance within the same window; This is a relative stability constant for brightness. ,in, , This refers to the grayscale dynamic range. It is a relatively stable constant for contrast. ,in, .

[0115] It needs to be explained that the structural similarity index, from brightness ( and Comparison), contrast ( and (comparison) and structure ( The similarity of local regions in two images is comprehensively evaluated using three dimensions: linear correlation (reflecting the linear correlation). An application is made to the evidence area. The hard constraints mean that the transmission mapping reconstruction result must be highly consistent with the preceding clean frame in terms of brightness, contrast and structure, so as to ensure that the morphological features of the evidence, such as human body outline and vehicle shape, are not distorted or erased during the enhancement process.

[0116] When adaptive weighted fusion outputs the final enhanced image, the transmission mapping reconstruction result will be used. With clean frames When performing adaptive weighted fusion to generate the final enhanced output, the pixel-by-pixel fusion weight map is first calculated: ;in, For pixels The fusion weight at the point has a value range of 1. This indicates the transport mapping reconstruction result in the final output. The proportion; This is the kurtosis parameter of the Sigmoid function, with a value of 10, which controls the transition speed of the weights from 0 to 1; To transfer the mapping reconstruction results in pixels The grayscale value at that location; For clean frames in pixels The grayscale value at that location; This represents the absolute difference between the two reconstruction results at this pixel. The larger of the two reconstruction results at this pixel is used to normalize the difference; To prevent division by zero, the value is taken as... .

[0117] It needs to be explained that the fusion weight The design is based on the principle of difference-driven design; when and When the differences are small, such as in the area of ​​physical evidence, due to the hard constraint that T≈J, the normalized differences... Approaching 0, after Sigmoid mapping, The two results are fused with equal weight, and the complementary average is taken. When the difference is large, such as in the background region, the transport mapping may produce a different enhancement effect than the clean frame. A deviation of 0.5 adaptively tends to yield a higher-quality result from the side with the larger difference; the smooth transition characteristics of the Sigmoid function avoid hard switching, which can lead to artifacts at the fusion boundary.

[0118] The final enhanced output is ;in, To produce the final output of evidence-grade enhanced surveillance footage, at the pixel level... The grayscale value at that location; For weighting; This is for transmitting the mapping and reconstruction results.

[0119] It should be explained that this formula is a pixel-by-pixel convex combination fusion, ensuring that the output value is always within the range of convex combination fusion. and Between these two ranges, no outliers exceeding the range of either will be generated; within the evidence area, The output is the average of the two reconstructions, combining the global brightness optimization capability of transport mapping with the local structure fidelity capability of clean frames; in the background region, Based on the difference between the two results, an adaptive adjustment is made, selecting the one with the better enhancement effect as the dominant factor, and the final output is... It combines global brightness balance with local evidence fidelity, achieving a final monitoring image with evidence-level enhancement.

[0120] It should be noted that this module is the final output stage of the entire system, undertaking the unified task of global enhancement and physical evidence fidelity; the clean frames output earlier... Although noise and blur are removed, the global brightness distribution may still be less than ideal due to limitations in the accuracy of variational inversion; for example, there may be insufficient enhancement in dark areas and overexposure in bright areas. By using Wasserstein optimal transport mapping, As the target anchor point, with the original degraded frame Using the source as a reference, brightness is redistributed at the global probability distribution level to further improve the overall image quality. At the same time, through the dual mechanisms of small target protection mask modulation transmission cost and structural similarity hard constraint, it is ensured that the global enhancement process will not destroy the protected physical structure. The final adaptive fusion combines the complementary advantages of the two reconstruction paths: transmission mapping is good at global brightness balance, and clean frames are good at local structure fidelity. The fusion of the two outputs a final image that combines the advantages of both.

[0121] In this embodiment, Wasserstein optimal transmission redistributes global brightness at the probability distribution level. Compared to pixel-by-pixel gamma correction or histogram equalization, it can maintain the spatial coherence and naturalness of the image, avoid local over-enhancement or under-enhancement, and fully enhance details in dark areas while preventing overexposure in bright areas. A dual protection mechanism is formed through spatial modulation of transmission costs (soft constraint) and hard constraints of structural similarity (hard constraint), ensuring that the structure of the evidence area is not destroyed by transmission mapping from both the optimization target level and the feasible region level. Compared with a single constraint method, it has stronger robustness. The fusion weights are adaptively adjusted according to the local differences between the two reconstruction results. The complementary average is taken in the evidence area and the optimal selection is made in the background area, avoiding the quality loss caused by the one-size-fits-all approach in fixed weight fusion, so that the final output is close to the optimal reconstruction at each spatial location.

[0122] In the several embodiments provided by this invention, it should be understood that the disclosed system can be implemented in other ways. For example, the embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0123] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module described above can be implemented in hardware or in the form of hardware plus software functional modules.

[0125] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A network surveillance video enhancement system based on machine learning, characterized in that, include: Virtual short exposure frame processing module: Based on the Poisson statistical characteristics of photon shot noise of CMOS sensor, a photon counting degradation model is established for the night long exposure frame output by the surveillance camera. Through variational Bayes inference, the single frame long exposure image is decomposed into multiple virtual short exposure sub-frames. At the same time, the pixel-by-pixel motion displacement field is estimated, and the virtual short exposure frame sequence and the corresponding motion displacement field are output. Image Local Region Processing Optimization Module: The local regions of the images in the obtained virtual short exposure frame sequence are represented as symmetric positive definite covariance matrices. On the Riemann manifold space, dictionary learning and sparse coding are performed through geodesic distance metric to achieve geometric decoupling between noise components and clean signal components. At the same time, the output motion displacement field is used to construct a pixel-by-pixel motion blur kernel. In the sparse coding process, a small target saliency map is introduced as a spatial adaptive regularization constraint. The output is a denoised and deblurred clean frame and a small target protection mask. The monitoring image enhancement processing module uses the output clean frame as the target distribution anchor point and the original long-exposure degraded frame as the source distribution to construct the Wasserstein optimal transmission mapping with the generated small target protection mask as the transmission cost weight modulation factor. During the mapping solution process, a structural similarity hard constraint is applied to the evidence area. The transmission mapping reconstruction result and the clean frame are adaptively weighted and fused to output the final monitoring image with evidence-level enhancement.

2. The network surveillance video enhancement system based on machine learning according to claim 1, characterized in that, When establishing a photon counting degradation model based on Poisson statistical properties, for a long exposure frame with coordinates... The pixels, in the exposure time Internally accumulated received photon count , obeys the parameter The Poisson distribution; Long exposure time Divide the data into N equal virtual short exposure time slices, with each time slice having a duration of: ;in, The duration of a single virtual short exposure time slice; N is the total number of virtual short exposure subframes; In the Within a virtual short exposure time frame, calculate pixels The expected number of photons arriving at a given location is equal to the sum of the number of photons arriving at all N virtual short exposure timepieces, satisfying the conservation constraint.

3. The network surveillance video enhancement system based on machine learning according to claim 2, characterized in that, When constructing the Poisson likelihood function based on motion compensation, the first... Pixels within a time slice The motion displacement vector is Then the pixel is in the th position. Within a given time slice, the photons actually received originate from locations within the scene. ; Based on motion compensation, the first A virtual short-exposure subframe, in pixels The expected number of photons arriving at the location is corrected to: ;in, To take motion compensation into account, the first The expected number of photons arriving at (x,y); Let be the actual photon arrival rate distribution function of the scene; For the first Within a time slice, pixels The horizontal displacement component; For the first Within a time slice, pixels The vertical displacement component.

4. The network surveillance video enhancement system based on machine learning according to claim 3, characterized in that, When updating parameters alternately through coordinate ascending variational inference, the coordinate ascending variational inference method is used to alternately fix one set of variables and update another set of variables. This includes fixing the motion displacement field and updating the variational parameters of the photon arrival rate; and fixing the photon arrival rate and updating the variational parameters of the motion displacement field.

5. The network surveillance video enhancement system based on machine learning according to claim 4, characterized in that, Implement iterative convergence determination. After convergence, output N virtual short exposure subframes and simultaneously output the pixel-by-pixel motion displacement field.

6. The network surveillance video enhancement system based on machine learning according to claim 5, characterized in that, When introducing adaptive sparse coding for small target saliency maps, after dictionary learning, sparse coding is performed on each data point. The temporal median is calculated pixel-by-pixel for N subframes to obtain the temporal median frame. The absolute deviation of each pixel from the temporal median in its neighborhood is calculated, and then Gaussian smoothing is applied to obtain the saliency map. Based on the small target saliency map, a spatial adaptive sparse coding optimization problem is defined, normalizing the small target saliency map to... After the interval, it is output as a small target protection mask.

7. The network surveillance video enhancement system based on machine learning according to claim 5, characterized in that, A pixel-by-pixel motion blur kernel is constructed using the motion displacement field and embedded during deblurring. The average displacement vectors of all N time slices are then taken to obtain the comprehensive displacement. Based on the comprehensive displacement, a linear motion blur kernel is generated. In the sparse coding reconstruction stage, when embedding the motion blur kernel into the reconstruction process, the reconstruction covariance matrix obtained by sparse coding is mapped back to image patches through the manifold exponent, and then non-blind deconvolution is performed on each image patch. All the deblurred image blocks are stitched together according to their original spatial positions, and the overlapping areas are weighted and averaged to obtain a clean frame after denoising and deblurring.

8. The network surveillance video enhancement system based on machine learning according to claim 7, characterized in that, The original long-exposure degraded frame and the output clean frame are defined as the source probability measure and the target probability measure in the optimal transmission problem, respectively.

9. The network surveillance video enhancement system based on machine learning according to claim 8, characterized in that, When solving the Wasserstein optimal transport map with hard constraints, an optimal transport optimization problem with hard constraints of the structural similarity of the evidence region is constructed based on the defined transport cost function. By using structural similarity hard constraints, from the perspective of optimizing the feasible domain, it is required that the structural similarity between the reconstructed evidence region and the clean frame is not lower than a preset value, ensuring that the edges, textures and contrasts of the evidence are not destroyed after transmission mapping.

10. The network surveillance video enhancement system based on machine learning according to claim 9, characterized in that, Based on the optimal transmission plan, the transmission mapping reconstruction result is calculated through centroid mapping; the transmission mapping reconstruction result is then adaptively weighted and fused with clean frames to generate the final enhanced output.

Images