Adaptive noise reduction processing method and system for video stream data

Abstract

The application relates to a self-adaptive noise reduction processing method and system of video stream data. The method comprises the following steps: acquiring a current frame video image and performing cross-scale multi-resolution noise feature pre-evaluation to obtain an initial noise feature map; based on the initial noise feature map, a historical frame video image and the current frame video image, a motion vector field, a motion estimation confidence map and reference frame pixel values are obtained through multi-reference frame joint motion estimation; the initial noise feature map, the motion estimation confidence map and a candidate noise reduction parameter set are input into a lightweight deep prediction network to output a multi-channel marginal benefit prediction map; based on the multi-channel marginal benefit prediction map, a multi-step optimization function is constructed and solved to obtain a time domain fusion weight control map, the current frame video image and the reference frame pixel values are weighted and fused to obtain a current frame noise reduction video image. The method can break the mutual damage spiral between motion estimation and noise reduction intensity and realize self-adaptive noise reduction processing of multi-step forward-looking optimization.

Description

Technical Field

[0001] This invention belongs to the field of video stream data noise reduction technology, and in particular relates to an adaptive noise reduction processing method and system for video stream data. Background Technology

[0002] With the rapid development of video acquisition and transmission technologies, high-definition and even ultra-high-definition video streams are increasingly widely used in everyday communication, security monitoring, and live streaming. However, due to limitations in sensor physical characteristics and channel environment, various types of noise inevitably get mixed into video signals during acquisition and transmission. Motion-compensated temporal denoising technology, because it can make full use of inter-frame redundancy information and preserve image details relatively well while suppressing noise, has become one of the mainstream methods in the field of video denoising.

[0003] Traditional motion-compensated temporal denoising methods typically begin by establishing pixel correspondences between the current frame and historical reference frames using a motion estimation algorithm. Then, they determine the reliability of the motion estimation based on a preset error tolerance threshold: if the estimation error is below the threshold, temporal fusion is performed; otherwise, spatial filtering is reverted to avoid artifacts. This approach achieves good results in low-noise environments, but the overall performance of the denoising system is highly dependent on the accuracy of the motion estimation.

[0004] However, the aforementioned traditional methods have a fundamental flaw: high-noise environments severely pollute the feature matching process of motion estimation, leading to a comprehensive decrease in motion estimation confidence; low-confidence motion estimation triggers the system to retreat to spatial domain filtering, leaving a large amount of noise that should have been eliminated by multi-frame averaging in the temporal domain in the current frame; this residual noise, in turn, pollutes the motion estimation of the next frame, forming a vicious cycle of "noise pollution → motion estimation inaccuracy → temporal domain denoising failure → increased noise residue", ultimately causing the system to completely lose its ability to perform temporal domain denoising. Summary of the Invention

[0005] Therefore, it is necessary to provide an adaptive noise reduction method that can break the mutually destructive spiral between motion estimation and noise reduction intensity and achieve multi-step forward-looking optimization to address the above-mentioned technical problems.

[0006] Firstly, this application provides an adaptive noise reduction processing method for video stream data, including:

[0007] The current frame video image is acquired, and cross-scale multi-resolution noise features are pre-evaluated on the current frame video image to obtain an initial noise feature map;

[0008] Based on historical frame video images in the historical state cache library, multi-reference frame joint motion estimation is performed on the current frame video image to obtain the motion vector field of the current frame video image relative to the historical frame video images. Based on each motion vector field, the temporal consistency deviation of each pixel position is calculated.

[0009] Based on the temporal consistency deviation and the initial noise feature map, the motion estimation confidence and the reference frame pixel value are calculated for each pixel location to obtain the motion estimation confidence map and the reference frame pixel map;

[0010] The initial noise feature map, motion estimation confidence map, and candidate denoising parameter set are input into a lightweight deep prediction network, which outputs a multi-channel marginal benefit prediction map; each channel of the multi-channel marginal benefit prediction map corresponds to a candidate denoising parameter.

[0011] A multi-step optimization function is constructed with the denoising quality of the current frame video image and the marginal gain of the motion estimation confidence of the future frame video image as the objective. Based on the multi-channel marginal gain prediction graph, the multi-step optimization function is solved to obtain the temporal fusion weight control graph.

[0012] Based on the temporal fusion weight control graph, the current frame video image and the reference frame pixel map are weighted and fused to obtain the current frame denoised video image.

[0013] Furthermore, a multi-scale, multi-resolution noise feature pre-evaluation is performed on the current frame video image to obtain an initial noise feature map, including:

[0014] Based on the current frame video image, construct an image pyramid including the original resolution layer, the 1 / 2 resolution layer, and the 1 / 4 resolution layer;

[0015] At each scale level of the image pyramid, the local evaluation window is slid in a step equal to half the window size, and the gray variance of each local evaluation window is calculated based on the pixel gray values ​​within the local evaluation window.

[0016] By combining the gray-level variances belonging to the same scale layer, we obtain the variance set map of each scale layer;

[0017] Based on the window gray-level distribution histogram of the corresponding pixel positions between each pair of adjacent scale layers, the cross-entropy difference of each pixel position between each pair of adjacent scale layers is calculated.

[0018] The cross-entropy difference at each pixel location is weighted and fused to obtain a comprehensive value of the cross-scale difference at each pixel location;

[0019] Based on the variance ensemble plot and the cross-scale difference comprehensive value, a weighted summation operation is performed on each pixel location to generate an initial noise feature map.

[0020] Furthermore, based on the temporal consistency deviation and the initial noise feature map, the motion estimation confidence score and the reference frame pixel value are calculated for each pixel location, resulting in a motion estimation confidence map and a reference frame pixel map, including:

[0021] Based on the initial noise feature map and the standard deviation of the preset noise model in the neighborhood of the current pixel, combined with the temporal consistency deviation, the motion estimation confidence of each pixel position is calculated, and all motion estimation confidences are combined to obtain the motion estimation confidence map.

[0022] Based on the motion vector field, the earliest frame video image in each historical frame video image is reverse twisted and mapped to obtain the reference frame pixel value at each pixel position. All reference frame pixel values ​​are combined to obtain the reference frame pixel map.

[0023] Furthermore, the initial noise feature map, motion estimation confidence map, and candidate denoising parameter set are input into a lightweight deep prediction network, which outputs a multi-channel marginal benefit prediction map, including:

[0024] According to the partitioning rules, the confidence level in the motion estimation confidence map is divided into high confidence intervals, medium confidence intervals and low confidence intervals, and candidate denoising parameters are assigned to the high confidence intervals, medium confidence intervals and low confidence intervals respectively, thus obtaining a set of candidate denoising parameters;

[0025] Obtain the local texture complexity map and local motion intensity map of the current frame video image;

[0026] The local texture complexity map, local motion intensity map, initial noise feature map, motion estimation confidence map, and candidate denoising parameter set are simultaneously input into a lightweight deep prediction network, which outputs a multi-channel marginal benefit prediction map.

[0027] Furthermore, a multi-step optimization function is constructed with the denoising quality of the current frame video image and the marginal gain of motion estimation confidence of the future frame video image as the objectives. Based on the multi-channel marginal gain prediction graph, the multi-step optimization function is solved to obtain the temporal fusion weight control graph, including:

[0028] A multi-step optimization objective function is constructed, with the goal of improving the denoising quality of the current frame video image and increasing the marginal gain of the motion estimation confidence of the future frame video image. The formula for calculating the multi-step optimization objective function is as follows:

[0029] ;

[0030] in, To optimize the objective function in multiple steps, This is a local noise reduction quality evaluation function. pixel position Local texture complexity at the location, pixel position The intensity of localized movement at the location The parameters corresponding to the multi-channel marginal revenue prediction graph The projected marginal revenue value, For the current parameter Local variance within the spatial neighborhood , and These are the preset weighting coefficients;

[0031] For each pixel location, the multi-step optimization objective function is discretely solved based on the multi-channel marginal benefit prediction map to obtain the optimal candidate noise reduction parameters for each pixel location;

[0032] The optimal candidate denoising parameters are determined as the optimal temporal fusion weights, and all optimal temporal fusion weights are combined to obtain the initial fusion weight map;

[0033] Using the luminance channel of the current frame video image as the guide map, the initial fusion weight map is subjected to guided filtering and edge-preserving smoothing to generate a temporal fusion weight control map.

[0034] Secondly, this application also provides an adaptive noise reduction processing system for video stream data, comprising:

[0035] The noise assessment module is used to acquire the current frame video image and perform cross-scale multi-resolution noise feature pre-assessment on the current frame video image to obtain an initial noise feature map;

[0036] The vector field generation module is used to perform multi-reference frame joint motion estimation on the current frame video image based on historical frame video images in the historical state cache library, to obtain the motion vector field of the current frame video image relative to the historical frame video images, and to calculate the temporal consistency deviation of each pixel position based on each motion vector field.

[0037] The confidence calculation module calculates the motion estimation confidence and reference frame pixel value for each pixel location based on the temporal consistency deviation and the initial noise feature map, thus obtaining the motion estimation confidence map and the reference frame pixel map.

[0038] The marginal revenue prediction module is used to input the initial noise feature map, motion estimation confidence map and candidate denoising parameter set into the lightweight deep prediction network, and output a multi-channel marginal revenue prediction map; each channel of the multi-channel marginal revenue prediction map corresponds to a candidate denoising parameter;

[0039] The function solving module is used to construct a multi-step optimization function with the goal of improving the noise reduction quality of the current frame video image and the marginal gain of the motion estimation confidence of the future frame video image. Based on the multi-channel marginal gain prediction graph, the multi-step optimization function is solved to obtain the temporal fusion weight control graph.

[0040] The image denoising module is used to weightedly fuse the current frame video image and the reference frame pixel image according to the temporal fusion weight control map to obtain the current frame denoised video image.

[0041] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement an adaptive noise reduction processing method for video stream data as described in any of the embodiments of this application.

[0042] Fourthly, this application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement an adaptive noise reduction processing method for video stream data as described in any of the embodiments of this application.

[0043] The aforementioned adaptive denoising method and system for video stream data obtains an initial noise feature map by acquiring the current frame video image and performing cross-scale multi-resolution noise feature pre-evaluation. Based on the initial noise feature map, and combining historical frame video images in the historical state cache with the current frame video image, motion vector field, motion estimation confidence map, and reference frame pixel values ​​are obtained through multi-reference frame joint motion estimation. The initial noise feature map, motion estimation confidence map, and candidate denoising parameter set are input into a lightweight deep prediction network, which outputs a multi-channel marginal benefit prediction map. Based on the multi-channel marginal benefit prediction map, a multi-step optimization function is constructed and solved with the denoising quality of the current frame video image and the marginal gain of motion estimation confidence of future frame video images as objectives, resulting in a temporal fusion weight control map. The current frame video image and reference frame pixel values ​​are then weighted and fused to obtain the current frame denoised video image.

[0044] It can fundamentally break the mutually destructive spiral between motion estimation error and noise reduction intensity in traditional noise reduction methods, and maintain the advantages of temporal domain noise reduction in high-noise environments. It avoids the system from collapsing into a pure spatial domain noise reduction mode. At the same time, through pixel-by-pixel adaptive strategy distribution, it achieves a better balance between texture protection and noise suppression, significantly improving the overall effect and temporal stability of video stream noise reduction. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating an adaptive noise reduction method for video stream data in one embodiment.

[0047] Figure 2This is a flowchart illustrating the steps of calculating the motion estimation confidence level and reference frame pixel value for each pixel location based on temporal consistency deviation and initial noise feature map in one embodiment, thereby obtaining the motion estimation confidence map and reference frame pixel map.

[0048] Figure 3 This is a schematic diagram of the structure of an adaptive noise reduction processing system for video stream data in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In one embodiment, an adaptive noise reduction method for video stream data is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. Figure 1 As shown, in this embodiment, the method includes the following steps:

[0051] S101: Acquire the current frame video image and perform cross-scale multi-resolution noise feature pre-evaluation on the current frame video image to obtain the initial noise feature map.

[0052] Among them, cross-scale multi-resolution noise feature pre-evaluation refers to constructing multiple image scale levels with different resolutions for the currently acquired original video image frame, analyzing the gray-scale statistical characteristics of local image regions at each scale, and further comparing the statistical feature differences at the same spatial location between different scales, thereby effectively distinguishing real image texture and noise interference, and finally generating an initial noise feature map that can reflect the noise level distribution of each pixel position in the current frame.

[0053] For example, taking the received video stream as the processing object, the current video image frame to be processed is extracted from the video stream, denoted as... ,in The frame number is used. A multi-scale, multi-resolution noise feature pre-evaluation is performed on the current frame video image to obtain an initial noise feature map.

[0054] S102, based on the historical frame video images in the historical state cache library, perform multi-reference frame joint motion estimation on the current frame video image to obtain the motion vector field of the current frame video image relative to the historical frame video images, and calculate the temporal consistency deviation of each pixel position based on each motion vector field.

[0055] The historical state cache refers to a data storage unit with a first-in, first-out (FIFO) queue structure, used to continuously maintain several recently processed video frames and their corresponding auxiliary information. This cache stores the most recent... The noise reduction output of the frame is usually The values ​​range from 5 to 8, and each frame contains the motion vector field, motion estimation confidence map, temporal fusion weight control map, residual noise evaluation map, and temporal fluctuation index map generated during the previous processing. When a new frame completes the denoising process, all its relevant data are pushed into the tail of the queue as a complete entry, while the oldest entry at the head of the queue is removed, thus ensuring that the cache always contains the latest historical information. Multi-reference frame joint motion estimation refers to using multiple processed video frames stored in the historical state cache as reference frames during the video denoising process, matching them with the current frame, calculating the block displacement (motion vector), and obtaining multiple sets of motion vector fields.

[0056] For example, the pixel position is denoted as ,in It is an explicit two-dimensional coordinate representation. For horizontal indexing, For vertical indexing; and This is the abbreviated form of the coordinates. Retrieve the three most recent noise-reduced video frames from the historical state cache, denoised as follows: , and These three frames are used as reference frames for motion estimation. The current frame is used as the reference frame. Divided into non-overlapping groups (e.g., an 8×8) pixel block, for each pixel block, with the center of the block at the corresponding position in the reference frame as the origin, the size is set to... A rectangular search window (e.g., 33×33) is used; the maximum search radius is set. The initial search step size is set with the position of the current block center in the reference frame as the origin. Starting from the origin, eight adjacent candidate points are placed along the horizontal and vertical directions at the current step size, totaling nine candidate points including the origin itself. The sum of absolute errors (SAD) of the current block and the reference block containing each candidate point is calculated. The candidate point with the smallest SAD is selected as the best matching point for the first step. Using the best matching point as the center, the search step size is halved, and eight more candidate points are placed around it, with the matching error value calculated again. The point with the smallest matching error is selected as the best matching point for this step. The search step size is halved again, and the above operation is repeated. This process continues until the step size converges to a preset minimum value, at which point the search terminates. The displacement between the current best matching point and the origin is the motion vector of the block. The formula for calculating the sum of absolute errors (SAD) is as follows:

[0057] .

[0058] Where B represents the side length of the current pixel block, in pixels; m and n are values ​​from 0 to n respectively. The loop index is used to iterate through each pixel within the block; it indicates the current block to be matched in the current frame. The coordinates of the top left corner are determined by the image segmentation process, and each segment corresponds to a unique coordinate. ; Indicates the current frame The coordinates within the current block The pixel grayscale value at that location, when When, it corresponds to the pixel at the top left corner of the block; when At that time, it corresponds to the pixel in the lower right corner of the block; This represents the offset vector within the search window in the reference frame, i.e., the candidate motion vector, where... This is the horizontal offset; a positive value indicates to the right, and a negative value indicates to the left. This is the vertical offset; a positive value indicates downward, and a negative value indicates upward. The search window range is determined by the maximum search radius. Decision, usually ; Indicates in reference frame In the middle, from the coordinates As the starting point at the top left corner of the block, the offset within the block The corresponding pixel grayscale value; Indicates the current offset Below, the sum of the absolute grayscale differences of all corresponding pixels between the current block and the reference block. The smaller this value, the smaller the difference between the two blocks, that is, the higher the degree of matching. The preset minimum value refers to the lowest value or parameter benchmark that is set in advance and cannot be reduced further. Bilinear interpolation refers to obtaining the value of the target point by using four adjacent known pixels in a two-dimensional plane, first by linear interpolation twice in the x direction, and then by interpolation once in the y direction (or vice versa).

[0059] For example, for the current frame After performing the above operations on each pixel block, the motion vector of each block is obtained. The optimal matching point around the integer pixel is then calculated. Within a pixel range, the sub-pixel positions are calculated using bilinear interpolation to determine the pixel values ​​of the reference block at each sub-pixel offset. The matching error of each candidate sub-pixel point is calculated using the same SAD criterion. The sub-pixel offset with the smallest SAD is selected to correct the motion vector, resulting in a sub-pixel precision motion vector. The sub-pixel precision motion vector of each block is assigned to all pixels within that block, yielding a sub-pixel precision motion vector field. This field is then smoothed using median filtering to eliminate outlier vectors, resulting in a motion vector field from the current frame. The motion vector field to the reference frame, for three reference frames , , By repeating all the above steps, three sets of motion vector fields can be obtained. , , Each set of motion vector fields is a two-dimensional vector field with the same size as the original image. Each pixel position stores the two-dimensional displacement vector of that pixel from the current frame to the corresponding reference frame. For the current frame... For each pixel position p, the timing consistency deviation is calculated using the following formula:

[0060] .

[0061] in, The timing consistency deviation, in its physical sense, means that if motion estimation is accurate, the displacement vector from the current frame to the two previous frames should be equal to the vector sum of the displacement vector from the current frame to the previous frame and the displacement vector from the previous frame to the two previous frames. This deviation value... The smaller the value, the better the motion vector field between the three frames satisfies temporal consistency, meaning the higher the reliability of the motion estimation. Median filtering smoothing refers to sorting all gray values ​​within a pixel's neighborhood (e.g., a 3×3 window) and replacing the center pixel of the window with the median value. For example, by traversing all pixel positions, a temporal consistency deviation map with the same size as the current frame is obtained.

[0062] S103, based on the temporal consistency deviation and the initial noise feature map, calculate the motion estimation confidence and the reference frame pixel value at each pixel location to obtain the motion estimation confidence map and the reference frame pixel map.

[0063] For example, by combining the temporal consistency deviation, the initial noise feature map, and the preset noise model standard deviation, the motion estimation confidence score is calculated for each pixel location. All motion estimation confidence scores are then combined by pixel location to obtain a motion estimation confidence map. Based on the motion vector field... The process involves performing a reverse warp mapping on the earliest frame of each historical video image to generate motion-compensated and aligned reference frame pixel values. These reference frame pixel values ​​are then combined according to their pixel positions to obtain a reference frame pixel map. The reverse warp mapping refers to using the motion vector field calculated from the previous frame to inversely deduce the corresponding pixel position of the target frame from the reference frame pixel coordinates, and then sampling and assigning the values.

[0064] S104 inputs the initial noise feature map, motion estimation confidence map, and candidate denoising parameter set into the lightweight deep prediction network and outputs a multi-channel marginal benefit prediction map; each channel of the multi-channel marginal benefit prediction map corresponds to a candidate denoising parameter.

[0065] For example, an initial noise feature map, a motion estimation confidence map, and a set of candidate denoising parameters are input into a lightweight deep prediction network, which outputs a multi-channel marginal benefit prediction map. Each channel corresponds to a candidate denoising parameter, and each pixel position within a channel outputs a marginal benefit prediction value, representing the expected change in motion estimation confidence at the same pixel position in the next frame after denoising with the candidate parameter in the current frame. Positive values ​​represent gains, which help break the mutual-harm spiral, while negative values ​​represent losses, which exacerbate the spiral.

[0066] S105, construct a multi-step optimization function with the objective of the noise reduction quality of the current frame video image and the marginal gain of the motion estimation confidence of the future frame video image, and solve the multi-step optimization function based on the multi-channel marginal gain prediction map to obtain the temporal fusion weight control map.

[0067] For example, a multi-step optimization function is constructed that considers both the denoising quality of the current frame and the marginal impact on the motion estimation confidence of future frames. The expression for this multi-step optimization function is:

[0068] .

[0069] in, This is a local noise reduction quality evaluation function; Indicates the current candidate parameter The variance of the selected parameters within the spatial neighborhood and between adjacent pixels; These are preset weighting coefficients. For example, for each candidate denoising parameter, the corresponding predicted marginal revenue value is read from the multi-channel marginal revenue prediction graph and substituted into the multi-step optimization function to calculate... Selecting The largest This serves as the optimal temporal fusion weight for that pixel location. The above selection is performed for all pixel locations to obtain the temporal fusion weight control map.

[0070] S106. Based on the temporal fusion weight control map, the current frame video image and the reference frame pixel map are weighted and fused to obtain the current frame denoised video image.

[0071] Weighted fusion refers to assigning weights to results from different sources and summing them up to obtain the fused result.

[0072] For example, for each pixel location, the corresponding temporal fusion weight is read from the temporal fusion weight control map. The value of this weight ranges from [0,1]. A value closer to 0 indicates that the pixel location relies more on the motion estimation result and tends to use multi-frame temporal averaging to achieve high-intensity noise reduction. A value closer to 1 indicates that the motion estimation at this location is unreliable, and more of the original pixel should be preserved or spatial filtering should be relied upon to avoid motion artifacts. Combining the pixel map of the reference frame, the noise-reduced output value is calculated pixel by pixel using a weighted fusion formula, as follows:

[0073] .

[0074] in, Indicates the pixel position of the current frame of video image. The final output pixel value after noise reduction processing; It is the original pixel value of the current frame video image at the same position, without any noise reduction processing; These are the reference frame pixel values ​​aligned through motion compensation. This refers to the temporal fusion weights. For example, the above fusion calculation is performed on all pixel locations to obtain the complete denoised video image of the current frame. This denoising result, along with the motion estimation confidence map, the temporal fusion weight control map, and other quality assessment metadata, is packaged and stored in a historical state cache for processing subsequent frames. Simultaneously, the denoised video image of the current frame is used as the historical reference frame for motion estimation in the next frame, forming a complete closed-loop feedback loop.

[0075] This application provides an adaptive noise reduction method for video stream data. Starting from the pre-evaluation of noise features of the current frame, the method obtains the motion vector field and confidence map through joint motion estimation of multiple reference frames. It uses a lightweight deep prediction network to predict the marginal impact of different noise reduction strategies on motion estimation of future frames. Under a multi-step optimization framework, it solves for the optimal temporal fusion weight that can balance the current noise reduction quality and the ability to maintain future motion estimation. It performs weighted fusion to obtain the noise reduction result and feeds the output back to the historical cache to form a closed loop.

[0076] It can fundamentally break the mutually destructive spiral between motion estimation error and noise reduction intensity in traditional noise reduction methods, and maintain the advantages of temporal domain noise reduction in high-noise environments. It avoids the system from collapsing into a pure spatial domain noise reduction mode. At the same time, through pixel-by-pixel adaptive strategy distribution, it achieves a better balance between texture protection and noise suppression, significantly improving the overall effect and temporal stability of video stream noise reduction.

[0077] In one embodiment, a cross-scale, multi-resolution noise feature pre-evaluation is performed on the current frame video image to obtain an initial noise feature map, including:

[0078] S201, based on the current frame video image, constructs an image pyramid including the original resolution layer, the 1 / 2 resolution layer, and the 1 / 4 resolution layer.

[0079] For example, the current frame video image With original width and height To capture the noise distribution characteristics at different scales, a Gaussian pyramid was constructed for the image: a Gaussian low-pass filter was applied to the previous frame of the video image, using a two-dimensional Gaussian function as the filter kernel to eliminate interference from high-frequency details in subsequent sampling. The filtered image was then downsampled, i.e., every other pixel was deleted (one row and one column were removed), resulting in a pyramid with a width of [missing information]. Height is A half-resolution layer image. Then, using this half-resolution image as input, repeat the low-pass filtering and downsampling operations to obtain a width of... Height is The image is a 1 / 4 resolution layer. The original resolution layer, the 1 / 2 resolution layer, and the 1 / 4 resolution layer together form a three-layer image pyramid. This pyramid is an image matrix with three scales, denoted as follows: , , ,in This is the original resolution layer.

[0080] Gaussian low-pass filtering is a commonly used linear filtering technique in image or signal processing. It uses a Gaussian function as a convolution kernel (i.e., a filter) to smooth the input signal or image.

[0081] S202, at each scale level of the image pyramid, the local evaluation window is slid in a step equal to half the window size, and the gray-level variance of each local evaluation window is calculated based on the pixel gray-level values ​​within the local evaluation window.

[0082] For example, for each scale layer in the image pyramid ( A local evaluation window size corresponding to this scale is set. The window size can be reduced accordingly as the resolution decreases; for example, a larger window is used at the original resolution layer, and a smaller window is used at the 1 / 4 resolution layer. The window is continuously slid horizontally and vertically with a step size equal to half the window size, maintaining a 50% overlap between adjacent windows to avoid information loss at window boundaries and control computational density. At each sliding window position, the grayscale values ​​of all pixels within the window are extracted, forming a one-dimensional data sequence. The sample variance of this sequence, i.e., the grayscale variance, is calculated using the following formula:

[0083] .

[0084] in, The variance is the grayscale variance, which reflects the degree of dispersion of pixel values ​​within the window: in flat areas, the variance is mainly contributed by noise; in textured areas, the variance also contains texture structure information. The total number of pixels within the window. For the first grayscale value of each pixel. The mean grayscale value within the window is used. For example, the variance value calculated for each window is assigned to the center pixel position of that window, serving as the local noise and texture blending feature at that location at that scale. After performing sliding and variance calculations for all windows, the grayscale variance for each local evaluation window corresponding to each scale layer is obtained.

[0085] S203 combines the gray-level variances belonging to the same scale layer to obtain the variance set map of each scale layer.

[0086] For example, for each scale layer in the image pyramid In step S202, a grayscale variance value has been calculated for each window on this layer, and each variance value is mapped to the center pixel coordinates of the corresponding window. Since adjacent windows have a 50% overlap, the variance values ​​of different windows spatially cover all pixel locations in the image of this layer, but each pixel location may be covered by multiple windows. A single variance representation of each pixel location on this scale layer can be obtained by summing the variance values ​​of all windows covering the same pixel location. Specifically, an empty matrix with the same size as the current scale layer image is initialized. This is used to store the final pixel-level variance value of the layer. It iterates through all window variance values. For the center pixel coordinates of each window, the variance value of that window is assigned to all pixels within a neighborhood centered at those center pixel coordinates. The size of this neighborhood is equal to the window size. When a pixel location is covered by multiple windows, the average of these window variance values ​​is taken as the final variance value for that pixel to smooth the transition at the window boundaries. For edge pixels not covered by any window center, nearest neighbor interpolation can be used for padding. After the above summary processing, each scale layer... Generate a variance ensemble plot that perfectly matches the resolution of this layer. The value at each pixel location represents the local variance feature at that location at that scale. Nearest neighbor interpolation means that the new pixel value is directly taken from the original image pixel whose coordinates are closest to it, without any weighted calculation.

[0087] S204: Based on the window gray-level distribution histogram of the corresponding pixel positions between each pair of adjacent scale layers, calculate the cross-entropy difference of each pixel position between each pair of adjacent scale layers.

[0088] For example, for two adjacent scale layers in an image pyramid, i.e., the original resolution layer and 1 / 2 resolution layer and 1 / 2 resolution layer and 1 / 4 resolution layer The cross-entropy difference between each pair of adjacent layers is calculated. Since the resolution of adjacent layers differs by a factor of two, the pixel positions of the lower-resolution layer are upsampled to the size of the higher-resolution layer using bilinear interpolation, ensuring a one-to-one correspondence between the pixel coordinates of the two layers. For each pixel position, a local window centered on that pixel, with the same size as the higher-resolution layer window in step S202, is extracted. The frequency of grayscale values ​​for all pixels within this local window is counted, generating a grayscale distribution histogram. The histogram of the high-resolution layer window is denoted as... The histogram of the window after upsampling the low-resolution layer at the corresponding position is denoted as... The two histograms are then normalized so that their sum of probabilities is 1. The calculation is performed from... arrive Kullback-Leibler divergence, also known as cross-entropy difference. The calculation formula is:

[0089] .

[0090] in This is the bin index of the histogram. The physical meaning of this dissimilarity is that the texture of a real image maintains a relatively stable gray-level distribution structure during multi-scale transformations, therefore... The value is relatively small; however, the randomness of the gray-level distribution during downsampling can cause a significant change in the distribution pattern, therefore... The value is relatively large. Through this calculation, two cross-entropy difference values ​​are obtained for each pixel location, corresponding to... Layer pairs and Layer pairing. Normalization refers to calculating the total sum of the values ​​across the two original histograms, then performing probability normalization by dividing the frequency by the sum of the corresponding histograms, resulting in a normalized probability histogram. The sum of all values ​​in each normalized histogram is 1. Upsampling refers to mapping and enlarging low-resolution, small-size feature maps / images into high-resolution, large-size data, thus increasing the number of pixels.

[0091] S205, weighted fusion of the cross-entropy difference at each pixel location to obtain the comprehensive value of the cross-scale difference at each pixel location.

[0092] For example, the two cross-entropy difference values ​​calculated for each pixel location are denoted as follows: (correspond and (layer pairs) and (correspond and (Layer pair). Set two weighting coefficients. and , respectively corresponding and The weights, and satisfying For each pixel location p, calculate the comprehensive value of cross-scale difference, using the following formula:

[0093] .

[0094] After performing calculations for all pixel locations, a cross-scale difference composite image with the same layer size as the original resolution is generated. The value of each pixel in this image is used to reflect the degree of grayscale distribution instability at that location under multi-scale transformation.

[0095] S206, based on the variance ensemble plot and the cross-scale difference comprehensive value, performs a weighted summation operation on each pixel location to generate an initial noise feature map.

[0096] For example, due to the variance set graph and The resolution is lower than the original resolution, so it is upsampled to the original resolution using bilinear interpolation. The same size ensures that all variance ensemble plots are spatially aligned pixel-by-pixel. For each pixel position p, a weighted summation is performed, calculated using the following formula:

[0097] .

[0098] in, The initial noise features for pixel position p; , , These are the preset weighting coefficients for the variance terms of the original resolution layer, the 1 / 2 resolution layer, and the 1 / 4 resolution layer, respectively. These are the preset weighting coefficients for the cross-scale variability term. Each preset weighting coefficient is set in advance and is used to control the contribution of the variances of the three scales and the cross-scale variability term to the results. This is a comprehensive map of cross-scale differences. It includes all pixel locations. The values ​​are combined to generate an initial noise feature map, which is the same size as the original video frame. The value of each pixel is used to characterize the overall degree of noise contamination and structural characteristics at that location.

[0099] In one embodiment, such as Figure 2As shown, based on the temporal consistency deviation and the initial noise feature map, the motion estimation confidence score and the reference frame pixel value are calculated for each pixel location, resulting in a motion estimation confidence map and a reference frame pixel map, including:

[0100] S301, based on the initial noise feature map and the standard deviation of the preset noise model in the neighborhood of the current pixel, combined with the temporal consistency deviation, calculates the motion estimation confidence of each pixel position, and combines all motion estimation confidences to obtain the motion estimation confidence map.

[0101] The initial noise feature map reflects the overall degree of noise contamination at each pixel position in the current frame; the preset noise model standard deviation is pre-calibrated or configured offline according to sensor characteristics, and is used to characterize the background noise level of the image acquisition device when there is no signal input, which is usually related to factors such as temperature; the image acquisition device refers to the hardware device that converts the real optical image into a digital image signal, and the core is photoelectric conversion; the temporal consistency deviation reflects the degree of temporal consistency between the motion vector fields of the current frame and three historical reference frames. The smaller the deviation, the more reliable the motion estimation.

[0102] For example, for each pixel position Based on the initial noise feature map, the standard deviation of the preset noise model in the neighborhood of the current pixel, and the temporal consistency deviation, the motion estimation confidence is calculated using the following formula:

[0103] .

[0104] in, To estimate the confidence level for motion, This is the initial noise feature map; This is a timing consistency deviation; This represents the standard deviation of the preset noise model within the neighborhood of the current pixel. The physical meaning of the formula is: the larger the temporal consistency deviation, the higher the initial noise eigenvalue, and the larger the preset noise standard deviation, the larger the overall deviation, and the closer the confidence level is to 0 after the Sigmoid transformation; conversely, the closer it is to 1. The Sigmoid function is a function that maps any real number to... The sigmoid function over an interval is expressed as:

[0105] .

[0106] Where z is any real number. For example, after traversing all pixel positions, the motion estimation confidence of each pixel position is obtained. All motion estimation confidences are combined to obtain a pixel-wise motion estimation confidence map with the same size as the current frame.

[0107] S302, based on the motion vector field, performs reverse distortion mapping on the earliest frame video image in each historical frame video image to obtain the reference frame pixel value at each pixel position, and combines all the reference frame pixel values ​​to obtain the reference frame pixel map.

[0108] For example, obtaining the motion vector field This vector field represents the current frame. up to the most recent historical reference frame Pixel-by-pixel displacement. Simultaneously, the earliest frame video image is read from the historical state cache. As the source image to be mapped. For each pixel coordinate in the current frame. From the motion vector field Read the displacement vector at that position Calculate the sampling coordinates in the source image Because motion vectors may have sub-pixel precision, sampling coordinates... These coordinates are usually not integers. Bilinear interpolation can be used to extract them from the source image. The pixel value at that location is obtained by weighted interpolation using the four surrounding integer pixels. Simultaneously, a weight reduction flag for each pixel is recorded: if the sampling coordinates... If a pixel exceeds the source image boundary or is located in a motion-occluded region (i.e., due to inconsistencies caused by multiple current frame pixels mapping to the same source pixel location), the marker is set to valid and used to reduce the fusion weight of that pixel during subsequent temporal fusion. After performing the inverse warp mapping on all pixels, the motion-compensated aligned reference frame pixel values ​​and their corresponding weight reduction marker maps are obtained. Combining all reference frame pixel values ​​yields the reference frame pixel map.

[0109] In one embodiment, an initial noise feature map, a motion estimation confidence map, and a set of candidate denoising parameters are input into a lightweight deep prediction network, which outputs a multi-channel marginal return prediction map, including:

[0110] S401. According to the partitioning rules, the confidence level in the motion estimation confidence map is divided into high confidence intervals, medium confidence intervals and low confidence intervals, and candidate denoising parameters are assigned to the high confidence intervals, medium confidence intervals and low confidence intervals respectively, thus obtaining a set of candidate denoising parameters.

[0111] For example, a pixel-by-pixel motion estimation confidence map is obtained, in which the value of each pixel is located at... Interval. Two confidence thresholds are preset, namely the first threshold. Second threshold And satisfy Based on these two thresholds, the confidence interval is divided into three sub-intervals: the low confidence interval is defined as... The middle confidence interval is defined as The high confidence interval is defined as Each sub-interval corresponds to a certain level of motion estimation confidence: a low confidence interval indicates that the motion estimation is unreliable, a medium confidence interval indicates that the motion estimation has some confidence but uncertainty, and a high confidence interval indicates that the motion estimation is highly reliable. A candidate denoising parameter is pre-assigned to each sub-interval, denoted as […]. , , ,in It tends to use strong spatial filtering, while having a larger weight for time-domain fusion. It tends to have strong temporal multi-frame averaging, while having relatively small temporal fusion weights; These three candidate parameters fall somewhere in between. They constitute the candidate denoising parameter set. .

[0112] S402, obtain the local texture complexity map and local motion intensity map of the current frame video image.

[0113] Among them, the local texture complexity map is used to characterize whether the region where each pixel is located in the current frame image is a flat region or a texture-rich region; the local motion intensity map is used to characterize the motion intensity of each pixel in the aforementioned multi-reference frame joint motion estimation.

[0114] For example, a texture determination index is calculated for each pixel location based on the initial noise feature map and the local gray-level variance ensemble map of the original resolution layer. Since the initial noise feature map contains both noise and texture information, and the local gray-level variance is significantly higher in textured areas than in flat areas, the ratio of the local gray-level variance to the noise-compensated reference frame pixel value can suppress noise interference and highlight the true texture. A larger ratio indicates richer texture, and a smaller ratio indicates flatter texture. After normalization, a local texture complexity map is obtained. This is applied to three sets of motion vector fields. , , Calculate the average of the magnitudes of the three displacement vectors at each pixel position using the following formula:

[0115] .

[0116] in The average value at pixel position p represents the intensity of local motion. The larger the average value, the more intense the motion at that position. After normalization, a local motion intensity map is obtained.

[0117] S403 simultaneously inputs the local texture complexity map, local motion intensity map, initial noise feature map, motion estimation confidence map, and candidate denoising parameter set into a lightweight deep prediction network, and outputs a multi-channel marginal benefit prediction map.

[0118] The lightweight deep prediction network adopts an encoder-decoder architecture, with the total number of parameters controlled below 200k to meet the requirements of real-time video stream processing. The specific parameter structure is as follows:

[0119] The encoder consists of five depthwise separable convolutional blocks stacked sequentially. Each depthwise separable convolutional block contains a depthwise convolutional layer and a pointwise convolutional layer. The depthwise convolutional layer performs spatial convolution operations independently on each input channel, and its parameter count is... ,in Input the number of channels. The kernel size is used for pointwise convolutional layers. Convolution achieves information fusion between channels, and its parameter count is... , This represents the number of output channels. The kernel size is [value to be filled in]. The number of channels in each block changes progressively, specifically as follows: the first block has 4 input channels, corresponding to the concatenation of channels from the four input feature maps, and outputs 16 channels; the second block has 16 input channels and outputs 32 channels; the third block has 32 input channels and outputs 64 channels; the fourth block has 64 input channels and outputs 128 channels; and the fifth block has 128 input channels and outputs 256 channels. Each depthwise separable convolutional block is followed by a batch normalization layer and a ReLU activation function to accelerate network convergence and enhance nonlinear expressive power. The encoder's five downsampling blocks successively halve the spatial size of the feature maps, resulting in a final output deep feature map size that is one-quarter of the original input.

[0120] The decoder employs a combination of subpixel convolution and depthwise separable convolution for upsampling, progressively restoring spatial resolution. The core operation of subpixel convolution is pixel shuffling: rearranging the channel dimensions of the input feature map to the spatial dimensions, thus achieving resolution upsampling. Specifically, for a size of... The input feature map can be transformed into a size of subpixel convolution. The output feature map, where The decoder consists of four upsampling stages. Each stage performs depthwise separable convolutions on the input feature map to extract high-resolution features, and then doubles the spatial resolution through subpixel convolutions. The first upsampling stage takes the output of the fifth block of the encoder as input, maintains 256 channels after depthwise separable convolution, and then passes through subpixel convolutions (the upsampling factor). The first upsampling stage outputs a feature map with 64 channels and doubled spatial size. The second, third, and fourth stages process the output of the previous stage sequentially, gradually restoring the original input resolution. Between every two upsampling stages, the decoder receives shallow features from the corresponding layer output of the encoder via skip connections. These shallow features are then concatenated and fused with the current decoder features along the channel dimension before being fed into the next upsampling stage. Skip connections compensate for the loss of spatial detail in deep features, enabling the network to utilize both high-level semantic information and low-level spatial location information simultaneously. The final output of the decoder is processed by depthwise separable convolution, and then a pointwise convolution is used to adjust the number of channels to be equal to the number of candidate denoising parameters, with each channel corresponding to one candidate denoising parameter. The output of this layer is the multi-channel marginal gain prediction map, with the same spatial size as the original input size of the current frame. The value of each pixel position in each channel represents the expected marginal gain of the motion estimation confidence at the same position in the next frame after using the corresponding candidate parameters.

[0121] Spatial convolution refers to using a convolutional kernel to slide across the spatial dimensions (height H, width W) of a feature map, performing a weighted summation at each position to obtain the output feature. Batch normalization is a technique used in deep neural networks to standardize the features of each layer within a batch, adjusting the numerical distribution of each feature channel to zero mean and unit variance. ReLU activation function, short for Rectified Linear Unit, is a non-linear activation function defined as follows: This is used to map negative inputs to 0, while keeping positive inputs unchanged. Depthwise separable convolution is a convolution operation that decomposes standard convolution into two independent steps, aiming to significantly reduce the number of parameters and computational cost: The first step is depthwise convolution, which performs spatial convolution on each channel of the input feature map individually, using an independent two-dimensional convolution kernel for each channel. The number of output channels is the same as the number of input channels, and it only extracts spatial features within each channel without inter-channel fusion. The second step is pointwise convolution, which uses a 1×1 convolution kernel to perform linear combination of the feature maps output by the depthwise convolution, achieving cross-channel information fusion. Skip connections refer to the operation of directly passing the feature maps from the shallow layers (encoder part) of the network to the corresponding layers in the deep layers (decoder part) and fusing them with the feature maps after several layers of transformation. Channel-dimensional concatenation fusion refers to the operation of merging two or more feature maps along the channel axis. Assuming the size of feature map A is... The size of feature map B is The size of the output feature map after splicing is .

[0122] The training process for this lightweight deep prediction network is as follows:

[0123] The network's training data can be generated using a combination of simulation and real-world data acquisition. This involves selecting video clips covering diverse scenes from publicly available video datasets, including indoor static scenes, outdoor dynamic scenes, and low-light scenes. For each original high-definition video clip, frame-by-frame annotation is performed in chronological order: professionals identify the regions with the highest and lowest motion estimation confidence in each frame, creating a ground truth map. Synthetic noise of varying intensities, including Gaussian noise, Poisson noise, and mixed noise, is superimposed on each video frame to generate denoising training samples at different noise levels. Based on this, for each "previous frame - next frame" pair, each parameter in the candidate denoising parameter set is enumerated. Apply a temporal fusion operation of appropriate intensity to the content at the beginning of the frame, then re-perform motion estimation, calculate the confidence change of each pixel position in the subsequent frame, and record it as the true value of the marginal benefit. The input to a training sample includes: a four-channel feature map, and candidate parameter identifiers as conditional information. The output is the corresponding Approximately 500,000 valid training sample pairs were collected and divided into training, validation, and test sets in an 8:1:1 ratio.

[0124] The training set is divided into several batches, with each batch containing 32 training samples. These samples are then input into the network for forward propagation calculations to obtain the predicted marginal revenue. Forward propagation refers to the process in a neural network where data is processed and results are passed layer by layer forward through the network, from the input layer to the hidden layer and then to the output layer. The total loss is then calculated using the backpropagation algorithm, which calculates the gradient of each layer's parameters with respect to the loss. This involves working backward from the output layer to the input layer, using the chain rule to find the partial derivatives (gradients) of each layer's weights with respect to the loss. The loss function consists of three terms: the first term is the regression loss for marginal revenue prediction, which can be measured using the mean squared error to determine the difference between the predicted and true values. Its definition is:

[0125] .

[0126] in The first term represents the total number of pixels involved in the calculation. The second term is the sign consistency loss for marginal gain prediction, used to guide the network to correctly distinguish between marginal gain and marginal loss, and is defined as:

[0127] .

[0128] in This is the Sigmoid function. The third term is the spatial smoothness regularization loss, which encourages the marginal reward predictions of adjacent pixels to remain spatially smooth; it is defined as:

[0129] .

[0130] in For pixels The four neighboring pixels of pixel p are the four adjacent pixels above, below, left, and right. These are the guiding weights set based on the gradient of the current frame image. The total loss function is defined as:

[0131] .

[0132] in and The pre-defined balancing coefficients are used. The Adam optimizer updates the network weights and biases based on gradient information, adjusting the values ​​of each convolutional kernel to minimize the loss function; after completing one batch of updates, it reads the next batch and continues training. The entire training process lasts for 50 rounds, that is, iterates through the entire training set 50 times. After each round, the network's predictive performance is evaluated using the validation set, monitoring the trend of the absolute error of the marginal return prediction. If the validation set loss no longer decreases after 5 consecutive rounds, training is terminated and the current model's parameter state is saved as the final prediction network configuration.

[0133] The initial parameter configuration for this lightweight deep prediction network is as follows:

[0134] The initial parameters of the network can be configured using the He uniform initialization method. This He uniform initialization method adaptively adjusts the initial variance of the weights based on the number of input channels of the current convolutional layer, effectively mitigating the vanishing or exploding gradient problem in deep networks, and is suitable for network structures using the ReLU activation function. Specifically, for each deep convolutional layer and pointwise convolutional layer, its weight parameters range from the interval... The samples were randomly obtained from a uniform distribution on the surface, where This represents the number of input channels for this layer. The bias parameters of all convolutional layers are initialized to zero. The scaling parameter in the batch normalization layer is initialized to 1, and the translation parameter is initialized to 0. At this point, the offline training phase of the deep prediction network is complete, and the network parameters after training convergence are stored.

[0135] For example, the local texture complexity map, local motion intensity map, initial noise feature map, motion estimation confidence map, and candidate denoising parameter set are simultaneously input into a lightweight deep prediction network to output a multi-channel marginal benefit prediction map.

[0136] In one embodiment, a multi-step optimization function is constructed with the objective of the marginal gain of the noise reduction quality of the current frame video image and the confidence of the motion estimation of the future frame video image as the objectives. Based on the multi-channel marginal gain prediction graph, the multi-step optimization function is solved to obtain the temporal fusion weight control graph, including:

[0137] S501, construct a multi-step optimization objective function with the goal of improving the noise reduction quality of the current frame video image and increasing the marginal gain of the motion estimation confidence of the future frame video image; the calculation formula for the multi-step optimization objective function is as follows:

[0138] .

[0139] in, To optimize the objective function in multiple steps, This is a local noise reduction quality evaluation function. pixel position Local texture complexity at the location, pixel position The intensity of localized movement at the location The parameters corresponding to the multi-channel marginal revenue prediction graph The projected marginal revenue value, For the current parameter Local variance within the spatial neighborhood , and These are the preset weighting coefficients.

[0140] For example, for each pixel location p, the candidate noise reduction parameters to be evaluated are... Construct a multi-step optimization objective function with variables as independent variables. The function consists of three parts: the first part is the local noise reduction quality evaluation function. Its input includes the local texture complexity of the current pixel. and local intensity of movement The specific implementation of this function is as follows: when a parameter is used... After noise reduction processing of the current frame, at the pixel Calculate a local approximation of the peak signal-to-noise ratio within the neighborhood of , and based on and Weighted corrections are applied to the calculation results—in texture-rich regions ( Larger values ​​increase the penalty for excessive smoothing, especially in flat regions. Smaller values ​​increase the penalty for noise residue, especially in areas of intense activity. (For larger values) appropriately reduce the weights of temporal fusion to avoid motion blur. The second term is the marginal gain of the confidence score for motion estimation in future frames. This value is read directly from the multi-channel marginal return prediction graph, i.e., based on the current candidate denoising parameters. Select the corresponding channel and obtain the pixel position of that channel. The value at this location indicates the use of Then, the expected change in confidence level is estimated for motion at the same position in the next frame. The third term is the spatial smoothness constraint term. Defined as the current parameter The variance within a local neighborhood centered at p is used to suppress abrupt changes in noise reduction weights between adjacent pixels. The three terms are multiplied by pre-defined weighting coefficients. , Summing λ and λ together yields the complete objective function expression:

[0141] .

[0142] in and The parameter λ is used to adjust the relative importance of local benefits and cost changes; λ controls the strength of variance constraints. A larger function value indicates the use of parameters. The higher the overall return, the better.

[0143] S502, for each pixel location, based on the multi-channel marginal benefit prediction map, the multi-step optimization objective function is discretely solved to obtain the optimal candidate noise reduction parameters for each pixel location.

[0144] For example, for each pixel position p, each candidate parameter in the candidate denoising parameter set is sequentially... Substitute into the objective function In the calculation process, the values ​​of the three objective functions are calculated respectively. The item corresponds to the multi-channel marginal revenue prediction graph. To obtain through the channel; The term is calculated by examining all pixels within a neighborhood window of a preset size surrounding the current pixel p. The sample variance is calculated from the value (i.e., the same candidate parameter currently evaluated by each pixel in the neighborhood); The term is based on the texture complexity of point p. and intensity of exercise and parameters The values ​​are obtained by looking up tables or calculating using analytical formulas. After calculating the objective function values ​​corresponding to the three candidate parameters, they are compared. , and The size of the parameter is used to select the candidate parameter that maximizes the objective function value as the optimal candidate noise reduction parameter for that pixel location, denoted as . Repeat the above operation for all pixel positions to obtain the optimal candidate noise reduction parameters for each pixel. The preset-size neighborhood window refers to a rectangular local region centered on the current pixel position p, extending a certain distance in each of the four directions (up, down, left, and right). This window can be pre-configured based on the video image resolution, noise level, and desired spatial smoothness.

[0145] S503 determines the optimal candidate denoising parameters as the optimal temporal fusion weights, and combines all the optimal temporal fusion weights to obtain the initial fusion weight map.

[0146] For example, the position of each pixel Optimal candidate noise reduction parameters The optimal temporal fusion weights assigned to this pixel ,Right now To ensure consistency between the decision results and the physical meaning of the noise reduction parameters: regions with low motion estimation confidence are assigned larger α values ​​to reduce the temporal fusion intensity and avoid introducing artifacts; regions with high motion estimation confidence are assigned smaller α values ​​to fully utilize temporal multi-frame averaging to suppress noise. The optimal temporal fusion weights for all pixel positions are organized into a two-dimensional matrix according to spatial coordinates, i.e., the initial fusion weight map. This map has the same size as the current frame image, and the value of each pixel is within... Within the interval, but since the candidate parameters only have three discrete values, the initial fusion weight graph contains only three different values.

[0147] S504 uses the luminance channel of the current frame video image as a guide map to perform guided filtering and edge-preserving smoothing on the initial fusion weight map, generating a temporal fusion weight control map.

[0148] For example, the luminance channel of the current frame video image is obtained. Since the initial fusion weight map contains only three discrete values, direct use will cause a step change in weights between adjacent pixels, resulting in unnatural spatial boundaries in the noise reduction effect. To eliminate the visual discontinuity caused by discrete jumps while maintaining clear boundaries in the motion estimation confidence region, guided filtering can be used to smooth the initial fusion weight map based on the luminance channel: For each pixel position p, a local window is set centered at p, assuming a linear relationship between the output and the guided map within the window. ,in Let q be the linear coefficients within the window, assumed to be constant, and q be the number of pixels within the window. This represents the luminance component of the current frame's video image at pixel location p. By minimizing the output... and input The difference between them, i.e., the solution The optimal linear coefficients within the window can be obtained, where This is a regularization parameter that can be preset based on the image noise level and the desired smoothness. The optimal temporal fusion weight at pixel position p in the initial fusion weight map. This represents a local window centered at pixel position p. The linear coefficients of all overlapping windows are averaged to calculate the output value for each pixel position. As input, After performing the above operations as a guide map, a new weight map is output. This map is used to provide smooth and adaptive pixel-wise weights for the weighted fusion in step S106.

[0149] Guided filtering is an edge-preserving image filtering technique. Its core idea is to filter the input image (i.e., the initial fusion weight map) under the guidance of a guide map (i.e., the current frame's luminance channel), ensuring the output image maintains a clear outline at the edges of the guide map and a smooth transition within flat areas. The luminance channel refers to the luminance component obtained after converting the current frame's color image from the RGB color space to the luminance-chrominance color space. This component characterizes the brightness of each pixel in the image but does not contain color information. The specific conversion method depends on the video standard used; for example, for high-definition video with standard dynamic range, a specific conversion method can be used. The linear weighted combination is calculated from the RGB values.

[0150] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0151] Based on the same inventive concept, this application also provides an adaptive noise reduction system for video stream data to implement the adaptive noise reduction method for video stream data described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the adaptive noise reduction system for video stream data provided below can be found in the limitations of the adaptive noise reduction method for video stream data described above, and will not be repeated here.

[0152] In one exemplary embodiment, such as Figure 3 As shown, an adaptive noise reduction processing system 300 for video stream data is provided, comprising:

[0153] The noise assessment module 301 is used to acquire the current frame video image and perform cross-scale multi-resolution noise feature pre-assessment on the current frame video image to obtain an initial noise feature map.

[0154] The vector field generation module 302 is used to perform multi-reference frame joint motion estimation on the current frame video image based on the historical frame video images in the historical state cache library, to obtain the motion vector field of the current frame video image relative to the historical frame video images, and to calculate the temporal consistency deviation of each pixel position based on each motion vector field.

[0155] The confidence calculation module 303 calculates the motion estimation confidence and reference frame pixel value for each pixel location based on the temporal consistency deviation and the initial noise feature map, and obtains the motion estimation confidence map and the reference frame pixel map.

[0156] The marginal revenue prediction module 304 is used to input the initial noise feature map, motion estimation confidence map and candidate denoising parameter set into the lightweight deep prediction network, and output a multi-channel marginal revenue prediction map; each channel of the multi-channel marginal revenue prediction map corresponds to a candidate denoising parameter;

[0157] The function solving module 305 is used to construct a multi-step optimization function with the goal of the noise reduction quality of the current frame video image and the marginal gain of the motion estimation confidence of the future frame video image, and solve the multi-step optimization function based on the multi-channel marginal gain prediction map to obtain the temporal fusion weight control map.

[0158] The image denoising module 306 is used to weightedly fuse the current frame video image and the reference frame pixel image according to the temporal fusion weight control map to obtain the current frame denoised video image.

[0159] Furthermore, the noise assessment module 301 is also used for:

[0160] Based on the current frame video image, construct an image pyramid including the original resolution layer, the 1 / 2 resolution layer, and the 1 / 4 resolution layer;

[0161] At each scale level of the image pyramid, the local evaluation window is slid in a step equal to half the window size, and the gray variance of each local evaluation window is calculated based on the pixel gray values ​​within the local evaluation window.

[0162] By combining the gray-level variances belonging to the same scale layer, we obtain the variance set map of each scale layer;

[0163] Based on the window gray-level distribution histogram of the corresponding pixel positions between each pair of adjacent scale layers, the cross-entropy difference of each pixel position between each pair of adjacent scale layers is calculated.

[0164] The cross-entropy difference at each pixel location is weighted and fused to obtain a comprehensive value of the cross-scale difference at each pixel location;

[0165] Based on the variance ensemble plot and the cross-scale difference comprehensive value, a weighted summation operation is performed on each pixel location to generate an initial noise feature map.

[0166] Furthermore, the confidence calculation module 303 is also used for:

[0167] Based on the initial noise feature map and the standard deviation of the preset noise model in the neighborhood of the current pixel, combined with the temporal consistency deviation, the motion estimation confidence of each pixel position is calculated, and all motion estimation confidences are combined to obtain the motion estimation confidence map.

[0168] Based on the motion vector field, the earliest frame video image in each historical frame video image is reverse twisted and mapped to obtain the reference frame pixel value at each pixel position. All reference frame pixel values ​​are combined to obtain the reference frame pixel map.

[0169] Furthermore, the marginal revenue forecasting module 304 is also used for:

[0170] According to the partitioning rules, the confidence level in the motion estimation confidence map is divided into high confidence intervals, medium confidence intervals and low confidence intervals, and candidate denoising parameters are assigned to the high confidence intervals, medium confidence intervals and low confidence intervals respectively, thus obtaining a set of candidate denoising parameters;

[0171] Obtain the local texture complexity map and local motion intensity map of the current frame video image;

[0172] The local texture complexity map, local motion intensity map, initial noise feature map, motion estimation confidence map and candidate denoising parameter set are simultaneously input into a lightweight deep prediction network to output a multi-channel marginal benefit prediction map.

[0173] By combining all the predicted marginal returns corresponding to each candidate denoising parameter, a multi-channel marginal return prediction map is obtained.

[0174] Furthermore, the function solver module 305 is also used for:

[0175] A multi-step optimization objective function is constructed, with the goal of improving the denoising quality of the current frame video image and increasing the marginal gain of the motion estimation confidence of the future frame video image. The formula for calculating the multi-step optimization objective function is as follows:

[0176] ;

[0177] in, To optimize the objective function in multiple steps, This is a local noise reduction quality evaluation function. pixel position Local texture complexity at the location, pixel position The intensity of localized movement at the location The parameters corresponding to the multi-channel marginal revenue prediction graph The projected marginal revenue value, For the current parameter Local variance within the spatial neighborhood , and These are the preset weighting coefficients;

[0178] For each pixel location, the multi-step optimization objective function is discretely solved based on the multi-channel marginal benefit prediction map to obtain the optimal candidate noise reduction parameters for each pixel location;

[0179] The optimal candidate denoising parameters are determined as the optimal temporal fusion weights, and all optimal temporal fusion weights are combined to obtain the initial fusion weight map;

[0180] Using the luminance channel of the current frame video image as the guide map, the initial fusion weight map is subjected to guided filtering and edge-preserving smoothing to generate a temporal fusion weight control map.

[0181] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the adaptive noise reduction processing method for video stream data as described above.

[0182] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0183] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0184] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. An adaptive noise reduction method for video stream data, characterized in that, The method includes: Acquire the current frame video image and perform cross-scale multi-resolution noise feature pre-evaluation on the current frame video image to obtain an initial noise feature map; Based on historical frame video images in the historical state cache library, multi-reference frame joint motion estimation is performed on the current frame video image to obtain the motion vector field of the current frame video image relative to the historical frame video images, and the temporal consistency deviation of each pixel position is calculated based on each motion vector field. Based on the temporal consistency deviation and the initial noise feature map, the motion estimation confidence and reference frame pixel value for each pixel location are calculated to obtain the motion estimation confidence map and the reference frame pixel map. The initial noise feature map, the motion estimation confidence map, and the candidate denoising parameter set are input into a lightweight deep prediction network, which outputs a multi-channel marginal benefit prediction map; each channel of the multi-channel marginal benefit prediction map corresponds to a candidate denoising parameter. A multi-step optimization function is constructed with the denoising quality of the current frame video image and the marginal gain of the motion estimation confidence of the future frame video image as the objective. Based on the multi-channel marginal gain prediction map, the multi-step optimization function is solved to obtain the temporal fusion weight control map. Based on the temporal fusion weight control graph, the current frame video image and the reference frame pixel map are weighted and fused to obtain the current frame denoised video image.

2. The method according to claim 1, characterized in that, The step of performing cross-scale, multi-resolution noise feature pre-evaluation on the current frame video image to obtain an initial noise feature map includes: Based on the current frame video image, an image pyramid is constructed, including an original resolution layer, a 1 / 2 resolution layer, and a 1 / 4 resolution layer; At each scale level of the image pyramid, a local evaluation window is slid in a step equal to half the window size, and the grayscale variance of each local evaluation window is calculated based on the pixel grayscale values ​​within the local evaluation window. The gray-level variances belonging to the same scale layer are combined to obtain a variance set map of each scale layer. The cross-entropy difference is calculated based on the window gray-level distribution histogram of the corresponding pixel positions between each pair of adjacent scale layers. Based on the pixel location, the cross-entropy difference between each pair of adjacent scale layers is weighted and fused to obtain a comprehensive cross-scale difference value for each pixel location; Based on the variance ensemble map and the cross-scale difference comprehensive value, a weighted summation operation is performed on each pixel location to generate the initial noise feature map.

3. The method according to claim 1, characterized in that, The step of calculating the motion estimation confidence score and reference frame pixel value for each pixel location based on the temporal consistency deviation and the initial noise feature map, to obtain the motion estimation confidence map and the reference frame pixel map, includes: Based on the initial noise feature map and the standard deviation of the preset noise model in the neighborhood of the current pixel, combined with the temporal consistency deviation, the motion estimation confidence of each pixel position is calculated, and all the motion estimation confidences are combined to obtain the motion estimation confidence map. Based on the motion vector field, the earliest frame video image in each of the historical frame video images is subjected to reverse distortion mapping to obtain the reference frame pixel value at each pixel position, and all the reference frame pixel values ​​are combined to obtain the reference frame pixel map.

4. The method according to claim 1, characterized in that, The process of inputting the initial noise feature map, the motion estimation confidence map, and the candidate denoising parameter set into a lightweight deep prediction network, and outputting a multi-channel marginal return prediction map, includes: According to the partitioning rules, the confidence level in the motion estimation confidence map is divided into high confidence intervals, medium confidence intervals and low confidence intervals, and candidate denoising parameters are assigned to the high confidence intervals, the medium confidence intervals and the low confidence intervals respectively to obtain a set of candidate denoising parameters; Obtain the local texture complexity map and local motion intensity map of the current frame video image; The local texture complexity map, the local motion intensity map, the initial noise feature map, the motion estimation confidence map, and the candidate denoising parameter set are simultaneously input into a lightweight deep prediction network to output a multi-channel marginal benefit prediction map.

5. The method according to claim 1, characterized in that, The construction of a multi-step optimization function with the objective of the marginal gain of noise reduction quality of the current frame video image and the confidence of motion estimation of the future frame video image, and the solution of the multi-step optimization function based on the multi-channel marginal gain prediction map to obtain the temporal fusion weight control map, includes: A multi-step optimization objective function is constructed, with the goal of improving the noise reduction quality of the current frame video image and increasing the marginal gain of the motion estimation confidence of the future frame video image. The calculation formula for the multi-step optimization objective function is as follows: ； in, For the multi-step optimization objective function, This is a local noise reduction quality evaluation function. pixel position Local texture complexity at the location, pixel position The intensity of localized movement at the location For the parameters corresponding to the multi-channel marginal revenue prediction graph The projected marginal revenue value, For the current parameter Local variance within the spatial neighborhood , and These are the preset weighting coefficients; For each pixel location, the multi-step optimization objective function is discretely solved based on the multi-channel marginal benefit prediction map to obtain the optimal candidate noise reduction parameters for each pixel location; The optimal candidate denoising parameters are determined as the optimal temporal fusion weights, and all the optimal temporal fusion weights are combined to obtain the initial fusion weight map; Using the luminance channel of the current frame video image as a guide map, the initial fusion weight map is subjected to guided filtering and edge-preserving smoothing processing to generate the temporal fusion weight control map.

6. An adaptive noise reduction processing system for video stream data, characterized in that, The system includes: The noise assessment module is used to acquire the current frame video image and perform cross-scale multi-resolution noise feature pre-assessment on the current frame video image to obtain an initial noise feature map; The vector field generation module is used to perform multi-reference frame joint motion estimation on the current frame video image based on historical frame video images in the historical state cache library, to obtain the motion vector field of the current frame video image relative to the historical frame video images, and to calculate the temporal consistency deviation of each pixel position based on each motion vector field. The confidence calculation module calculates the motion estimation confidence and reference frame pixel value for each pixel location based on the temporal consistency deviation and the initial noise feature map, thereby obtaining a motion estimation confidence map and a reference frame pixel map. The marginal revenue prediction module is used to input the initial noise feature map, the motion estimation confidence map, and the candidate denoising parameter set into a lightweight deep prediction network, and output a multi-channel marginal revenue prediction map; each channel of the multi-channel marginal revenue prediction map corresponds to a candidate denoising parameter; The function solving module is used to construct a multi-step optimization function with the denoising quality of the current frame video image and the marginal gain of the motion estimation confidence of the future frame video image as the objectives, and solve the multi-step optimization function based on the multi-channel marginal gain prediction map to obtain the temporal fusion weight control map. The image denoising module is used to weightedly fuse the current frame video image and the reference frame pixel image according to the temporal fusion weight control map to obtain the current frame denoised video image.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images