Video noise reduction method, system and device and storage medium

Abstract

The invention provides a video noise reduction method, system and device and a storage medium, and belongs to the field of video noise reduction, and the method comprises the steps: collecting video frame data, searching pixels in target frame data and pixel values of a receptive field where the pixels are located in a pre-constructed multi-dimensional lookup table, and obtaining a first noise reduction result; performing noise reduction processing on the target frame data and the associated frame data by adopting a first neural network model to generate a second noise reduction result containing time sequence consistency and a dynamic weight vector; and based on the dynamic weight vector, carrying out weighted fusion on the first noise reduction result and the second noise reduction result to generate a final video noise reduction result. According to the invention, the noise reduction result containing time sequence consistency is generated on the basis of the first neural network model by combining the target frame data and the associated frame data thereof; the common problems of flicker and noise jumping in a traditional noise reduction method are avoided, so that the image change between continuous video frames is smoother and more natural.

Description

Background Art

[0002] In the field of video processing, high-quality videos often have good temporal consistency, which can ensure smooth and natural image changes between consecutive frames. However, low-quality videos often have noise jumps and flickers, which affect the audience experience. Especially in low-light or high-noise environments, the quality of video images is greatly reduced, and noise interference is particularly serious. Traditional video denoising methods mainly process single-frame images, but when single-frame denoising algorithms are directly applied to video streams, the problem of timing inconsistency may be aggravated, resulting in obvious jumps or flickers between consecutive frames.

[0003] Image or video denoising is essentially a delicate balance between detail restoration and smooth denoising. Both noise and detail are high-frequency information. Although noise is removed during denoising, some texture details may be removed, resulting in image blur or detail loss. At this point, how to balance denoising and detail preservation becomes a difficult challenge to unify. Especially in video denoising, not only the denoising effect of a single frame must be considered, but also the temporal consistency between video frames must be maintained to avoid the impact of denoising on the overall video stream.

[0004] Most existing video noise reduction methods focus on single-frame image noise reduction, while ignoring the temporal relationship between frames, resulting in unnatural frame skipping in dynamic scenes. In addition, the characteristics of video noise are dynamically changing, and the setting of noise reduction intensity usually depends on the type and intensity of noise. Intensity adjustment during the noise reduction process often needs to rely on user experience and lacks an automated adjustment mechanism. Therefore, how to dynamically adjust the noise reduction intensity and effectively utilize temporal consistency while ensuring video quality is an urgent problem to be solved in the field of video noise reduction. Summary of the invention

[0005] In view of the problems in the prior art, the present invention aims to provide a video noise reduction method, system, device and storage medium, aiming to solve the problems of poor temporal consistency, difficulty in automatically adjusting the noise reduction intensity and high computational complexity in the video noise reduction processing in the prior art. By combining a multidimensional lookup table and a neural network model, the noise reduction intensity is dynamically adjusted, which can effectively remove noise and maintain the temporal consistency and details of the video.

[0006] The embodiment of the present invention provides a video noise reduction method, which includes the following steps:

[0007] Collecting video frame data, searching pixels in the target frame data and pixel values ​​of the receptive field in which they are located through a pre-built multi-dimensional lookup table to obtain a first denoising result;

[0008] Using the first neural network model to perform noise reduction processing on the target frame data and its associated frame data, generating a second noise reduction result and a dynamic weight vector including time consistency;

[0009] Based on the dynamic weight vector, the first denoising result and the second denoising result are weightedly fused to generate a final video denoising result.

[0010] In some embodiments, the multidimensional lookup table is pre-constructed based on the second neural network model, and the construction process includes the following steps:

[0011] By uniformly sampling the 2D noise reduction parameters and 3D noise reduction parameters of the video frame data, N 2D sampling points and M 3D sampling points are generated at a predetermined sampling interval, and N×M noise degradation conditions are obtained;

[0012] Based on N×M noise degradation situations, corresponding N×M training subsets are generated respectively, and N×M independent second neural network models are trained respectively to obtain second neural network models corresponding to the N×M noise degradation situations one by one;

[0013] For each second neural network model, all possible combinations of receptive field pixel values ​​are generated as input samples according to the size of its receptive field, and the corresponding denoising output results are obtained by performing inference through the second neural network model;

[0014] Each second neural network model input sample and its corresponding denoising output result are stored in a lookup table to generate N×M multi-dimensional lookup tables.

[0015] In some embodiments, in the inference stage, based on the pixels in the target frame data and the pixel values ​​of their receptive fields, searches are performed in N×M lookup tables respectively to obtain N×M search results as the first denoising results.

[0016] In some embodiments, the first neural network model is an encoder-decoder network structure, the encoder receives the target frame data and its associated frame data as input, the decoder outputs a second denoising result including temporal consistency, and the decoder unit further includes a parallel branch, the parallel branch is used to generate a dynamic weight vector for denoising adjustment, and the parallel branch includes:

[0017] At least one pooling layer, used to compress the feature map output by the decoder into a low-dimensional vector;

[0018] At least one fully connected layer is used to generate a dynamic weight vector of size N×M+1 according to the low-dimensional vector output by the pooling layer.

[0019] In some embodiments, the associated frame data includes at least one adjacent frame before and after the target frame, and the first neural network model receives multiple frames of data as input through a time-series stacking method to generate a second noise reduction result.

[0020] In some embodiments, the method of weighted fusion of the first denoising result and the second denoising result based on the dynamic weight vector includes:

[0021] A first weight value based on the dynamic weight vector is used to adjust the timing consistency of the first noise reduction result and the second noise reduction result;

[0022] The second weight value based on the dynamic weight vector is used to adjust the global noise reduction degree of the fusion ratio of the outputs of different lookup tables in the multidimensional lookup table.

[0023] In some embodiments, the noise reduction method is implemented in combination with hardware resources, specifically including:

[0024] A graphics processing unit for processing a multi-dimensional lookup table lookup process;

[0025] A neural network processing unit, used for processing a noise reduction process of the first neural network model;

[0026] The GPU and the NNPU perform their respective processing tasks in parallel.

[0027] The embodiment of the present invention further provides a video noise reduction system, which is applied to the above-mentioned video noise reduction method, and the system includes:

[0028] A video acquisition module, used to acquire video frame data, target frame data and associated frame data;

[0029] A single-frame denoising module is used to obtain a first denoising result by searching a pixel in the target frame data and a pixel value of a receptive field in which the pixel is located in a pre-built multi-dimensional lookup table;

[0030] A consistency module, used for receiving target frame data and its associated frame data, and outputting a second noise reduction result and a dynamic weight vector including time sequence consistency;

[0031] The denoising fusion module is used to perform weighted fusion on the first denoising result and the second denoising result based on the dynamic weight vector to generate a final denoising result.

[0032] The present invention generates a denoising result including temporal consistency on the basis of a first neural network model by combining target frame data and its associated frame data. Due to the maintenance of temporal consistency, the flicker and noise jitter problems common in traditional denoising methods are avoided, making the image changes between continuous video frames smoother and more natural. The present invention performs weighted fusion of different denoising results through dynamically generated weight vectors, and can automatically adjust the denoising intensity according to the pixel values ​​of the pixels in the target frame data and the receptive fields in which they are located and user needs. In particular, under different noise levels, the system can flexibly adjust the denoising effect to avoid losing details due to excessive denoising or failing to effectively remove noise due to insufficient denoising, thereby achieving a good balance between denoising effect and detail retention. A fast search mechanism based on a multidimensional lookup table is adopted, combined with the denoising processing of the neural network model, to achieve real-time video denoising. The pre-construction and inference process of the lookup table enables the system to quickly find the pixel values ​​of the pixels in the target frame data and the receptive fields in which they are located and perform denoising processing, which greatly reduces the computational complexity and is particularly suitable for real-time video processing scenarios that require fast response.

[0033] The embodiment of the present invention further provides a video noise reduction device, including:

[0034] processor;

[0035] a memory in which executable instructions for the processor are stored;

[0036] The processor is configured to execute the steps of the above-mentioned video noise reduction method by executing executable instructions.

[0037] By adopting the video noise reduction device provided by the present invention, the processor executes the above-mentioned video noise reduction method when executing executable instructions, thereby obtaining the beneficial effects of the above-mentioned video noise reduction method.

[0038] The embodiment of the present invention further provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the above-mentioned video noise reduction method are implemented.

[0039] By adopting the computer-readable storage medium provided by the present invention, the program stored therein implements the steps of the video noise reduction method when being executed, thereby obtaining the beneficial effects of the above-mentioned video noise reduction method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0041] Figure 1 is a flow chart of a video noise reduction method according to an embodiment of the present invention;

[0042] Figure 2is a flow chart of a process of constructing a pre-constructed multi-dimensional lookup table according to an embodiment of the present invention;

[0043] Figure 3 This is a flow chart of weighted fusion of a first denoising result and a second denoising result based on a dynamic weight vector according to an embodiment of the present invention;

[0044] Figure 4 is a structural schematic diagram of a video noise reduction system according to an embodiment of the present invention;

[0045] Figure 5 is a structural schematic diagram of a video noise reduction device according to an embodiment of the present invention;

[0046] Figure 6 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted.

[0048] The video denoising method of the present invention aims to solve the balance problem between noise removal and temporal consistency maintenance in the video. How to retain the texture and details of the image while removing the noise is the key problem of denoising. The video denoising scheme of the present invention effectively removes the noise while maintaining the details and texture information of the image through single-frame denoising, temporal consistency maintenance and fusion of different denoising results. In video denoising, noise usually appears as random and irregular interference signals, which affect the quality of the image and may cause the loss of image details. The goal of video denoising is to remove these noise signals while maintaining the detailed information of the image as much as possible. The embodiment of the present invention uses a pre-built lookup table to perform single-frame denoising on the pixels in the target frame data and the pixel values ​​of the receptive field in which they are located, quickly identify the noise area in the target frame and perform single-frame denoising. The role of the lookup table is to provide a fast reference framework so that during the processing, the noise estimation and removal can be completed with minimal computational overhead. The lookup table method saves the mapping results obtained by the pixel receptive field area through the pre-built denoising model in a table. During reasoning, each pixel can quickly look up the table according to the index of its receptive field area in the table to obtain the reconstruction result, that is, the single-frame denoising result. However, the denoising method using the lookup table alone cannot fully solve the problem of temporal consistency. In a video, there is often a strong temporal correlation between multiple adjacent frames. Simply performing denoising on each frame separately may lead to inconsistency between frames, and incoherence or flickering. Therefore, the embodiment of the present invention applies a time-based neural network model to process the data of the target frame and its associated frames. By jointly modeling the time sequence information of these adjacent frames, the temporal consistency of the video can be better retained, avoiding the introduction of unnatural time sequence jumps in the denoising process. In the process of video denoising, different denoising methods produce results with different characteristics. How to balance the contribution of these different results is also one of the technical problems to be solved by the embodiment of the present invention. The embodiment of the present invention fuses different denoising results by designing a dynamic weighting mechanism. This weighted fusion mechanism adjusts the weight ratio of the first noise reduction result and the second noise reduction result according to the dynamic weight vector, so as to achieve the best noise reduction effect and timing consistency. Therefore, the present invention can effectively solve the contradiction between noise removal and timing consistency in the video noise reduction process. By combining the lookup table fast noise reduction with timing neural network modeling, it is possible to maintain the natural transition between video frames while removing noise, avoiding flickering and incoherence. The dynamic weighted fusion method further improves the flexibility of the noise reduction results, so that the noise reduction effect not only adapts to different types of noise, but also can adjust the noise reduction intensity as needed, thereby improving video quality and user experience.

[0049] like Figure 1As shown, an embodiment of the present invention provides a video noise reduction method, which solves the balance problem between noise removal and timing consistency in the video through the following steps, ensuring the noise reduction effect while maintaining the timing consistency of the video.

[0050] S100, collecting video frame data, searching pixels in the target frame data and pixel values ​​of the receptive field in which they are located in a pre-built multidimensional lookup table to obtain a first denoising result;

[0051] In an embodiment of the present invention, the video frame data is an image sequence acquired by a video acquisition device. Each frame of the image contains information at a certain moment of the video. In an embodiment of the present invention, the acquired video frame data includes target frame data and its associated frame data, which may be several frames before and after the target frame. The pixel values ​​of the pixels in the target frame data and the receptive fields in which they are located are quickly searched through a pre-built multidimensional lookup table to obtain a single-frame denoising result. The lookup table is a tool that has been trained to include the mapping relationship between the pixel values ​​of the pixels and the receptive fields in which they are located and the corresponding denoising results. In the inference stage, the lookup table greatly reduces the amount of calculation and improves the denoising efficiency by directly retrieving the pre-calculated denoising results. Compared with traditional denoising methods, the latter usually rely on neural network hardware modules to perform a large number of multiplication and addition operations, resulting in high computational costs. When using a lookup table, there is no need to perform complex real-time calculations, only to find and return the denoising results from the memory, which makes the entire denoising process more efficient and faster. In addition, the use of the lookup table avoids the reliance on neural network hardware modules such as NPU, thereby reducing the system's demand for high-performance hardware and reducing hardware costs. These advantages enable this technical solution to show significant performance advantages in practical applications, especially in video noise reduction processing with high real-time requirements.

[0052] S200, using the first neural network model to perform noise reduction processing on the target frame data and its associated frame data, to generate a second noise reduction result and a dynamic weight vector including time consistency;

[0053] In an embodiment of the present invention, the first neural network model in the embodiment of the present invention adopts an encoder-decoder network structure for temporal consistency modeling, and the encoder receives the target frame data and its associated frame data as input to perform feature extraction. The decoder outputs the temporal denoising result. The introduction of associated frame data is to enhance the temporal consistency through the joint modeling of multi-frame information, and to maintain the denoising effect of consistent details between multiple frames. Through this structure, the first neural network model can effectively combine the timing information between multiple frames and reduce the timing inconsistency in the denoising process. This process generates a second denoising result and a dynamic weight vector at the same time. The dynamic weight vector is used to weightedly fuse the denoising results from different sources, so as to more flexibly control the fusion ratio of different denoising intensities.

[0054] S300: Based on the dynamic weight vector, weighted fusion is performed on the first denoising result and the second denoising result to generate a final video denoising result.

[0055] In an embodiment of the present invention, the first denoising result and the second denoising result are weightedly fused by a dynamic weight vector. During weighted fusion, the fusion ratio between the first denoising result (obtained by multidimensional lookup table search) and the second denoising result (obtained by first neural network processing) is adjusted by the dynamic weight. Through this weighted process, the denoising intensity can be automatically adjusted according to the pixel values ​​of the pixels in the video content and target frame data and the receptive field in which they are located, thereby achieving the effect of balancing noise removal and detail retention.

[0056] The embodiments of the present invention can effectively solve the balance problem between noise removal and timing consistency in video noise reduction. First, the application of the lookup table makes the process of single-frame noise reduction more efficient and less computationally intensive. Secondly, the first neural network can generate a more timing-consistent noise reduction result by considering the timing information between the target frame and the associated frame. Finally, the use of dynamic weight vectors makes the noise reduction result more flexible, and the noise reduction intensity can be automatically adjusted according to the needs of the actual video content. In summary, the embodiments of the present invention can achieve more efficient and accurate video noise reduction by combining the lookup table for fast noise reduction and the first neural network timing modeling, while ensuring the timing consistency of the video and reducing phenomena such as flickering and noise jumping.

[0057] like Figure 2 As shown, in one embodiment, a training method for constructing a hybrid LUT (lookup table) model is provided, and a multidimensional lookup table (LUT) is pre-constructed based on a second neural network model to cope with different degrees of noise degradation. For different noise degradation conditions defined by a combination of 2D and 3D noise reduction parameters, independent single-frame noise reduction networks are trained, and the noise reduction output results of these networks are stored in the lookup table. The following steps are included:

[0058] S110, uniformly sampling 2D noise reduction parameters and 3D noise reduction parameters of video frame data, generating N 2D sampling points and M 3D sampling points at a predetermined sampling interval, and obtaining N×M noise degradation conditions;

[0059] In this step, in order to cope with different types and degrees of noise degradation, 2D noise reduction parameters and 3D noise reduction parameters are used as input for sampling processing. These sampling points represent different noise degradation conditions, and can cover various common noise scenarios during noise reduction processing. This uniform sampling helps the system capture a wide range of noise patterns, thereby improving the adaptability of noise reduction. The 2D noise reduction parameters and 3D noise reduction parameters generate N sampling points and M sampling points respectively, and N×M different noise degradation conditions are obtained through this sampling method, covering degradation modes under different noise levels and spatial dimensions. The sampling method is not limited to uniform sampling, and non-uniform sampling can also be used, such as sampling according to the probability density function of the noise distribution, and performing denser sampling in areas with higher noise probability. The specific types and value ranges of 2D and 3D noise reduction parameters, as well as the sampling interval and sampling method, can be adjusted according to the actual application scenario and do not constitute a limitation of the present invention.

[0060] S120, based on N×M noise degradation situations, respectively generate corresponding N×M training subsets, respectively train N×M independent second neural network models, and obtain second neural network models corresponding to the N×M noise degradation situations one by one;

[0061] In an embodiment of the present invention, the second neural network model, i.e., the single-frame denoising network, calculates the denoising output within the receptive field area of ​​each training subset pixel point, thereby obtaining the optimized denoising result of the area under the specific noise degradation condition. Specifically, the training subset can be extracted from a video data set containing different noise levels, and the neural network calculates the denoising result of the point according to the receptive field of each pixel point and its surrounding pixels (i.e., the neighborhood area around the pixel point). The size and structure of the receptive field determine how the network extracts pixels from the neighborhood information and performs effective denoising. The denoising network can be trained by supervised learning or self-supervised learning methods, and different neural network architectures and training strategies can be adjusted according to the denoising effect. The second neural network model can be U-Net, DnCNN, etc. based on convolutional neural network (CNN), or other types of networks, such as Transformer, etc. The specific structure, training method and hyperparameters of the second neural network model can be adjusted according to the actual application scenario and do not constitute a limitation of the present invention.

[0062] S130, for each second neural network model, according to the size of its receptive field, generating all possible receptive field pixel value combinations as input samples, and performing reasoning through the second neural network model to obtain a corresponding denoising output result;

[0063] In an embodiment of the present invention, in this step, for each trained second neural network model (i.e., a single-frame denoising network), all possible combinations of receptive field pixel values ​​are generated as input samples according to the preset receptive field size, and the model is used for reasoning to obtain the corresponding denoising output result. This step is intended to prepare data for the subsequent construction of a lookup table. The size of the receptive field, the boundary processing method, and the arrangement of the input samples can be adjusted according to the actual application scenario, and the embodiment of the present invention does not make specific limitations.

[0064] S140, storing each second neural network model input sample and its corresponding denoising output result in a lookup table, and generating N×M multi-dimensional lookup tables.

[0065] In an embodiment of the present invention, the input samples of each second neural network model generated in the above steps and their corresponding denoising output results are stored in a lookup table, thereby constructing N×M multidimensional lookup tables, each of which corresponds to a noise degradation situation. The storage structure, indexing method, quantization method and compression algorithm of the lookup table can be adjusted according to the actual application scenario, and the embodiment of the present invention does not make specific limitations.

[0066] In some optional embodiments, in the inference stage, based on the pixel values ​​in the target frame data and the pixel values ​​of its receptive field, searches are performed in N×M lookup tables respectively to obtain N×M search results as the first denoising results.

[0067] In the inference stage of the present invention, a denoising method based on a multidimensional lookup table (LUT) is used for the pixels in the target frame data and the pixels in its receptive field. Specifically, each pixel in the target frame data will be searched in the pre-constructed N×M lookup tables together with the pixel value in its receptive field to obtain the denoising result. Each lookup table provides a corresponding denoising output for each pixel in a pre-constructed manner for different noise degradation conditions and denoising parameters, and these search results serve as the first denoising result. Specifically, the inference stage takes the pixels in the target frame data and the pixel values ​​of its receptive field as input, and searches for the corresponding denoising output in the pre-constructed multidimensional lookup table. This process can improve the inference speed, because the denoising process has obtained the best denoising output through training in the preprocessing stage, and in actual applications, it is only necessary to quickly find and apply these denoising results.

[0068] In some embodiments, the first neural network model is an encoder-decoder network structure, comprising:

[0069] The encoder unit is used to extract multi-scale features of the target frame data and the associated frame data. In an embodiment of the present invention, the encoder unit is used to extract multi-scale features from the input target frame data and the associated frame data. The encoder is usually composed of multiple convolutional layers, pooling layers, etc., and its function is to gradually reduce the spatial dimension of the input data while increasing its abstract feature representation. In this way, the encoder can effectively capture high-level features in the video frame data and generate a compressed representation. The decoder unit is used to gradually restore the spatial details of the target frame. In an embodiment of the present invention, the main function of the decoder unit is to gradually restore the spatial details of the target frame. The decoder restores the feature map to the same resolution as the original target frame by upsampling the compressed features extracted by the encoder. This process not only restores the spatial structure of the image, but also finely processes the image details to ensure that important detail information is not lost too much while reducing noise. The jump connection is used to transfer multi-scale features between the encoder unit and the decoder unit to enhance temporal consistency. The jump connection is a technology for connecting different layers of the network, and usually connects the output of certain layers of the encoder unit directly to the corresponding layers in the decoder unit. This structure can provide low-level features in the encoder layer in U-Net while the decoder partially restores spatial details, thereby avoiding excessive loss of spatial information during the decoding process. In video denoising tasks, skip connections are particularly helpful in maintaining temporal consistency, ensuring that temporal features between multiple frames are effectively fused and reducing noise fluctuations and flickering.

[0070] In the embodiment of the present invention, taking the U-Net structure as an example, the data features of the target frame and its associated frames can be extracted through the encoder unit, a multi-scale feature representation can be generated, and the spatial details of the target frame can be gradually restored through the decoder unit. The jump connection provides the decoder with low-level feature information from the encoder to help restore more details, especially to maintain important spatial structures during the denoising process. The interaction between the encoder and decoder parts of U-Net is that the multi-scale features extracted by the encoder part can effectively enhance the denoising model's perception of multi-level noise in the video frame, while the gradual recovery of the decoder ensures that the denoising process does not destroy the details in the frame. At the same time, the jump connection enhances the temporal consistency, making the denoising result of each frame smoother in time, thereby avoiding possible "flickering" or "jumping" phenomena.

[0071] The decoder unit further comprises a parallel branch for generating a dynamic weight vector for noise reduction adjustment, the parallel branch comprising:

[0072] At least one pooling layer is used to compress the feature map output by the decoder into a low-dimensional vector. The pooling operation reduces the spatial dimension of the feature map by sampling the maximum value or average value of the local area. This process can extract the key features of the feature map while reducing the amount of calculation and providing the necessary low-dimensional input for generating the dynamic weight vector.

[0073] At least one fully connected layer is used to generate a dynamic weight vector of size N×M+1 based on the low-dimensional vector output by the pooling layer. The function of the fully connected layer is to further process and generate a dynamic weight vector of size N×M+1 based on the low-dimensional vector output by the pooling layer. The fully connected layer connects the low-dimensional input of the pooling layer with multiple neurons in the network so that the pooled feature information can be effectively converted into weight parameters suitable for adjusting the denoising intensity. The dynamic weight vector is then used to weight the output of the decoder unit to further control the detail preservation and noise suppression in the denoising process.

[0074] The parallel-branched pooling layer in the embodiment of the present invention can effectively compress the feature map in the decoder unit into a low-dimensional representation, reducing the amount of calculation while retaining key features. Afterwards, these low-dimensional features are converted into dynamic weight vectors through the fully connected layer. These weight vectors not only play a key role in adjusting the denoising results, but also can adjust the denoising process according to different video content and noise levels, thereby improving the accuracy and flexibility of the denoising effect. The generation and adjustment process of the dynamic weight vector makes the decoder unit more than just a static image restoration module. By flexibly adjusting the weights, the denoising process can apply different denoising strategies at different times according to different noise conditions and image features to achieve the best balance between noise and details. The common problems of "over-smoothing" or "loss of details" in video denoising are solved, thereby improving the effect of video denoising processing.

[0075] In some embodiments, the associated frame data includes at least one adjacent frame before and after the target frame, and the first neural network model receives multiple frames of data as input through a time-series stacking method to generate a second denoising result. In an embodiment of the present invention, multiple frame data are input into the neural network model, which can effectively process the timing information between video frames and improve the time consistency problem of the video stream. In the neural network model of multi-frame denoising, the input multiple frame data can be composed of the target frame and at least one adjacent frame before and after it. For example, the target frame is set to H2, and one frame H1 and H3 are selected before and after to form three frames of data H1, H2 and H3. These three frames of data are input through a time-series stacking method. In this process, the channel information of multiple frame data will be spliced ​​according to the time dimension to form an input tensor with more information. In a specific implementation, the three frames of images H1, H2 and H3 will be spliced ​​along the channel dimension to form a six-channel tensor X = [H1, H2, H3] as the input of the network. Time-series stacking enables the network to not only consider the content of the current frame when processing each frame, but also integrate the relevant information of the previous and next frames. This approach helps the model take full advantage of the temporal dependencies in the video, that is, the dynamic relationship between adjacent frames. The motion information and scene changes between adjacent frames can be effectively learned through temporal stacking, thus avoiding the incoherence that may be caused by single-frame processing.

[0076] The multi-frame input and timing stacking in the embodiments of the present invention make it possible to effectively solve the consistency problem in video noise reduction. Through timing stacking, the network not only considers the content of the current frame, but also combines the information of the previous and next frames to better understand the timing characteristics in the video. The enhanced temporal consistency brought about by multi-frame input can avoid the flickering and noise beating phenomena in the video caused by the inconsistency of single-frame noise reduction, and significantly improve the effect and quality of video noise reduction. In this way, the present invention can effectively improve the smoothness and naturalness of the video stream and enhance the viewing experience of the video.

[0077] like Figure 3 As shown, the method for weighted fusion of the first denoising result and the second denoising result based on the dynamic weight vector includes:

[0078] S310, using a first weight value based on a dynamic weight vector to perform timing consistency adjustment on a first denoising result and a second denoising result;

[0079] S320: A second weight value based on the dynamic weight vector is used to adjust the global noise reduction degree of the fusion ratio of the outputs of different lookup tables in the multidimensional lookup table.

[0080] In an embodiment of the present invention, weighted fusion is a technical means for finely adjusting the noise reduction result according to different weight values ​​in the dynamic weight vector. The first weight value and the second weight value in the dynamic weight vector respectively assume different functions: the first weight value is used to adjust the timing consistency of the first noise reduction result and the second noise reduction result. By adjusting the timing consistency, it is ensured that the noise reduction results between multiple frames are consistent in the time dimension, avoiding flickering and incoherence in the video. In the video sequence, the timing of noise changes requires that the noise reduction process should not only consider the noise characteristics of a single frame image, but also take into account the timing stability of the previous and next frames. By controlling the timing consistency through the first weight value, the time jump effect caused by noise reduction can be effectively balanced, thereby improving the viewing experience of the video. The second weight value adjusts the fusion ratio of the first noise reduction result output by the multidimensional lookup table, thereby affecting the global noise reduction degree. Through this adjustment means, the noise reduction intensity of the overall image can be flexibly controlled, so that the noise reduction effect can be retained in detail, and the interference of noise can be suppressed, thereby improving the video quality.

[0081] The first neural network model and the second neural network model adopt a two-stage training strategy. Specifically: In the first stage, the neural network of the first neural network model and the neural network of the second neural network model are trained independently to obtain the initial weights of their respective networks. The outputs of the first neural network model and the second neural network model are weighted fused and the network parameters are fine-tuned based on the initial weights obtained in the first stage, thereby achieving joint training. The weights in the temporal consistency fusion formula are controlled by the first weight value. During the training process, the first neural network model not only learns how to maintain temporal consistency, but also learns how to dynamically adjust the contribution ratio of different multidimensional lookup tables according to image content or noise characteristics. Therefore, the second weight value does not simply "adjust" the LUT itself, but adjusts the fusion ratio of different LUT output results. Through joint training, the first neural network can learn the optimal fusion strategy so that the final denoising result achieves the best balance between temporal consistency, denoising effect and detail retention.

[0082] In the inference stage of a specific implementation, the noise reduction strength is adjusted and integrated according to the N×M+1 values ​​of the dynamic weight vector. The weighting method is as follows:

[0083] For the output of N×M LUTs, the output of each LUT is multiplied by the corresponding weight value. To obtain the fused output. The formula for this process is:

[0084]

[0085] Where i = 1, 2...N, j = 1, 2...M, is the second weight value.

[0086] In this way, the proportions of different lookup table (LUT) outputs can be adjusted according to dynamic weights to balance the noise reduction strength and temporal consistency of the image.

[0087] The following weighting method is used to adjust the fusion process of the second denoising result output by the first neural network model and the first denoising result obtained by searching the multidimensional lookup table:

[0088]

[0089] Among them, Output_lut is the first denoising result obtained by searching the multidimensional lookup table, and Output_t is the second denoising result output by the first neural network model. is the first weight value. This formula effectively balances the effects of single-frame noise reduction and temporal consistency by weighted fusion of the first noise reduction result and the second noise reduction result.

[0090] In an embodiment of the present invention, through weighted fusion based on dynamic weight vectors, the present invention can adjust the noise reduction results in real time during the reasoning process, which not only ensures the temporal consistency of image noise reduction, but also can flexibly control the intensity of global noise reduction. Specifically, the application of the first weight value ensures the temporal consistency between multiple frames of video, so that the video avoids flickering or jumping caused by asynchronous noise reduction during playback; while the second weight value plays a role in global noise reduction, adjusts the fusion ratio of different LUTs, and ensures reasonable control of the global noise reduction degree. Therefore, the weighted fusion mechanism can significantly improve the noise reduction effect of the video, ensure the continuity of objects or backgrounds in dynamic scenes, improve the viewing experience of the video stream, and effectively improve the video quality, especially in scenes with dynamic and complex backgrounds.

[0091] In some embodiments, the noise reduction method is implemented in combination with hardware resources, specifically including:

[0092] Graphics processing units (GPUs) are used to process multidimensional lookup table lookups. GPUs have powerful parallel computing capabilities and can process large amounts of pixel data simultaneously, making them particularly suitable for tasks involving large-scale data processing, such as image processing and lookup table (LUT) lookups. Using GPUs to accelerate this process can significantly shorten the lookup time of multidimensional lookup tables during the inference phase and improve inference speed.

[0093] A neural network processing unit (NPU) is used to process the noise reduction process of the first neural network model. The NPU is specially designed to efficiently perform inference tasks of deep learning models. It can efficiently process neural network-based noise reduction tasks, especially suitable for the noise reduction process of the first neural network model. By running the neural network model on the NPU, more efficient calculations and lower latency can be achieved, ensuring the real-time noise reduction effect.

[0094] In an embodiment of the present invention, the GPU and the NPU perform their respective processing tasks in parallel. This parallel execution mode enables the two hardware to play their respective advantages and process different computing tasks respectively. Among them, the GPU is responsible for the calculation of multi-dimensional lookup table lookup. This process usually requires parallel calculation of a large amount of pixel data, so the parallel computing capability of the GPU enables the process to be completed quickly. The NPU is responsible for running the first neural network model to perform denoising processing on the video frame. The NPU is specially optimized for the computing tasks of the neural network, so that the neural network reasoning process can be more efficient and reduce the delay in the traditional CPU computing process. Due to the parallel execution of the two, the entire denoising process is accelerated, especially when processing complex video scenes, it can ensure that efficient denoising effects are achieved while ensuring video quality. This parallel computing mode can give full play to the computing advantages of the hardware, avoid resource waste and computing bottlenecks, and ensure the efficient completion of video denoising tasks. In actual use, the functions of the graphics processing unit (GPU) and the neural network processing unit (NPU) can be implemented by other hardware acceleration units. For example, the tensor processing unit (TPU) can also be used to accelerate the denoising process of the neural network, while the digital signal processor (DSP) can be used to accelerate specific types of image processing tasks. According to the characteristics of different hardware architectures, suitable hardware units can be flexibly selected to optimize the noise reduction process, and the present invention does not impose specific limitations on this.

[0095] like Figure 4 As shown, an embodiment of the present invention further provides a video noise reduction system, which is applied to the above-mentioned video noise reduction method, and the system includes:

[0096] The video acquisition module M100 is used to acquire video frame data, target frame data and its associated frame data. The main function of this module is to acquire input video frame data, including target frame data and associated frame data (such as previous and next adjacent frames). The target frame data is the frame to be processed, while the associated frame data is used to help capture timing information, thereby improving the noise reduction effect.

[0097] The single-frame noise reduction module M200 is used to search the pixels in the target frame data and the pixel values ​​of the receptive field in which they are located in a pre-built multidimensional lookup table to obtain a first noise reduction result. Specifically, the module searches the pixels in the target frame data and the pixel values ​​of the receptive field in which they are located in a pre-built multidimensional lookup table, and finds the noise reduction result in the pre-built lookup table. The single-frame noise reduction module enables the system to process noise for each frame separately, thereby obtaining a single-frame noise reduction result.

[0098] The consistency module M300 is used to receive the target frame data and its associated frame data, and output a second denoising result and a dynamic weight vector containing temporal consistency. Temporal consistency is a key issue in video denoising because there is a strong temporal dependency between adjacent frames in a video sequence. By considering the data of the previous and next frames, the consistency module can generate a smooth and stable denoising effect.

[0099] The noise reduction fusion module M400 is used to perform weighted fusion on the first noise reduction result and the second noise reduction result based on the dynamic weight vector to generate a final noise reduction result. The fusion process adjusts the balance between the output of the single-frame noise reduction module M200 and the output of the consistency module M300 according to the weight vector, thereby generating a noise reduction video that maintains the details within the frame and has temporal consistency.

[0100] Through the video noise reduction system of the present invention, the noise reduction process in the video stream can be efficiently realized while ensuring the timing consistency. The modules of the system have clear division of labor and work in coordination, thereby effectively improving the overall effect of video noise reduction. The single-frame noise reduction module M200 ensures the noise reduction quality of each frame, and the consistency module M300 adjusts the noise reduction result through multi-frame timing information to avoid the problem of inconsistent timing in the video stream. The noise reduction fusion module M400 uses a dynamic weight vector to flexibly adjust the ratio of the two noise reduction results, so that the final output video has a better visual effect, which is both smooth and retains details. In addition, the whole system can quickly process high-quality video streams through the combination of hardware acceleration, which provides effective technical support for real-time video applications (such as video conferencing, live broadcasting, security monitoring, etc.).

[0101] An embodiment of the present invention further provides a video noise reduction device, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to execute the steps of the video noise reduction method by executing the executable instructions.

[0102] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0103] Refer to the following Figure 5 The electronic device 600 according to this embodiment of the present invention is described. Figure 5 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0104] like Figure 5As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0105] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the electronic prescription circulation processing method section of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0106] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0107] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0108] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0109] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0110] By adopting the video noise reduction device provided by the present invention, the processor executes the video noise reduction method when executing the executable instructions, thereby achieving the beneficial effects of the above-mentioned video noise reduction method.

[0111] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the video noise reduction method are implemented when the program is executed by a processor. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned electronic prescription circulation processing method section of this specification.

[0112] refer to Figure 6 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0113] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0114] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0115] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0116] By adopting the computer-readable storage medium provided by the present invention, the program stored therein implements the steps of the video noise reduction method when being executed, thereby obtaining the beneficial effects of the above-mentioned video noise reduction method.

[0117] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A video noise reduction method, characterized in that: The steps include: Collecting video frame data, searching pixels in the target frame data and pixel values ​​of the receptive field in which they are located in a pre-built multidimensional lookup table to obtain a first denoising result; Using the first neural network model to perform noise reduction processing on the target frame data and its associated frame data, generating a second noise reduction result and a dynamic weight vector including time consistency; Based on the dynamic weight vector, the first denoising result and the second denoising result are weightedly fused to generate a final video denoising result.

2. The video noise reduction method according to claim 1, characterized in that: The multidimensional lookup table is pre-constructed based on the second neural network model, and the construction process includes the following steps: By uniformly sampling the 2D noise reduction parameters and 3D noise reduction parameters of the video frame data, N 2D sampling points and M 3D sampling points are generated at a predetermined sampling interval, and N×M noise degradation conditions are obtained; Based on the N×M noise degradation conditions, corresponding N×M training subsets are generated respectively, and N×M independent second neural network models are trained respectively to obtain second neural network models corresponding to the N×M noise degradation conditions one by one; For each of the second neural network models, according to the size of its receptive field, all possible combinations of receptive field pixel values ​​are generated as input samples, and reasoning is performed through the second neural network model to obtain a corresponding denoising output result; Each of the second neural network model input samples and its corresponding denoising output result are stored in a lookup table to generate N×M multi-dimensional lookup tables.

3. The video noise reduction method according to claim 2, characterized in that: In the inference stage, based on the pixels in the target frame data and the pixel values ​​of their receptive fields, searches are performed in N×M lookup tables respectively to obtain N×M search results as the first denoising results.

4. The video noise reduction method according to claim 1, characterized in that: The first neural network model is an encoder-decoder network structure, the encoder receives the target frame data and its associated frame data as input, the decoder outputs a second denoising result containing temporal consistency, the decoder further comprises a parallel branch, the parallel branch is used to generate a dynamic weight vector for denoising adjustment, the parallel branch comprises: At least one pooling layer, used to compress the feature map output by the decoder into a low-dimensional vector; At least one fully connected layer is used to generate a dynamic weight vector of size N×M+1 according to the low-dimensional vector output by the pooling layer.

5. The video noise reduction method according to claim 1, characterized in that: The associated frame data includes at least one adjacent frame before and after the target frame. The first neural network model receives multiple frames of data as input through a time-series stacking method to generate a second denoising result.

6. The video noise reduction method according to claim 1, characterized in that: The method for weighted fusion of the first denoising result and the second denoising result based on the dynamic weight vector includes: A first weight value based on the dynamic weight vector is used to adjust the timing consistency of the first noise reduction result and the second noise reduction result; The second weight value based on the dynamic weight vector is used to adjust the global noise reduction degree of the fusion ratio of the outputs of different lookup tables in the multidimensional lookup table.

7. The video noise reduction method according to claim 1, characterized in that: The noise reduction method is implemented in combination with hardware resources, specifically including: A graphics processing unit for processing a multi-dimensional lookup table lookup process; A neural network processing unit, used for processing a noise reduction process of the first neural network model; The graphics processing unit and the neural network processing unit perform their respective processing tasks in parallel.

8. A video noise reduction system, characterized in that: The video noise reduction method applied to any one of claims 1 to 7, wherein the system comprises: A video acquisition module, used to acquire video frame data, target frame data and associated frame data; A single-frame denoising module is used to obtain a first denoising result by searching a pixel in the target frame data and a pixel value parameter of the receptive field in which the pixel is located in a pre-built multi-dimensional lookup table; A consistency module, used for receiving target frame data and its associated frame data, and outputting a second noise reduction result and a dynamic weight vector including time sequence consistency; The denoising fusion module is used to perform weighted fusion on the first denoising result and the second denoising result based on the dynamic weight vector to generate a final denoising result.

9. A video noise reduction device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to perform the steps of the video noise reduction method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the video noise reduction method according to any one of claims 1 to 7 are implemented.

Images