Time sequence random noise generation method and device based on deep learning
Through the time sequence random noise generation method based on deep learning, a noise addition model is constructed and trained to generate time sequence-changing noise, which solves the problem of low timing denoising accuracy in motion scenes and achieves an efficient timing denoising effect.
Patent Information
- Application Number
- CN202411983562.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
Existing video denoising algorithms are difficult to effectively handle timing denoising in motion scenes, especially because the existing noise-added model cannot effectively carry the noise of timing variation, resulting in low timing denoising accuracy of motion scenes.
Using a time-sequence random noise generation method based on deep learning, a noise-added model is constructed, which includes a convolutional layer, residual unit and noise modulation module. A paired data set is constructed using video frames and high-quality images of static scenes, and the noise-added model is trained to generate time-sequence-changing noise.
It effectively solves the difficulties in the acquisition of pairing data in sports scenes, greatly saves manpower and material costs, and at the same time improves the ability of the timing denoising model and improves the timing denoising accuracy of the sports scenes.
Smart Images
Figure CN119919672A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of video denoising, and in particular to a method and device for generating temporal random noise based on deep learning. Background Art
[0002] Denoising algorithms based on deep learning are mainly supervised learning methods, which require noise-clean paired image pairs as model training data. Since the original camera noise can usually be expressed as a near-Gaussian distribution, it is relatively easy to collect paired data for static scenes. As long as the temporal mean filter is used, a sample image with good clarity can be obtained. However, paired data for moving scenes is difficult to collect, and usually noise needs to be superimposed using a simulation method. However, since the simulated noise is different from the actual noise, the temporal denoising of moving objects is not well solved after the constructed noise paired data is used for video denoising.
[0003] Currently, existing noise-adding models are usually single-input and single-output, that is, after the same definition image is processed by the model, only a single noise image can be obtained. This is inconsistent with reality, because the noise in static scenes such as camera recordings or real-time conferences changes over time.
[0004] With respect to the above-mentioned related technologies, the inventors found that the existing video frame training data has the problem of difficulty in carrying noise with temporal variations and the constructed noise pairing data has low accuracy when applied to temporal denoising of motion scenes. Summary of the invention
[0005] In order to generate time-series varying noise and improve the accuracy of time-series denoising of motion scenes, the present application provides a method and device for generating time-series random noise based on deep learning.
[0006] In a first aspect, the present application provides a method for generating time series random noise based on deep learning.
[0007] This application is achieved through the following technical solutions:
[0008] A method for generating time series random noise based on deep learning, comprising the following steps:
[0009] Obtain video frames and corresponding high-quality images in static scenes to build a paired dataset;
[0010] Constructing a noise adding model, wherein the noise adding model comprises a convolution layer, a residual unit and a noise modulation module connected in sequence, the residual unit is composed of two convolution layers and an addition short circuit, the residual unit is used for feature extraction, the noise modulation module is composed of a convolution layer and two parallel residual units, and the noise modulation module is used to generate an overall noise image by multiplication and addition of the input random noise;
[0011] Using the paired data set to train the noise adding model;
[0012] The current high-quality image frame, the previous video frame and random noise are input into the trained denoising model, and the time-series denoised video frames are output.
[0013] In a preferred example, the present application can be further configured as follows: the step of inputting the current high-quality image frame, the previous video frame and the random noise into the trained noise adding model and outputting the time-series noise adding video frame includes:
[0014] Input the current high-quality image frame into the trained denoising model to perform denoising on shallow features, and obtain a first denoising result;
[0015] Fusing the first noise addition result with the shallow features of the previous high-quality image frame to obtain a second noise addition result;
[0016] Performing noise addition on the deep features based on the second noise addition result to obtain a third noise addition result;
[0017] The third noise addition result is added to the current high-quality image frame to obtain the temporal noise addition video frame.
[0018] In a preferred example, the present application can be further configured as follows: the step of acquiring video frames and corresponding high-quality images in a static scene and constructing a paired data set includes:
[0019] Collect several static scene videos with different backgrounds under different environmental conditions;
[0020] Filter and denoise the video frames of all static scene videos to obtain the corresponding denoised images;
[0021] Each denoised image is copied to the same length as a video frame of the static scene video to obtain a high-quality image corresponding to the video frame as the paired data set.
[0022] In a preferred example, the present application may be further configured as follows: the step of using the paired data set to train the noise adding model further includes:
[0023] Constructing a loss function, the loss function comprising an image loss part for ensuring that the noisy video frame still has a denoising effect after filtering, an image size part for ensuring that the noise level of the video frame and the actual noise level have the same scale, and an image timing difference part for keeping the noise difference between the previous and next video frames consistent with the original video frame;
[0024] Based on the loss function, the paired data set is input into the denoising model for training, time-series denoised video frames are output, and model parameters are optimized until the denoising model converges, and the denoising model at this time is used as a trained denoising model.
[0025] In a preferred example, the present application can be further configured as follows: the image loss part of the loss function adopts the following formula:
[0026] L image (LQ, Y) = L1 (Tem ave (LQ 0-n ), Tem ave (GT 0-n ))
[0027] Where, L image refers to image loss, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and Tem ave Refers to time series mean filtering, LQ 0-n Refers to the first video frame, GT 0-n Refers to the high-quality image corresponding to the first video frame, and n represents the length of the video frame.
[0028] In a preferred example, the present application can be further configured as follows: the image size part of the loss function adopts the following formula,
[0029]
[0030] Where, L ave refers to the image size, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and LQ i Refers to the current video frame of the static scene video, GT i Refers to the high-quality image corresponding to the current video frame, Y i Refers to the noisy frame corresponding to the current video frame, and n represents the length of the video frame.
[0031] In a preferred example, the present application can be further configured as follows: the image time sequence difference part of the loss function adopts the following formula,
[0032]
[0033] Where, L var refers to the image time difference, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and LQ i Refers to the current video frame of the static scene video, LQ i-1 Refers to the previous video frame of the current video frame, Y iRefers to the noise-added frame corresponding to the current video frame, Y i-1 The noisy frame corresponding to the previous video frame of the current video frame, where n represents the length of the video frame.
[0034] In a preferred example, the present application can be further configured as follows: the formula of the loss function includes:
[0035]
[0036] In the formula, is the loss function, L image Refers to image loss, L ave refers to the image size, λ ave Refers to the image size weight, L var refers to the image temporal difference, λ var refers to the image temporal difference weight, LQ refers to the video frame of the static scene video, and Y refers to the noisy frame.
[0037] In a second aspect, the present application provides a time series random noise generation device based on deep learning.
[0038] This application is achieved through the following technical solutions:
[0039] A time series random noise generating device based on deep learning, comprising:
[0040] The video data module is used to obtain video frames and corresponding high-quality images in static scenes and construct a paired data set;
[0041] A model construction module, used to construct a noise-added model, wherein the noise-added model includes a convolutional layer, a residual unit and a noise modulation module connected in sequence, the residual unit is composed of two convolutional layers and an addition short circuit, the residual unit is used for feature extraction, the noise modulation module is composed of a convolutional layer and two parallel residual units, and the noise modulation module is used to generate an overall noise image by multiplication and addition of input random noise;
[0042] A model training module, used for training the noise adding model using the paired data set;
[0043] The denoising module is used to input the current high-quality image frame, the previous video frame and random noise into the trained denoising model and output a time-series denoised video frame.
[0044] In a third aspect, the present application provides a computer device.
[0045] This application is achieved through the following technical solutions:
[0046] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any one of the above-mentioned methods for generating temporal random noise based on deep learning are implemented.
[0047] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:
[0048] The video frames and the corresponding high-quality images in static scenes are obtained, and a paired data set is constructed to be used as model training data; a denoising model is constructed, and the denoising model is trained using the paired data set to learn the mapping relationship between the video frames and the corresponding high-quality images, simulate the denoising process of static scenes, and generate an overall noise image by multiplication and addition of the input random noise, so that a large number of noise images with the same distribution but different noise positions can be generated, which is suitable for static scenes and dynamic scenes; the current high-quality image frame, the previous video frame and the random noise are input into the trained denoising model, and the time-series denoised video frame is output to generate time-series noise, which solves the difficulty of paired data collection for motion scenes and greatly saves the manpower and material resources spent on paired data collection. At the same time, the generated noise / clean paired image data set can effectively improve the ability of the time-series denoising model and improve the accuracy of time-series denoising for motion scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 A schematic diagram of the overall process of a method for generating temporal random noise based on deep learning is provided as an exemplary embodiment of the present application.
[0050] Figure 2 A structural block diagram of a noise adding model of a method for generating temporal random noise based on deep learning is provided as an exemplary embodiment of the present application.
[0051] Figure 3 A structural block diagram of a timing random noise generating device based on deep learning is provided as another exemplary embodiment of the present application. DETAILED DESCRIPTION
[0052] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.
[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0054] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.
[0055] In order to generate time-varying noise and improve the accuracy of time-series denoising of motion scenes, this application collects a small number of static scene time-series dynamic noise images as LQ (Low Quality Image), and then obtains the corresponding high-quality image GT (GT, Ground Truth, positive sample) through filtering and denoising; then builds a deep learning model, learns the mapping relationship from GT to LQ, simulates the noise addition process of static scenes, and uses different random numbers to generate a large number of random noises with the same distribution but different noise positions. By introducing the upper and lower frame feature fusion module, time-series noise images with the same distribution but different positions of adjacent frames can be generated, which can be used for denoising of static and dynamic scenes at the same time, solving the difficulty of motion scene paired data collection, and saving labor costs and training costs by building a noise / clean paired image data set, and can also effectively improve the capabilities of the time-series denoising model.
[0056] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.
[0057] Reference Figure 1 , an embodiment of the present application provides a method for generating time series random noise based on deep learning, and the main steps of the method are described as follows.
[0058] S1: Obtain video frames and corresponding high-quality images in static scenes to build a paired dataset;
[0059] S2: constructing a noise adding model, wherein the noise adding model comprises a convolutional layer, a residual unit and a noise modulation module connected in sequence, the residual unit is composed of two convolutional layers and an addition short circuit, the residual unit is used for feature extraction, the noise modulation module is composed of a convolutional layer and two parallel residual units, and the noise modulation module is used to generate an overall noise image by multiplication and addition of the input random noise;
[0060] S3: training the noise adding model using the paired data set;
[0061] S4: Input the current high-quality image frame, the previous video frame and random noise into the trained denoising model, and output the time-series denoised video frame.
[0062] In one embodiment, the step of acquiring video frames and corresponding high-quality images in a static scene and constructing a paired data set includes:
[0063] Collect several static scene videos with different backgrounds under different environmental conditions;
[0064] Filter and denoise the video frames of all static scene videos to obtain the corresponding denoised images;
[0065] Each denoised image is copied to the same length as a video frame of the static scene video to obtain a high-quality image corresponding to the video frame as the paired data set.
[0066] In one embodiment, the step of inputting the current high-quality image frame, the previous video frame and the random noise into the trained noise adding model and outputting the time-series noise adding video frame comprises:
[0067] Input the current high-quality image frame into the trained denoising model to perform denoising on shallow features, and obtain a first denoising result;
[0068] Fusing the first noise addition result with the shallow features of the previous high-quality image frame to obtain a second noise addition result;
[0069] Performing noise addition on the deep features based on the second noise addition result to obtain a third noise addition result;
[0070] The third noise addition result is added to the current high-quality image frame to obtain the temporal noise addition video frame.
[0071] In one embodiment, the backbone structure of the noise-added model is linear.
[0072] In one embodiment, the step of training the noise adding model using the paired data set further includes:
[0073] Constructing a loss function, the loss function comprising an image loss part for ensuring that the noisy video frame still has a denoising effect after filtering, an image size part for ensuring that the noise level of the video frame and the actual noise level have the same scale, and an image timing difference part for keeping the noise difference between the previous and next video frames consistent with the original video frame;
[0074] Based on the loss function, the paired data set is input into the denoising model for training, time-series denoised video frames are output, and model parameters are optimized until the denoising model converges, and the denoising model at this time is used as a trained denoising model.
[0075] The detailed description of the above embodiments is as follows.
[0076] Use cameras and other equipment to aim at several static scenes with different backgrounds under different environmental conditions, collect static scene videos, and obtain video frames in static scenes.
[0077] For all video frames LQ (0-n) Perform time series mean filtering to denoise and obtain the denoising result, then copy the denoising result to the same length as the video frame to obtain a high-quality image GT corresponding to the video frame 0-n , construct a paired data set. The specific formula is as follows:
[0078]
[0079] GT 0-n =Copy[GT'](2)
[0080] In the formula, GT' is the denoising result, LQ 0-n is the video frame, Tem ave It is the time series mean filtering denoising, GT 0-n is a high-quality image corresponding to a video frame.
[0081] Among them, the device only needs to collect a small amount of static paired data set, which saves a lot of manpower and training costs compared to the existing technology that needs to collect a large number of noise-clean paired image pairs as model training data.
[0082] Next, refer to Figure 2 , construct a noise model, and the backbone of the noise model adopts a linear type. The noise model includes a convolutional layer, a residual unit ResBlock and a noise modulation module ModBlock connected in sequence. Among them, the residual unit consists of two convolutional layers and an addition short circuit, which is used for feature extraction. The noise modulation module consists of a convolutional layer and two parallel residual units, which are used to control the input random noise through multiplication and addition to generate an overall noise image. The random noise can be Gaussian random noise.
[0083] The denoising model receives the input high-definition image frame GT i (i.e. the current high-quality image frame), and then pass through the convolution layer and N (residual units + noise modulation modules) to add noise to the shallow features to obtain the first noise addition result; the first noise addition result is then concat-connected to the previous high-definition image frame GT i-1After the shallow features are obtained, the convolution layer performs feature fusion to obtain the second noisy result; then the second noisy result is denoised by deep features through M (residual units + noise modulation modules) to obtain the third noisy result; finally, the third noisy result and the high-definition image frame GT are combined. i Add together to get the final noisy frame Y i (i.e., time-sequentially noisy video frames). Wherein, N and M are positive integers, and the number can be set according to actual needs, for example, N=3, M=2.
[0084] Furthermore, a loss function is constructed to train the denoising model. The loss function includes an image loss part for ensuring that the denoised video frame still has a denoising effect after filtering, an image size part for ensuring that the noise level of the video frame and the actual noise level have the same scale, and an image timing difference part for keeping the noise difference between the previous and next video frames consistent with the original video frame.
[0085] By constructing a loss function, an image loss function is designed to ensure that the noisy video frame can still achieve the denoising effect after temporal mean filtering and has the same properties as the original noise; an image size function is designed to ensure that the noise level of each frame and the actual noise level have the same scale and size; an image temporal difference function is designed to ensure that the noise difference between the previous and next frames is consistent with that in the original video frame, avoiding that each frame of the noise image is in the same position, so that the model can obtain the optimal solution, and the output image Y of the model and the real noise image LQ have the same noise distribution to obtain a temporal noisy video frame.
[0086] In one embodiment, the loss function also includes a model loss part for evaluating the authenticity of the generated noise image and the actual noise image to optimize the model.
[0087] By designing the model loss function, the authenticity of the generated noise image and the actual noise image is evaluated, the model optimization direction is guided, and the model parameters are further optimized so that the image Y output by the model is closer to the real noise image LQ, so as to better simulate the time-series noisy video frame.
[0088] In this embodiment, the formula of the loss function can be as follows:
[0089]
[0090] In the formula, is the loss function, L image Refers to image loss, L ave refers to the image size, λ ave Refers to the image size weight, L var refers to the image temporal difference, λ var Refers to the image temporal difference weight, L grefers to the model loss, λ g refers to the model loss weight, LQ refers to the video frame of the static scene video, and Y refers to the noisy frame.
[0091] In one embodiment, the image loss part of the loss function adopts the following formula:
[0092] L image (LQ,Y)=L1(Tem ave (LQ 0-n ),Tem ave (GT 0-n )) (4)
[0093] Where, L image refers to image loss, L1 represents a norm, which is the pixel difference between two images, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, and Tem ave Refers to time series mean filtering, LQ 0-n Refers to the first video frame, GT 0-n Refers to the high-quality image corresponding to the first video frame, and n represents the length of the video frame.
[0094] In one embodiment, the image size portion of the loss function uses the following formula,
[0095]
[0096] Where, L ave refers to the image size, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and LQ i Refers to the current video frame of the static scene video, GT i Refers to the high-quality image corresponding to the current video frame, Y i Refers to the noisy frame corresponding to the current video frame, and n represents the length of the video frame.
[0097] In one embodiment, the image timing difference part of the loss function adopts the following formula:
[0098]
[0099] Where, L var refers to the image time difference, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and LQ i Refers to the current video frame of the static scene video, LQ i-1 Refers to the previous video frame of the current video frame, Y i Refers to the noise-added frame corresponding to the current video frame, Y i-1 The noisy frame corresponding to the previous video frame of the current video frame, where n represents the length of the video frame.
[0100] In one embodiment, the model loss part of the loss function adopts the following formula:
[0101] L g (LQ,Y)=L1(LQ 0-n ,Y 0-n ) (7)
[0102] Where, L g refers to the model loss, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and LQ 0-n Refers to the first video frame, Y 0-n Refers to the noise-added frame corresponding to the first video frame.
[0103] When the noise model converges during training, the HD video frame sequence is converted into GT 0-n , the previous video frame and random noise such as N 0-n Input into the trained model can generate time-series noisy video frames such as Y 0-n , to provide a large amount of training data for deep learning video denoising models.
[0104] Due to the existence of temporal feature fusion, the denoising model can simultaneously process static and dynamic scenes of high-definition video frames, and combine different random numbers to control the noise. Without changing the noise intensity, the random position of the noise is changed to generate a large number of random noise image frames with different positions but the same distribution, which can be used for denoising model training, making it convenient to build a large number of paired data sets.
[0105] In summary, a method for generating temporal random noise based on deep learning obtains video frames and corresponding high-quality images in static scenes, constructs a paired data set, and uses the paired data set to train the denoising model to learn the mapping relationship between video frames and corresponding high-quality images, simulate the denoising process of static scenes, and generates an overall noise image by multiplication and addition of the input random noise, which can generate a large number of noise images with the same distribution but different noise positions, and is suitable for static scenes and dynamic scenes; the current high-quality image frame, the previous video frame and the random noise are input into the trained denoising model, and the temporal denoised video frame is output to generate temporally changing noise, which solves the difficulty of paired data collection for motion scenes and greatly saves the manpower and material resources spent on paired data collection. At the same time, the generated noise / clean paired image data set can effectively improve the ability of the temporal denoising model and improve the accuracy of temporal denoising for motion scenes.
[0106] A deep learning-based temporal random noise generation method uses large computing power to input video frames in constructed static scenes and corresponding high-quality images to generate approximately distributed noise and obtain temporal noisy video frames. Benefiting from the large computing power, the output results are closer to the real noise, thereby improving the denoising accuracy of subsequent models.
[0107] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0108] Reference Figure 3 The embodiment of the present application also provides a timing random noise generation device based on deep learning, which corresponds one-to-one to a timing random noise generation method based on deep learning in the above embodiment. The timing random noise generation device based on deep learning includes:
[0109] The video data module is used to obtain video frames and corresponding high-quality images in static scenes and construct a paired data set;
[0110] A model construction module, used to construct a noise-added model, wherein the noise-added model includes a convolutional layer, a residual unit and a noise modulation module connected in sequence, the residual unit is composed of two convolutional layers and an addition short circuit, the residual unit is used for feature extraction, the noise modulation module is composed of a convolutional layer and two parallel residual units, and the noise modulation module is used to generate an overall noise image by multiplication and addition of input random noise;
[0111] A model training module, used for training the noise adding model using the paired data set;
[0112] The denoising module is used to input the current high-quality image frame, the previous video frame and random noise into the trained denoising model and output a time-series denoised video frame.
[0113] For the specific limitations of a timing random noise generating device based on deep learning, please refer to the limitations of a timing random noise generating method based on deep learning above, which will not be repeated here.
[0114] Each module in the above-mentioned timing random noise generation device based on deep learning can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0115] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, any of the above-mentioned methods for generating temporal random noise based on deep learning is implemented.
[0116] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned methods for generating time series random noise based on deep learning is implemented.
[0117] In one embodiment, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it implements any of the above-mentioned methods for generating time series random noise based on deep learning.
[0118] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0119] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.
Claims
1. A method for generating temporal random noise based on deep learning, characterized in that: The following steps are included: Obtain video frames and corresponding high-quality images in static scenes to build a paired dataset; Constructing a noise adding model, wherein the noise adding model comprises a convolution layer, a residual unit and a noise modulation module connected in sequence, the residual unit is composed of two convolution layers and an addition short circuit, the residual unit is used for feature extraction, the noise modulation module is composed of a convolution layer and two parallel residual units, and the noise modulation module is used to generate an overall noise image by multiplication and addition of the input random noise; Using the paired data set to train the noise adding model; The current high-quality image frame, the previous video frame and random noise are input into the trained denoising model, and the time-series denoised video frames are output.
2. The method for generating temporal random noise based on deep learning according to claim 1, characterized in that: The step of inputting the current high-quality image frame, the previous video frame and random noise into the trained noise adding model and outputting the time-series noise adding video frame comprises: Input the current high-quality image frame into the trained denoising model to perform denoising on shallow features, and obtain a first denoising result; Fusing the first noise addition result with the shallow features of the previous high-quality image frame to obtain a second noise addition result; Performing noise addition on the deep features based on the second noise addition result to obtain a third noise addition result; The third noise addition result is added to the current high-quality image frame to obtain the temporal noise addition video frame.
3. The method for generating temporal random noise based on deep learning according to claim 1, characterized in that: The step of acquiring video frames and corresponding high-quality images in a static scene and constructing a paired data set includes: Collect several static scene videos with different backgrounds under different environmental conditions; Filter and denoise the video frames of all static scene videos to obtain the corresponding denoised images; Each denoised image is copied to the same length as a video frame of the static scene video to obtain a high-quality image corresponding to the video frame as the paired data set.
4. The method for generating temporal random noise based on deep learning according to claim 1, characterized in that: The step of training the noise adding model using the paired data set also includes: Constructing a loss function, the loss function comprising an image loss part for ensuring that the noisy video frame still has a denoising effect after filtering, an image size part for ensuring that the noise level of the video frame and the actual noise level have the same scale, and an image timing difference part for keeping the noise difference between the previous and next video frames consistent with the original video frame; Based on the loss function, the paired data set is input into the denoising model for training, time-series denoised video frames are output, and model parameters are optimized until the denoising model converges, and the denoising model at this time is used as a trained denoising model.
5. The method for generating temporal random noise based on deep learning according to claim 4, characterized in that: The image loss part of the loss function adopts the following formula, L image (LQ,Y)=L1(item ave (LQ 0-n ),Tem ave (GT 0-n )) Where, L image refers to image loss, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and Tem ave Refers to time series mean filtering, LQ 0-n Refers to the first video frame, GT 0-n Refers to the high-quality image corresponding to the first video frame, and n represents the length of the video frame.
6. The method for generating temporal random noise based on deep learning according to claim 4, characterized in that: The image size part of the loss function uses the following formula, Where, L ave refers to the image size, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and LQ i Refers to the current video frame of the static scene video, GT i Refers to the high-quality image corresponding to the current video frame, Y i Refers to the noisy frame corresponding to the current video frame, and n represents the length of the video frame.
7. The method for generating temporal random noise based on deep learning according to claim 4, characterized in that: The image temporal difference part of the loss function adopts the following formula, Where, L var refers to the image time difference, LQ refers to the video frame of the static scene video, Y refers to the noisy frame, L1 represents a norm, which is the pixel difference between the two images, and LQ i Refers to the current video frame of the static scene video, LQ i-1 Refers to the previous video frame of the current video frame, Y i Refers to the noise-added frame corresponding to the current video frame, Y i-1 The noisy frame corresponding to the previous video frame of the current video frame, where n represents the length of the video frame.
8. The method for generating temporal random noise based on deep learning according to claim 4, characterized in that: The formula of the loss function includes: In the formula, is the loss function, L image Refers to image loss, L ave refers to the image size, λ ave Refers to the image size weight, L var refers to the image temporal difference, λ var refers to the image temporal difference weight, LQ refers to the video frame of the static scene video, and Y refers to the noisy frame.
9. A time series random noise generation device based on deep learning, characterized in that: include, The video data module is used to obtain video frames and corresponding high-quality images in static scenes and construct a paired data set; A model construction module, used to construct a noise-added model, wherein the noise-added model includes a convolutional layer, a residual unit and a noise modulation module connected in sequence, the residual unit is composed of two convolutional layers and an addition short circuit, the residual unit is used for feature extraction, the noise modulation module is composed of a convolutional layer and two parallel residual units, and the noise modulation module is used to generate an overall noise image by multiplication and addition of input random noise; A model training module, used for training the noise adding model using the paired data set; The denoising module is used to input the current high-quality image frame, the previous video frame and random noise into the trained denoising model and output a time-series denoised video frame.
10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.