Video de-noising data set construction method, system, device and medium
By performing multiple noise superposition and preprocessing of video information, analyzing and generating noise image information, the complex and time-consuming problem of noise data set construction in the prior art is solved, and more realistic noise simulation and more efficient video denoising processing are achieved.
Patent Information
- Application Number
- CN202411981966.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-06
AI Technical Summary
Existing video denoising methods rely on building high-quality noise or clean image pairing data sets. This process is complex and time-consuming. The noise superimposed by the existing methods in the RGB domain cannot represent real sensor noise. The noise image needs to go through a series of transformation processes from acquisition to imaging, and the scene of collecting real pairing data is limited.
By acquiring the initial image information of the video information, performing a first noise superimposed on the initial image information, obtaining the initial noise superimposed image information, and then performing noise-free preprocessing on the initial image information to obtain the preprocessed image information, performing noise analysis based on the initial noise superimposed image information and the preprocessed image information, proposing the noise image information in the initial noise superimposed image information, and finally performing a second noise superimposed on the initial image information and the noise image information to obtain the output noise superimposed image information.
By simulating real scene noise pairing data more realistically, the problem of inefficient acquisition of real pairing data is solved, the processing efficiency is improved, and the development efficiency of video denoising algorithm is improved.
Smart Images
Figure CN119941553A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of video denoising, and in particular to a method, system, device and medium for constructing a video denoising dataset. Background Art
[0002] With the popularization of video conferencing technology and the increasing demand for video communication, high-definition video images and smooth user experience have become key factors, especially in the application of fixed-focus lenses. When zooming in on a specific person, the video quality is easily affected by the noise caused by digital amplification, resulting in blurry images and significant noise. Although traditional video denoising methods are effective, they rely on building a high-quality noisy or clean image pairing dataset, which is complex and time-consuming.
[0003] At present, the paired data construction methods include collecting clean images, directly superimposing Gaussian and Poisson noise on the original RGB images as simulated noise images; controlling the exposure time, using long-exposure images as clean images and short-exposure images as noise images; and fitting and simulating noise distribution by shooting images with multiple sensitivities of the device. The noise superimposed on the RGB domain by the above existing methods cannot represent the real sensor noise. The noise image needs to go through a series of transformation processes from collection to imaging. In addition, the scenarios for collecting real paired data are limited, and noisy video images cannot be effectively processed, which consumes a lot of manpower and material resources. The above problems need to be solved. Summary of the invention
[0004] In order to simulate real scene noise paired data more realistically than the original noise simulation method, solve the problem of low efficiency in collecting real paired data, and improve processing efficiency, this application provides a video denoising dataset construction method, system, device and medium, using the following technical solutions:
[0005] In a first aspect, the present application provides a method for constructing a video denoising dataset, comprising:
[0006] Acquire initial image information of the video information, and perform a first noise superposition on the initial image information to obtain initial noise superposition image information;
[0007] Performing noise-free preprocessing on the initial image information to obtain preprocessed image information;
[0008] Perform noise analysis according to the initial noise superimposed image information and the preprocessed image information to obtain noise image information;
[0009] A second noise superposition is performed on the initial image information and the noise image information to obtain output noise superposition image information.
[0010] Preferably, the specific steps of performing the first noise superposition on the initial image information to obtain the initial noise superposition image information are:
[0011] Perform mosaic operation on the initial image information to obtain an initial single-channel image;
[0012] Add noise data to the initial single-channel image to obtain a noisy single-channel image;
[0013] The noisy single-channel image is demosaiced to obtain the initial noise superimposed image information.
[0014] Preferably, the specific steps of adding noise data to the initial single-channel image to obtain a noisy single-channel image are:
[0015] The initial single-channel image is converted into a four-channel image to obtain an initial four-channel image;
[0016] Add noise data to the initial four-channel image to obtain a noisy four-channel image;
[0017] The noisy four-channel image is restored to obtain a noisy single-channel image.
[0018] Preferably, the specific steps of performing noise-free preprocessing on the initial image information to obtain preprocessed image information are:
[0019] Perform mosaic operation on the initial image information to obtain an initial single-channel image;
[0020] The initial single-channel image is demosaiced to obtain preprocessed image information.
[0021] Preferably, the specific steps of performing noise analysis based on the initial noise superimposed image information and the preprocessed image information to obtain the noise image information are:
[0022] The initial noise-superimposed image information and the preprocessed image information are subtracted to obtain the noise image information in the initial noise-superimposed image information.
[0023] Preferably, the specific steps of performing a second noise superposition on the initial image information and the noise image information to obtain the output noise superposition image information are:
[0024] The initial image information and the noise image information are superimposed to obtain output noise superimposed image information.
[0025] Preferably, the specific steps of adding noise data to the initial four-channel image to obtain the noisy four-channel image are:
[0026] Gaussian noise or Poisson noise is constructed for the initial four-channel image according to the covariance matrix to obtain a noisy four-channel image.
[0027] In a second aspect, the present application provides a video denoising dataset construction system, comprising:
[0028] A first superposition module is used to obtain initial image information of the video information, and perform a first noise superposition on the initial image information to obtain initial noise superposition image information;
[0029] A preprocessing module, used for performing noise-free preprocessing on the initial image information to obtain preprocessed image information;
[0030] A noise analysis module, used for performing noise analysis based on the initial noise superimposed image information and the pre-processed image information to obtain noise image information;
[0031] The second superposition module is used to perform a second noise superposition on the initial image information and the noise image information to obtain output noise superposition image information.
[0032] In a third aspect, the present application provides a video denoising dataset construction device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the video denoising dataset construction method as described above.
[0033] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the video denoising dataset construction method as described above when running.
[0034] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:
[0035] The present application first obtains initial image information in video information, performs a first noise superposition on the initial image information to obtain initial noise superposition image information, then performs noise-free preprocessing on the initial image information to obtain preprocessed image information, performs noise analysis on the initial noise superposition image information and the preprocessed image information, proposes noise image information in the initial noise superposition image information, and finally obtains output noise superposition image information based on the superposition of the initial image information and the noise image information. By more realistically simulating real scene noise pairing data than the original simulated noise, the problem of inefficient collection of real paired data is solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a flowchart of a method for constructing a video denoising dataset described in an embodiment of the present application.
[0037] Figure 2 It is a flowchart of the method for generating a noise preprocessed image according to an embodiment of the present application.
[0038] Figure 3It is a module diagram of a video denoising dataset construction system described in an embodiment of the present application.
[0039] Description of reference numerals:
[0040] 1. First superposition module; 2. Preprocessing module; 3. Noise analysis module; 4. Second superposition module. DETAILED DESCRIPTION
[0041] The following combination Figure 1-Figure 3 The present application is described in further detail. The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting.
[0042] In a video conference scenario, when a fixed-focus lens is used to zoom in on a person, digital zoom is used. This method uses software interpolation to amplify the signal, and the noise in the original acquisition signal will also be amplified synchronously. Static noise and motion noise become very obvious, affecting the user experience. Using deep learning for video denoising has been proven to be effective, but deep learning requires the construction of a paired data set of noisy or clean images. Paired data sets are relatively difficult to construct. Existing paired data construction methods are mainly divided into three categories. The first is to collect clean images, and Gaussian and Poisson noise are directly superimposed on the original RGB image as a simulated noise image. The second is to control the exposure time, using a long exposure image as a clean image and a short exposure as a noise image. The third is to fit the noise distribution and simulate by shooting images with multiple sensitivities of the device. The problem with the above three methods is that the noise superimposed on the RGB domain cannot represent the real sensor noise. The noise image needs to go through a series of transformation processes from acquisition to imaging. The scene of collecting real paired data is limited, and it takes a lot of manpower and material resources. In order to deal with the above problems, the present application provides a video denoising dataset construction method, system, device and medium.
[0043] Reference Figure 1 , a method for constructing a video denoising dataset involved in this application specifically includes:
[0044] Acquire initial image information of the video information, and perform a first noise superposition on the initial image information to obtain initial noise superposition image information;
[0045] Performing noise-free preprocessing on the initial image information to obtain preprocessed image information;
[0046] Perform noise analysis according to the initial noise superimposed image information and the preprocessed image information to obtain noise image information;
[0047] A second noise superposition is performed on the initial image information and the noise image information to obtain output noise superposition image information.
[0048] Specifically, the present application obtains video information that needs to be constructed into a denoising data set, obtains initial image information in the video information, performs a first noise superposition on the initial image information, and obtains initial noise superposition image information. The initial noise superposition image information not only has noise, but also has signal loss, so it needs further processing. The initial image information is noise-free preprocessed to obtain preprocessed image information, and noise analysis is performed on the initial noise superposition image information and the preprocessed image information to propose noise image information in the initial noise superposition image information. At this time, the noise image information has been free of loss signals, and finally the output noise superposition image information is obtained by superimposing the initial image information and the noise image information. The present application effectively simulates the noise coupling problem in the RGB three channels, reduces the loss of manpower, material resources, and time in the process of paired data acquisition, and improves the development efficiency of the video denoising algorithm.
[0049] As one implementation manner, the specific steps of performing a first noise superposition on the initial image information to obtain the initial noise superposition image information are:
[0050] Perform mosaic operation on the initial image information to obtain an initial single-channel image;
[0051] Add noise data to the initial single-channel image to obtain a noisy single-channel image;
[0052] The noisy single-channel image is demosaiced to obtain the initial noise superimposed image information.
[0053] Reference Figure 2 Specifically, the embodiment of the present application needs to perform a first noise superposition on the initial image information, specifically, the original RGB image of the initial image information is converted into a Raw domain single-channel image of RGGB through a mosaic operation, and noise data is added by noise superposition to obtain a noisy Raw domain image, that is, a noisy single-channel image. Finally, it is converted into a noisy RGB image through a demosaic operation, that is, the initial noise superposition image information.
[0054] The Raw domain image of the embodiment of the present application is a single-channel image, which is usually arranged as RGGB or BGGR, that is, the values of the three channels RGB are adjacent in the HW space. In this case, the noise added, RGB has its own correlation, which is more consistent with the actual sensor noise. On the contrary, if noise is added directly in the RGB domain, the correlation of the noise of the three channels RGB cannot be guaranteed, and the authenticity is insufficient.
[0055] The mosaic operation and de-mosaic operation adopted in the embodiment of the present application are commonly used function processing methods. There is some signal loss when converting the three-channel to a single-channel Raw image. At this time, the pre-processed image is incomplete and there is signal loss, so further processing is required.
[0056] As one implementation method, the specific steps of adding noise data to the initial single-channel image to obtain a noisy single-channel image are as follows:
[0057] The initial single-channel image is converted into a four-channel image to obtain an initial four-channel image;
[0058] Add noise data to the initial four-channel image to obtain a noisy four-channel image;
[0059] The noisy four-channel image is restored to obtain a noisy single-channel image.
[0060] Specifically, in order to process a noisy single-channel image, the embodiment of the present application needs to simulate the noise coupling phenomenon between the three RGB colors in the real sensor during the noise superposition process. Specifically, the original single-channel Raw domain image is converted into an RGGB four-channel image, that is, the initial four-channel image, through a space2depth operation, and noise is added to the initial four-channel image to obtain a noisy four-channel image. Finally, it is restored to a single-channel noisy image, that is, a noisy single-channel image, through a depth2space operation.
[0061] As one implementation method, the specific steps of adding noise data to the initial four-channel image to obtain the noisy four-channel image are as follows:
[0062] Gaussian noise or Poisson noise is constructed for the initial four-channel image according to the covariance matrix to obtain a noisy four-channel image.
[0063] Specifically, the embodiment of the present application constructs multivariate Gaussian noise and Poisson noise through the covariance matrix, superimposes them on the four-channel image, and adds noise data to the four-channel image, which can simulate the imperfections in the actual image, so that it can be used as an important data enhancement method in tasks such as image denoising, generative adversarial network training, image classification and segmentation, and help the image processing system better deal with noise and uncertainty.
[0064] As one implementation method, the initial image information is subjected to noise-free preprocessing to obtain the preprocessed image information in the following specific steps:
[0065] Perform mosaic operation on the initial image information to obtain an initial single-channel image;
[0066] The initial single-channel image is demosaiced to obtain preprocessed image information.
[0067] Specifically, after obtaining the initial noise superimposed image information, since the initial noise superimposed image information still has signal loss, it is necessary to remove the signal loss data and the initial data therein, and further process to obtain the noise map, so the initial image information is preprocessed without noise superposition. First, a mosaic operation is performed. After the mosaic operation is performed and converted into a single-channel image, it is not necessary to superimpose noise, and a de-mosaic operation is performed to obtain preprocessed image information. At this time, the preprocessed image information is image information without superimposed noise but with signal loss.
[0068] As one implementation method, the specific steps of performing noise analysis based on the initial noise superimposed image information and the preprocessed image information to obtain the noise image information are as follows:
[0069] The initial noise-superimposed image information and the preprocessed image information are subtracted to obtain the noise image information in the initial noise-superimposed image information.
[0070] Specifically, after constructing the initial noise superposition image information and the preprocessed image information, the embodiment of the present application needs to process the initial noise superposition image information and the preprocessed image information to obtain the noise image information, and then use the noise image information to process a noisy image with reduced loss. Since the initial noise superposition image information and the preprocessed image information are both obtained through the same mosaic processing and de-mosaic processing, the difference is that the initial noise superposition image information is noise superimposed, so there is the same form of signal loss. By subtracting the two, a superimposed noise map can be obtained. Through the subtraction process, the noise map can eliminate the signal loss generated in the Mosaic and DeMosaic processes to obtain lossless noise image information.
[0071] As one implementation manner, the specific steps of performing a second noise superposition on the initial image information and the noise image information to obtain the output noise superposition image information are:
[0072] The initial image information and the noise image information are superimposed to obtain output noise superimposed image information.
[0073] Specifically, after acquiring the noise image information without signal loss, the embodiment of the present application needs to perform a superposition process with the original image, that is, superimpose it with the initial image information.
[0074] The embodiment of the present application specifically superimposes a noise image on the original image to obtain a final noise image. The noise image retains the original signal while superimposing three-channel coupled noise in the RGB domain, which can be closer to the generation method of real noise.
[0075] Finally, the original image and the noisy image form a paired data set, which can be sent to the deep learning denoising network to learn the denoising ability of the denoising model.
[0076] The embodiment of the present application simulates the transformation process of the Raw domain and the RGB domain during the image acquisition process, so that the superimposed noise has RGB coupling phenomenon, and considers the possible signal loss in the conversion process, constructs a preprocessed image of the same process to obtain a pure noise image, and superimposes the pure noise image on the original image to obtain a noise image without signal loss, thereby ensuring the univariate property in the paired data, effectively simulating the noise coupling problem in the RGB three channels, reducing the manpower, material resources and time loss in the paired data acquisition process, and improving the development efficiency of the video denoising algorithm.
[0077] Reference Figure 3 , a video denoising dataset construction system is provided for an embodiment of the present application, the system comprising:
[0078] A first superposition module is used to obtain initial image information of the video information, and perform a first noise superposition on the initial image information to obtain initial noise superposition image information;
[0079] A preprocessing module, used for performing noise-free preprocessing on the initial image information to obtain preprocessed image information;
[0080] A noise analysis module, used for performing noise analysis based on the initial noise superimposed image information and the pre-processed image information to obtain noise image information;
[0081] The second superposition module is used to perform a second noise superposition on the initial image information and the noise image information to obtain output noise superposition image information.
[0082] As one implementation manner, the first superposition module performs first noise superposition on the initial image information to obtain the initial noise superposition image information in the following specific steps:
[0083] Perform mosaic operation on the initial image information to obtain an initial single-channel image;
[0084] Add noise data to the initial single-channel image to obtain a noisy single-channel image;
[0085] The noisy single-channel image is demosaiced to obtain the initial noise superimposed image information.
[0086] As one implementation manner, the first superposition module adds noise data to the initial single-channel image to obtain a noisy single-channel image in the following specific steps:
[0087] The initial single-channel image is converted into a four-channel image to obtain an initial four-channel image;
[0088] Add noise data to the initial four-channel image to obtain a noisy four-channel image;
[0089] The noisy four-channel image is restored to obtain a noisy single-channel image.
[0090] As one implementation method, the preprocessing module performs noise-free preprocessing on the initial image information, and the specific steps of obtaining the preprocessed image information are as follows:
[0091] Perform mosaic operation on the initial image information to obtain an initial single-channel image;
[0092] The initial single-channel image is demosaiced to obtain preprocessed image information.
[0093] As one implementation manner, the noise analysis module performs noise analysis based on the initial noise superimposed image information and the preprocessed image information, and the specific steps of obtaining the noise image information are as follows:
[0094] The initial noise-superimposed image information and the preprocessed image information are subtracted to obtain the noise image information in the initial noise-superimposed image information.
[0095] As one implementation manner, the second superposition module performs a second noise superposition on the initial image information and the noise image information to obtain the output noise superposition image information in the following specific steps:
[0096] The initial image information and the noise image information are superimposed to obtain output noise superimposed image information.
[0097] As one implementation manner, the first superposition module adds noise data to the initial four-channel image to obtain the noisy four-channel image in the following specific steps:
[0098] Gaussian noise or Poisson noise is constructed for the initial four-channel image according to the covariance matrix to obtain a noisy four-channel image.
[0099] An embodiment of the present application provides a video denoising dataset construction device, including a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to execute the video denoising dataset construction method as described above.
[0100] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the video denoising dataset construction method as described above when running.
[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and product can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0102] In several embodiments provided in this application, it should be understood that the disclosed methods, systems, devices and program products may be implemented in other ways.
[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0104] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for constructing a video denoising dataset, characterized in that: include: Acquire initial image information of the video information, and perform a first noise superposition on the initial image information to obtain initial noise superposition image information; Performing noise-free preprocessing on the initial image information to obtain preprocessed image information; Perform noise analysis according to the initial noise superimposed image information and the preprocessed image information to obtain noise image information; A second noise superposition is performed on the initial image information and the noise image information to obtain output noise superposition image information.
2. The method for constructing a video denoising dataset according to claim 1, characterized in that: The specific steps of performing the first noise superposition on the initial image information to obtain the initial noise superposition image information are: Perform mosaic operation on the initial image information to obtain an initial single-channel image; Add noise data to the initial single-channel image to obtain a noisy single-channel image; The noisy single-channel image is demosaiced to obtain the initial noise superimposed image information.
3. The method for constructing a video denoising dataset according to claim 2, characterized in that: The specific steps of adding noise data to the initial single-channel image to obtain a noisy single-channel image are: The initial single-channel image is converted into a four-channel image to obtain an initial four-channel image; Add noise data to the initial four-channel image to obtain a noisy four-channel image; The noisy four-channel image is restored to obtain a noisy single-channel image.
4. The method for constructing a video denoising dataset according to claim 1, characterized in that: The specific steps of performing noise-free preprocessing on the initial image information to obtain preprocessed image information are: Perform mosaic operation on the initial image information to obtain an initial single-channel image; The initial single-channel image is demosaiced to obtain preprocessed image information.
5. The method for constructing a video denoising dataset according to claim 1, characterized in that: The specific steps of performing noise analysis based on the initial noise superimposed image information and the pre-processed image information to obtain the noise image information are: The initial noise-superimposed image information and the preprocessed image information are subtracted to obtain the noise image information in the initial noise-superimposed image information.
6. The method for constructing a video denoising dataset according to claim 1, characterized in that: The specific steps of performing a second noise superposition on the initial image information and the noise image information to obtain the output noise superposition image information are: The initial image information and the noise image information are superimposed to obtain output noise superimposed image information.
7. The method for constructing a video denoising dataset according to claim 3, characterized in that: The specific steps of adding noise data to the initial four-channel image to obtain a noisy four-channel image are: Gaussian noise or Poisson noise is constructed for the initial four-channel image according to the covariance matrix to obtain a noisy four-channel image.
8. A video denoising dataset construction system, characterized in that: include: A first superposition module is used to obtain initial image information of the video information, and perform a first noise superposition on the initial image information to obtain initial noise superposition image information; A preprocessing module, used for performing noise-free preprocessing on the initial image information to obtain preprocessed image information; A noise analysis module, used for performing noise analysis based on the initial noise superimposed image information and the pre-processed image information to obtain noise image information; The second superposition module is used to perform a second noise superposition on the initial image information and the noise image information to obtain output noise superposition image information.
9. A device for constructing a video denoising dataset, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method for constructing a video denoising dataset according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the video denoising dataset construction method according to any one of claims 1 to 7 when running.