Video data processing method, device and equipment and computer readable storage medium

Abstract

The invention discloses a video data processing method, device and equipment and a computer readable storage medium, and relates to the technical field of image processing, and the method comprises the steps: carrying out the noise reduction processing of each frame of original image data in original video data, and obtaining multi-frame noise reduction image data, performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data; on the basis of the enhanced image data corresponding to the original image data, performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to the original image data to obtain target image data; and performing fusion processing on the target image data to obtain target video data. According to the invention, the video quality of the video data shot in the low-illumination scene can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] With the continuous development and progress of science and technology, video technology, as an important means of information transmission and information recording, has been widely used in various fields of daily life. In this context, users have higher and higher requirements for the quality of video data and generally pursue high-quality visual experience.

[0003] Currently, in low-light scenes, such as at night or in dimly lit indoor scenes, if you want to obtain video data with good video effects, you usually need to use high-cost hardware equipment to shoot the video. However, the video data shot by most shooting equipment in low-light scenes usually does not have good video effects, and may have quality problems such as poor clarity, contrast, and color saturation.

[0004] Therefore, how to improve the video quality of video data captured in low-light scenes is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the present application is to provide a video data processing method, device, equipment and computer-readable storage medium, aiming to improve the video quality of video data captured in low-light scenes.

[0006] To achieve the above object, the present application provides a video data processing method, the video data processing method comprising:

[0007] Performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data;

[0008] Based on the enhanced image data corresponding to each of the original image data, respectively, the denoised image data corresponding to each of the original image data are subjected to pixel-by-pixel gamma transformation to obtain each of the target image data;

[0009] The target image data are fused to obtain target video data.

[0010] In one embodiment, the step of performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data includes:

[0011] The original image data of each frame in the original video data is respectively input into the denoising model to obtain the denoised image data corresponding to each of the original image data, wherein the denoising model is trained using clean image data as model training labels and using noisy image data obtained by adding noise to the clean image data as model input data, and the clean image data is image data with a noise content less than a preset threshold.

[0012] In one embodiment, the method further comprises:

[0013] Acquire multiple clean image data, and perform noise addition processing on each of the clean image data based on a pre-calibrated noise parameter to obtain each noisy image data;

[0014] Constructing a plurality of sample data respectively based on each of the clean image data and the noise image data corresponding to each of the clean image data;

[0015] A denoising data set is constructed based on the plurality of sample data, and an initial denoising model is trained based on the denoising data set to obtain the denoising model.

[0016] In one embodiment, the step of obtaining the noise-reduced image data corresponding to each of the original image data includes:

[0017] For any first image data in each of the original image data, down-sample the first image data by using the noise reduction model to obtain second image data with multiple resolutions;

[0018] The first image data and each of the second image data are subjected to denoising processing respectively through the denoising model to obtain third image data of multiple resolutions, and each of the third image data is subjected to upsampling processing to obtain denoised image data corresponding to the first image data.

[0019] In one embodiment, the step of performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data includes:

[0020] Down-sampling the original image data of each frame in the original video data to obtain fourth image data;

[0021] Inputting each of the fourth image data into an enhancement model respectively to obtain each of the fifth image data, wherein the enhancement model is trained by using a correctly exposed image taken in dim light as a model training label and using incorrectly exposed image data corresponding to the correctly exposed image as model input data;

[0022] Up-sampling is performed on each of the fifth image data to obtain enhanced image data corresponding to each of the original image data.

[0023] In one embodiment, the step of performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data to obtain each target image data comprises:

[0024] Determine a pixel-by-pixel gamma matrix for each of the enhanced image data;

[0025] Based on the pixel-by-pixel gamma matrices corresponding to the original image data, pixel-by-pixel gamma transformation is performed on the noise reduction image data corresponding to the original image data to obtain target image data.

[0026] In one embodiment, the step of fusing the target image data to obtain the target video data includes:

[0027] Inputting each of the target image data into a fusion network, fusing each of the target image data through the fusion network to obtain a video frame sequence;

[0028] The video frame sequence is converted into the RGB domain to obtain target video data.

[0029] In addition, to achieve the above-mentioned purpose, the present application also provides a video data processing device, the video data processing device comprising:

[0030] A processing module, configured to perform noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and to perform image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data;

[0031] A transformation module, configured to perform pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data, to obtain each target image data;

[0032] The fusion module is used to perform fusion processing on each of the target image data to obtain target video data.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a program for implementing the video data processing method is stored on the computer-readable storage medium. The program for implementing the video data processing method is executed by a processor to implement the steps of the video data processing method as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the video data processing method as described above are implemented.

[0035] The present application provides a video data processing method. The present application first performs noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performs image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data. Then, based on the enhanced image data corresponding to each original image data, pixel-by-pixel gamma transformation is performed on the noise-reduced image data corresponding to each original image data to obtain each target image data. Finally, each target image data is fused to obtain target video data.

[0036] In summary, the present application performs noise reduction and image enhancement processing on each frame of original image data in the original video data to reduce noise interference in the image and enhance features such as details and contrast of the image, and then performs pixel-by-pixel gamma transformation on the corresponding denoised image data based on the enhanced image data obtained after the image enhancement processing, so as to combine the respective advantageous features of the enhanced image data and the denoised image data to obtain the target image data, so that the target image data retains the clarity after noise reduction and has the rich details and contrast after enhancement, that is, the image quality of each frame of original image data is improved, and then the target image data corresponding to each frame of original image data are fused to obtain the denoised and enhanced video data. In this way, the video data obtained based on the fusion of the image data with improved quality also has a better video effect. In other words, the present application can improve the video quality of video data shot in low-illumination scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 This is a flow chart of the first embodiment of the video data processing method of the present application;

[0040] Figure 2 A schematic diagram of a signal conversion process involved in an embodiment of a video data processing method of the present application;

[0041] Figure 3 A schematic diagram of a noise reduction model processing flow involved in an embodiment of a video data processing method of the present application;

[0042] Figure 4 A schematic diagram of an enhanced model processing flow involved in an embodiment of a video data processing method of the present application;

[0043] Figure 5 A schematic diagram of the overall process of video data processing involved in an embodiment of the video data processing method of the present application;

[0044] Figure 6 This is a schematic diagram of the module structure of the video data processing device of the present application;

[0045] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the video data processing method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of the present application is: respectively perform noise reduction processing on the original image data of each frame in the original video data to obtain multiple frames of noise-reduced image data, and respectively perform image enhancement processing on the original image data of each frame in the original video data to obtain multiple frames of enhanced image data; respectively, based on the enhanced image data corresponding to each of the original image data, perform pixel-by-pixel gamma transformation on the noise-reduced image data corresponding to each of the original image data to obtain each target image data; perform fusion processing on each of the target image data to obtain target video data.

[0050] Currently, in low-light scenes, such as at night or in dimly lit indoor scenes, if you want to obtain video data with good video effects, you usually need to use high-cost hardware equipment to shoot the video. However, the video data shot by most shooting equipment in low-light scenes usually does not have good video effects, and may have quality problems such as poor clarity, contrast, and color saturation.

[0051] The present application performs noise reduction and image enhancement processing on each frame of original image data in the original video data to reduce noise interference in the image and enhance features such as details and contrast of the image, and then performs pixel-by-pixel gamma transformation on the corresponding denoised image data based on the enhanced image data obtained after the image enhancement processing to combine the respective advantageous features of the enhanced image data and the denoised image data to obtain target image data, so that the target image data retains the clarity after noise reduction and has the rich details and contrast after enhancement, that is, the image quality of each frame of original image data is improved, and then the target image data corresponding to each frame of original image data are fused to obtain denoised and enhanced video data. In this way, the video data obtained based on the fusion of the image data with improved quality also has better video effects. In other words, the present application can improve the video quality of video data shot in low-illumination scenes.

[0052] The present application combines the adjustment of the heat dissipation parameter value of the heat dissipation device with the adjustment of the coefficient value of the system power coefficient to achieve heat dissipation of the video data processing system by integrating multiple heat dissipation means, thereby overcoming the technical defect that the video data processing system cannot have a good heat dissipation effect due to a single heat dissipation means, and improving the heat dissipation efficiency of the video data processing system.

[0053] It should be noted that the execution subject of the method in each embodiment of the video data processing method of the present application can be a video data processing system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a video data processing device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the video data processing system as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0054] Based on this, the present application proposes a video data processing method of the first embodiment, please refer to Figure 1 , the video data processing method comprises steps S10 to S30:

[0055] Step S10, performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data;

[0056] It should be noted that the above original video data is video data captured in a low-light scene.

[0057] Perform noise reduction processing on each frame of image data in the original video data (hereinafter referred to as original image data for distinction) to obtain multiple frames of noise-reduced image data (hereinafter referred to as noise-reduced image data for distinction), and perform image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data (hereinafter referred to as enhanced image data for distinction). It should be understood that one original image data corresponds to one noise-reduced image data and one enhanced image data.

[0058] Step S20, performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data, to obtain each target image data;

[0059] Based on the enhanced image data corresponding to each of the original image data, the denoised image data corresponding to each of the original image data is subjected to pixel-by-pixel gamma transformation to obtain each transformed image data (hereinafter referred to as target image data for distinction). That is, for each denoised image data, a pixel-by-pixel gamma transformation is performed on it based on the enhanced image data corresponding to the denoised image data, so as to combine the clarity of the denoised image data after denoising and the enhanced image details, contrast and color saturation in the enhanced image data, so as to obtain target image data with higher quality.

[0060] In this embodiment, the step S20 may include:

[0061] Step S201, determining a pixel-by-pixel gamma matrix of each enhanced image data;

[0062] Step S202 , performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the pixel-by-pixel gamma matrix corresponding to each of the original image data, to obtain each target image data.

[0063] The pixel-by-pixel gamma matrix of each enhanced image data is determined, and then the denoised image data corresponding to each original image data is subjected to pixel-by-pixel gamma transformation based on the pixel-by-pixel gamma matrix corresponding to each original image data to obtain each target image data.

[0064] For example, for any enhanced image data, the pixel-by-pixel gamma matrix of the enhanced image data is represented as γ, and the noise reduction image data corresponding to the enhanced image data is represented as X, then the pixel-by-pixel gamma transformation formula can be expressed as: Y=1-(1-X) γ , where Y is the target image data obtained after transformation.

[0065] Step S30, performing fusion processing on each of the target image data to obtain target video data.

[0066] The target image data are fused to obtain processed video data (hereinafter referred to as target video data for distinction).

[0067] In this embodiment, the step S30 may include:

[0068] Step S301, inputting each of the target image data into a fusion network, and fusing each of the target image data through the fusion network to obtain a video frame sequence;

[0069] Step S302: convert the video frame sequence into RGB domain to obtain target video data.

[0070] It should be noted that the fusion network is used to fuse multiple frames of image data and also to convert raw domain image data into RGB domain.

[0071] After obtaining multiple frames of target image data, each target image data is input into a fusion network, through which each target image data is fused to obtain a video frame sequence, and then the video frame sequence is converted into video data that conforms to the RGB domain color standard, namely, the target video data.

[0072] Exemplarily, after receiving each frame of target image data, the fusion network extracts and fuses the image data of adjacent frames based on the arrangement order of the target image data to obtain a video frame sequence, and then converts the video frame sequence from the raw domain to the RGB domain to output target video data that meets the RGB domain color standard. It should be understood that the embodiment of the present application can achieve real-time acquisition of video data recorded by the camera, input it into the video data processing system frame by frame, perform noise reduction and enhancement respectively, obtain enhanced and noise-reduced raw data, convert it into video data in the RGB domain, and push the target video data to the SoC (System on Chip).

[0073] The embodiment of the present application performs noise reduction and image enhancement processing on each frame of original image data in the original video data to reduce noise interference in the image and enhance features such as details and contrast of the image. Then, based on the enhanced image data obtained after the image enhancement processing, the corresponding denoised image data is subjected to pixel-by-pixel gamma transformation to combine the respective advantageous features of the enhanced image data and the denoised image data to obtain target image data, so that the target image data retains the clarity after noise reduction and has the rich details and contrast after enhancement, that is, the image quality of each frame of original image data is improved. Then, the target image data corresponding to each frame of original image data are fused to obtain denoised and enhanced video data. In this way, the video data obtained based on the fusion of the image data with improved quality also has a better video effect. That is, the embodiment of the present application can improve the video quality of video data shot in low-illumination scenes.

[0074] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. On this basis, the step S10 may include:

[0075] Step S101, respectively inputting the original image data of each frame in the original video data into the denoising model to obtain the denoised image data corresponding to each of the original image data, wherein the denoising model is trained using clean image data as a model training label and using noisy image data obtained by denoising the clean image data as model input data, and the clean image data is image data with a noise content less than a preset threshold.

[0076] It should be noted that the denoising model is used to perform denoising on image data, and the denoising model is trained using clean image data as the model training label and image data obtained after adding noise to the clean image data (hereinafter referred to as noisy image data for distinction) as the model input data. Among them, a noise content threshold is set in advance, that is, the above-mentioned preset threshold. The embodiment of the present application does not limit the specific size of the preset threshold. The smaller the preset threshold, the less noise content in the clean image data. Clean image data refers to image data with a noise content less than the preset threshold. Noisy image data is image data after adding noise to the clean image data. In this way, it should be understood that the denoising model obtained by training the above-mentioned sample data can reduce the noise content in the input image data to below the preset threshold.

[0077] The original image data of each frame in the original video data is input into the noise reduction model respectively to obtain the noise reduction image data corresponding to each original image data.

[0078] In this embodiment, the video data processing method further includes:

[0079] Step A10, acquiring a plurality of clean image data, and performing noise addition processing on each of the clean image data based on a pre-calibrated noise parameter to obtain each noisy image data;

[0080] Step A20, constructing sample data based on each of the clean image data and the noise image data corresponding to each of the clean image data;

[0081] Step A30: construct a denoising data set based on the plurality of sample data, and train an initial denoising model based on the denoising data set to obtain the denoising model.

[0082] It should be noted that the pre-calibrated noise parameters include but are not limited to noise parameters in the noise model, black level errors and fixed pattern noise parameters.

[0083] Acquire multiple clean image data, and then perform noise processing on each clean image data based on pre-calibrated noise parameters to obtain each noise image data after noise addition. Then construct multiple sample data based on each clean image data and the noise image data corresponding to each clean image data, where it should be understood that each sample data includes a clean image data and the noise image data corresponding to the clean image data. Then construct a denoising data set based on the multiple sample data, and train the initial noise model based on the denoising data set until the corresponding loss function converges, and obtain a denoising model for denoising the image data.

[0084] For example, a camera is used to shoot a series of static high-quality raw image data using a low ISO (sensitivity) long exposure method, and the average of multiple frames of raw image data is calculated to obtain clean image data, and the clean image data is used as label data, and for each pixel point of the clean image, all pixels in the corresponding multiple frames of raw data are found, and then the variance is calculated, with the horizontal axis as the mean and the vertical axis as the variance fitting line, the slope of the line is k, and the intercept is σ2 variance. The k and σ parameters of the noise model are calibrated according to the above method at different ISO (sensitivity); and then the k and σ parameters at different ISOs are fitted respectively.

[0085] Specifically, a camera sensor that needs to be calibrated is used to shoot multiple frames of grayscale cards in a dark light laboratory according to the analog gain, and the noise parameters of the sensor are calibrated using the Poisson-Gaussian noise model. Figure 2 The figure shows a schematic diagram of the signal conversion process. The imaging process of the noise image data is as follows: based on the conversion efficiency α, photons u are photoelectrically converted to obtain electrons, and based on the dark noise n dConvert the electrons to voltage, then process the voltage based on the analog gain g to get the processed voltage, and then based on the read noise n r and quantization noise n q The processed voltage is converted from digital to analog to obtain the pixel value x. In this way, the entire imaging process introduces dark noise, read noise, quantization noise, and Poisson noise caused by the wave-particle duality of photons.

[0086] Among them, the calculation formula of the pixel value x obtained by fitting the Poisson Gaussian noise model is:

[0087]

[0088] Among them, P represents Poisson distribution, N represents normal distribution, u follows Poisson distribution, σ d and σ r point

[0089] Among them, x * is the theoretical pixel value, x is the output pixel value, and further we get:

[0090] E(x)=x *

[0091] Var(x)=kx * +σ 2

[0092] Among them, E(x) represents the pixel mean and Var(x) represents the pixel variance.

[0093] In addition to the Poisson and Gaussian noises mentioned above, there are also black level errors and fixed pattern noises that need to be calibrated, where fixed pattern noise will be generated both before and after analog gain. First, calibrate the black level error, capture a black frame under analog gain and possible exposure time, calculate the average as the black level error, fit the linear relationship between the black level error and the analog gain and exposure time, and complete the black level error calibration; capture multiple black frames at the same ISO, and add and average them as the sum of the black level error and fixed pattern noise. Since the black level error calibration has been completed, continue to fit the linear relationship between the fixed pattern noise and ISO to complete the calibration. When training the model, take the clean image data and the noisy image data obtained by adding noise to the clean image data based on k and σ under the corresponding ISO, as well as the black level error and fixed pattern noise to form a data pair for training.

[0094] In this way, the embodiment of the present application performs deep fitting on the noise model of the sensor to improve the noise reduction effect, and uses a noise reduction network in parallel to perform dark light enhancement, significantly reducing video processing delay.

[0095] In this embodiment, the step S101 may include:

[0096] Step S1011, for any first image data in each of the original image data, down-sample the first image data by using the noise reduction model to obtain second image data of multiple resolutions;

[0097] Step S1012: performing denoising processing on the first image data and each of the second image data respectively through the denoising model to obtain third image data of multiple resolutions, and upsampling processing on each of the third image data to obtain denoised image data corresponding to the first image data.

[0098] For any original image data among the original image data (hereinafter referred to as the first image data for distinction), the first image data is downsampled in multiple dimensions through the denoising model to obtain multiple image data with different resolutions (hereinafter referred to as the second image data for distinction), and then the first image data and each second image data are denoised by the denoising model to obtain multiple denoised image data (hereinafter referred to as the third image data for distinction), and each third image data has a different resolution, and then the multiple frames of third image data with different resolutions are upsampled to obtain denoised image data corresponding to the first image data.

[0099] It should be noted that the denoising model includes a multi-scale network structure for acquiring image information of different scales. Downsampling can reduce the image size to improve the signal-to-noise ratio of the image, while a high-resolution image size is conducive to retaining image details. Therefore, the embodiment of the present application performs multiple rounds of downsampling processing on the output original image data based on the multi-scale network structure to obtain image data of different sizes, and then performs denoising processing on the original image data and the image data obtained after downsampling to obtain image data of each size after denoising, and then performs upsampling processing on each denoised image data in the direction of size from small to large to obtain image data of original resolution, that is, denoised image data. For example, if Figure 3The figure shows a schematic diagram of the denoising model processing flow. For each frame of original image data, the denoising model downsamples the input original image data to obtain image a, then downsamples image a to obtain image b, then downsamples image b to obtain image c, and performs denoising on the image data of the four branches respectively to obtain the original denoised image, denoised image a, denoised image b and denoised image c, then upsamples denoised image c to denoised image b, upsamples denoised image b to denoised image a, and upsamples denoised image a to the original denoised image to obtain the denoised image data of the current original image data. In addition, it is worth emphasizing that the downsampling factor is adjusted in combination with the hardware computing power and denoising effect, and the downsampling factor is fixed during deployment.

[0100] In this embodiment, the step S10 may include:

[0101] Step S102, down-sampling each frame of original image data in the original video data to obtain fourth image data;

[0102] Step S103, respectively inputting each of the fourth image data into an enhancement model to obtain each of the fifth image data, wherein the enhancement model is trained using the correctly exposed image taken in dim light as a model training label and using the incorrectly exposed image data corresponding to the correctly exposed image as model input data;

[0103] Step S104: performing up-sampling processing on each of the fifth image data to obtain enhanced image data corresponding to each of the original image data.

[0104] It should be noted that the enhancement model is used to perform image enhancement processing on image data. This enhancement mode is obtained by training with the correctly exposed image taken in dim light as the model training label and the incorrectly exposed image data corresponding to the correctly exposed image as the model input data. Among them, the correctly exposed image refers to the image data with normal exposure taken in dim light, and the incorrectly exposed image corresponding to the correctly exposed image refers to the image data taken after the exposure is adjusted in dim light (overexposed or underexposed image data). Specifically, in one feasible implementation, images of different exposures (e.g., underexposure, normal exposure, and overexposure) are taken in dim light in advance and used to synthesize high-quality label images. Image synthesis is mainly based on contrast, brightness, color saturation, and objective quality evaluation indicators. The image is downsampled to different sizes at different multiples and the image area is cut out at a certain size. The contrast, brightness, color saturation, and objective quality evaluation indicators are calculated to obtain a reference score. The image and area with the highest score are used to synthesize label data as model training labels, and the image and area with the lowest score are used as model input data. An enhanced data set is constructed based on each model training label and the model input data corresponding to each model training label, so as to train the initial enhanced model based on the enhanced data set until the corresponding loss function converges to obtain an enhanced model for enhancing the image data.

[0105] First, down-sample the original image data of each frame in the original video data to obtain each processed image data (hereinafter referred to as the fourth image data for distinction), input each fourth image data into the enhancement model to obtain each enhanced image data (hereinafter referred to as the fifth image data for distinction), and finally up-sample each fifth image data to obtain the enhanced image data corresponding to each original image data.

[0106] Exemplarily, the enhancement model uses a structural re-parameterized convolution module to build a network structure. Since the input original image data has not been denoised, downsampling is beneficial to improving the signal-to-noise ratio of the image, and the brightness and contrast enhancement focuses on global and local information. Therefore, before the original image data is input into the network, the input original image data is downsampled, and no downsampling is performed during the enhancement network processing. The size of the feature map remains unchanged, and the final output feature map is upsampled to obtain a pixel-by-pixel γ matrix, and a gamma transform is performed on the denoised image. Specifically, if Figure 4 The figure shows a schematic diagram of the enhancement model processing flow. For any original image data, before the original image data is input into the enhancement model, the original image data is downsampled to obtain an image d with a smaller resolution, and then the image d is input into the enhancement model to obtain an enhanced image d after enhancement processing, and then the enhanced image d is upsampled to obtain enhanced image data corresponding to the current original image data.

[0107] In addition, if Figure 5 The figure shows a schematic diagram of the overall process of video data processing. First, the original video data is input into the denoising model and the enhancement model respectively. The denoising model is used to perform denoising on the original image data of each frame in the original video data to obtain multiple frames of denoised image data. The enhancement model is used to perform image enhancement on the original image data of each frame in the original video data to obtain enhanced image data. The pixel-by-pixel gamma matrix of each enhanced image data is further determined. Then, the denoised images are gamma transformed based on the pixel-by-pixel gamma transform formula to obtain multiple target image data. Then, the target image data are fused into target video data based on the fusion network.

[0108] In this way, the embodiment of the present application obtains a denoising model for denoising image data and an enhancement model for image enhancement processing by training the initial denoising model and the initial enhancement model respectively, so as to perform denoising and enhancement processing on each frame of original image data in the original video data to obtain each denoised image data and each enhanced image data, thereby providing a data basis for subsequently obtaining higher quality image data.

[0109] In addition, most of the current video data processing ISPs (Image Signal Processors) use traditional algorithms, and each module processes the data one by one, which is not enough for low-light scenes. In addition, there is also a way to use AI algorithms to replace a certain module in the ISP, but this AI algorithm can only complete a single task. For example, AI noise reduction can only be used for noise reduction, and there is a problem of low processing efficiency. The present application realizes the end-to-end ISP video processing flow through AI algorithm, including the raw domain data obtained from the sensor to the output RGB domain data for storage, display and transmission. First, the relevant parameters of the camera sensor are calibrated, and the training data set is synthesized, and then the neural network model is designed to realize the function of ISP. In this way, compared with the traditional ISP processing method and AI processing method, the present application proposes an end-to-end data set, namely the rawtorgb data set, and the noise parameters of the sensor are calibrated for data set synthesis to obtain a noise reduction data set; obtain image data of different exposures, use the fusion algorithm to build high-quality labels, and obtain an enhanced data set; build a neural network, conduct supervised training, and let the model learn the ISP processing flow, such as noise reduction, enhancement, de-mosaicing, color mapping, etc., so as to realize the entire ISP process using a neural network, build a neural network based on the performance of related hardware and conduct supervised training, and finally output a video with low noise, high clarity, good contrast and color. It should be understood that the present application optimizes the instructions of video data from the perspective of the algorithm, and obtains video data with good video effects while controlling costs.

[0110] The present application also provides a video data processing device. Figure 6 , the video data processing device comprises:

[0111] The processing module 10 is used to perform noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and to perform image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data;

[0112] The transformation module 20 is used to perform pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data, so as to obtain each target image data;

[0113] The fusion module 30 is used to perform fusion processing on each of the target image data to obtain target video data.

[0114] Optionally, the processing module 10 is further used for:

[0115] The original image data of each frame in the original video data is respectively input into the denoising model to obtain the denoised image data corresponding to each of the original image data, wherein the denoising model is trained using clean image data as model training labels and using noisy image data obtained by adding noise to the clean image data as model input data, and the clean image data is image data with a noise content less than a preset threshold.

[0116] Optionally, the video data processing device further includes a noise reduction data set construction module, and the noise reduction data set construction module is used to:

[0117] Acquire multiple clean image data, and perform noise addition processing on each of the clean image data based on a pre-calibrated noise parameter to obtain each noisy image data;

[0118] Constructing a plurality of sample data respectively based on each of the clean image data and the noise image data corresponding to each of the clean image data;

[0119] A denoising data set is constructed based on the plurality of sample data, and an initial denoising model is trained based on the denoising data set to obtain the denoising model.

[0120] Optionally, the processing module 10 is further used for:

[0121] For any first image data in each of the original image data, down-sample the first image data by using the noise reduction model to obtain second image data with multiple resolutions;

[0122] The first image data and each of the second image data are subjected to denoising processing respectively through the denoising model to obtain third image data of multiple resolutions, and each of the third image data is subjected to upsampling processing to obtain denoised image data corresponding to the first image data.

[0123] Optionally, the processing module 10 is further used for:

[0124] Down-sampling the original image data of each frame in the original video data to obtain fourth image data;

[0125] Inputting each of the fourth image data into an enhancement model respectively to obtain each of the fifth image data, wherein the enhancement model is trained by using a correctly exposed image taken in dim light as a model training label and using incorrectly exposed image data corresponding to the correctly exposed image as model input data;

[0126] Up-sampling is performed on each of the fifth image data to obtain enhanced image data corresponding to each of the original image data.

[0127] Optionally, the transformation module 20 is further used for:

[0128] Determine a pixel-by-pixel gamma matrix for each of the enhanced image data;

[0129] Based on the pixel-by-pixel gamma matrices corresponding to the original image data, pixel-by-pixel gamma transformation is performed on the noise reduction image data corresponding to the original image data to obtain target image data.

[0130] Optionally, the fusion module 30 is further used for:

[0131] Inputting each of the target image data into a fusion network, fusing each of the target image data through the fusion network to obtain a video frame sequence;

[0132] The video frame sequence is converted into the RGB domain to obtain target video data.

[0133] The video data processing device provided in the embodiment of the present application adopts the video data processing method in the above embodiment, which can solve the technical problem of how to improve the video quality of video data captured in low-light scenes. Compared with the prior art, the beneficial effects of the video data processing device provided in the embodiment of the present application are the same as the beneficial effects of the video data processing method provided in the above embodiment, and other technical features in the video data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0134] The present application provides a video data processing device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video data processing method in the above-mentioned embodiment 1.

[0135] Reference below Figure 7 , which shows a schematic diagram of the structure of a video data processing device suitable for implementing an embodiment of the present application. The video data processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The video data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0136] like Figure 7As shown, the video data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the video data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the video data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the video data processing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0137] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0138] The video data processing device provided by the present application adopts the video data processing method in the above embodiment, which can solve the technical problem of how to improve the video quality of video data captured in low-light scenes. Compared with the prior art, the beneficial effects of the video data processing device provided by the present application are the same as the beneficial effects of the video data processing method provided by the above embodiment, and other technical features in the video data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0139] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0140] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0141] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the video data processing method in the above-mentioned embodiment.

[0142] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0143] The computer-readable storage medium may be included in the video data processing device; or may exist independently without being assembled into the video data processing device.

[0144] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the video data processing device, the video data processing device: performs noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performs image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data; performs pixel-by-pixel gamma transformation on the noise-reduced image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data to obtain each target image data; and performs fusion processing on each of the target image data to obtain target video data.

[0145] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0148] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video data processing method, and can solve the technical problem of how to improve the video quality of video data captured in low-light scenes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the video data processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0149] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned video data processing method when executed by a processor.

[0150] The computer program product provided in this application can improve the video quality of video data captured in low-light scenes. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the video data processing method provided in the above embodiment, which will not be repeated here.

[0151] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A video data processing method, characterized in that: The video data processing method comprises: Performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data; Based on the enhanced image data corresponding to each of the original image data, respectively, the denoised image data corresponding to each of the original image data are subjected to pixel-by-pixel gamma transformation to obtain each of the target image data; The target image data are fused to obtain target video data.

2. The method according to claim 1, characterized in that The step of performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data comprises: The original image data of each frame in the original video data is respectively input into the denoising model to obtain the denoised image data corresponding to each of the original image data, wherein the denoising model is trained using clean image data as model training labels and using noisy image data obtained by denoising the clean image data as model input data, and the clean image data is image data with a noise content less than a preset threshold.

3. The method according to claim 2, characterized in that The method further comprises: Acquire multiple clean image data, and perform noise addition processing on each of the clean image data based on a pre-calibrated noise parameter to obtain each noisy image data; Constructing a plurality of sample data respectively based on each of the clean image data and the noise image data corresponding to each of the clean image data; A denoising data set is constructed based on the plurality of sample data, and an initial denoising model is trained based on the denoising data set to obtain the denoising model.

4. The method according to claim 2, characterized in that The step of obtaining the noise reduction image data corresponding to each of the original image data comprises: For any first image data in each of the original image data, down-sample the first image data by using the noise reduction model to obtain second image data with multiple resolutions; The first image data and each of the second image data are subjected to denoising processing respectively through the denoising model to obtain third image data of multiple resolutions, and each of the third image data is subjected to upsampling processing to obtain denoised image data corresponding to the first image data.

5. The method according to claim 1, characterized in that The step of performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data comprises: Down-sampling the original image data of each frame in the original video data to obtain fourth image data; Inputting each of the fourth image data into an enhancement model respectively to obtain each of the fifth image data, wherein the enhancement model is trained by using a correctly exposed image taken in dim light as a model training label and using incorrectly exposed image data corresponding to the correctly exposed image as model input data; Up-sampling is performed on each of the fifth image data to obtain enhanced image data corresponding to each of the original image data.

6. The method according to claim 1, characterized in that The step of performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data to obtain each target image data comprises: Determine a pixel-by-pixel gamma matrix for each of the enhanced image data; Based on the pixel-by-pixel gamma matrices corresponding to the original image data, pixel-by-pixel gamma transformation is performed on the noise reduction image data corresponding to the original image data to obtain target image data.

7. The method according to any one of claims 1 to 6, characterized in that The step of fusing the target image data to obtain the target video data comprises: Inputting each of the target image data into a fusion network, fusing each of the target image data through the fusion network to obtain a video frame sequence; The video frame sequence is converted into the RGB domain to obtain target video data.

8. A video data processing device, characterized in that: The video data processing device comprises: A processing module, configured to perform noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and to perform image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data; A transformation module, configured to perform pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data, to obtain each target image data; The fusion module is used to perform fusion processing on each of the target image data to obtain target video data.

9. A video data processing device, characterized in that: The video data processing device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video data processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video data processing method according to any one of claims 1 to 7 are implemented.

Images