Video data processing method, device, equipment and computer-readable storage medium

Abstract

The present application discloses a video data processing method, apparatus, device, and computer-readable storage medium. The present application relates to the field of image processing technology. The method comprises: performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data; performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data; performing pixel-by-pixel gamma transformation on the noise-reduced image data corresponding to each frame of original image data based on the enhanced image data corresponding to each frame of original image data to obtain target image data; and performing fusion processing on each frame of target image data to obtain target video data. The present application can improve the video quality of video data captured in low-light scenes.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the continuous development and advancement of science and technology, video technology, as an important means of information transmission and recording, has been widely applied in various areas of daily life. In this context, users have increasingly higher requirements for the quality of video data and generally pursue a high-quality visual experience.

[0003] Currently, capturing high-quality video data in low-light scenarios, such as at night or in dimly lit indoor environments, typically requires expensive hardware. However, most video capture devices often produce suboptimal video data in low-light scenarios, potentially exhibiting poor clarity, contrast, and color saturation.

[0004] Therefore, how to improve the video quality of video data captured in low-light scenes is a problem that needs to be solved urgently. Summary of the Invention

[0005] The main purpose of this application is to provide a video data processing method, device, equipment and computer-readable storage medium, aiming to improve the video quality of video data captured in low-light scenes.

[0006] To achieve the above objectives, the present application provides a video data processing method, which includes:

[0007] Performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data;

[0008] performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data to obtain each target image data;

[0009] The target image data are fused to obtain target video data.

[0010] In one embodiment, the step of performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data includes:

[0011] The original image data of each frame in the original video data is input into the denoising model respectively to obtain the denoised image data corresponding to each of the original image data, wherein the denoising model is trained using clean image data as the model training label and using the noisy image data obtained by adding noise to the clean image data as the model input data, and the clean image data is image data with a noise content less than a preset threshold.

[0012] In one embodiment, the method further comprises:

[0013] Acquire multiple clean image data, and perform noise processing on each of the clean image data based on a pre-calibrated noise parameter to obtain each noisy image data;

[0014] constructing a plurality of sample data based on each of the clean image data and the noise image data corresponding to each of the clean image data;

[0015] A denoising data set is constructed based on the plurality of sample data, and an initial denoising model is trained based on the denoising data set to obtain the denoising model.

[0016] In one embodiment, the step of obtaining the noise-reduced image data corresponding to each of the original image data includes:

[0017] For any first image data in each of the original image data, down-sampling the first image data using the noise reduction model to obtain second image data of multiple resolutions;

[0018] The first image data and each of the second image data are subjected to denoising processing respectively by the denoising model to obtain third image data of multiple resolutions, and each of the third image data is subjected to upsampling processing to obtain denoised image data corresponding to the first image data.

[0019] In one embodiment, the step of performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data includes:

[0020] down-sampling the original image data of each frame in the original video data to obtain fourth image data;

[0021] Inputting each of the fourth image data into an enhancement model to obtain each of the fifth image data, wherein the enhancement model is trained using correctly exposed images taken in dim light as model training labels and incorrectly exposed image data corresponding to the correctly exposed images as model input data;

[0022] Up-sampling is performed on each of the fifth image data to obtain enhanced image data corresponding to each of the original image data.

[0023] In one embodiment, the step of performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data to obtain each target image data includes:

[0024] Determining a pixel-by-pixel gamma matrix for each of the enhanced image data;

[0025] Based on the pixel-by-pixel gamma matrix corresponding to each of the original image data, pixel-by-pixel gamma transformation is performed on the noise reduction image data corresponding to each of the original image data to obtain each target image data.

[0026] In one embodiment, the step of fusing the target image data to obtain target video data includes:

[0027] Inputting each of the target image data into a fusion network, and fusing each of the target image data through the fusion network to obtain a video frame sequence;

[0028] The video frame sequence is converted into RGB domain to obtain target video data.

[0029] In addition, to achieve the above-mentioned purpose, the present application further provides a video data processing device, the video data processing device comprising:

[0030] a processing module, configured to perform noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and to perform image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data;

[0031] a transformation module, configured to perform pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data, to obtain each target image data;

[0032] The fusion module is used to perform fusion processing on the target image data to obtain target video data.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores a program for implementing the video data processing method, and the program for implementing the video data processing method is executed by a processor to implement the steps of the video data processing method as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video data processing method when executed by a processor.

[0035] The present application provides a video data processing method, which first performs noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and then performs image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data. Then, based on the enhanced image data corresponding to each original image data, the noise-reduced image data corresponding to each original image data is subjected to pixel-by-pixel gamma transformation to obtain each target image data. Finally, each target image data is fused to obtain target video data.

[0036] In summary, the present application performs noise reduction and image enhancement processing on each frame of original image data in the original video data to reduce noise interference in the image and enhance features such as details and contrast of the image, and then performs pixel-by-pixel gamma transformation on the corresponding denoised image data based on the enhanced image data obtained after the image enhancement processing, so as to combine the respective advantageous features of the enhanced image data and the denoised image data to obtain the target image data, so that the target image data retains the clarity after noise reduction and has the rich details and contrast after enhancement, that is, the image quality of each frame of original image data is improved, and then the target image data corresponding to each frame of original image data are fused to obtain the denoised and enhanced video data. In this way, the video data obtained based on the fusion of the image data with improved quality also has a better video effect. In other words, the present application can improve the video quality of video data shot in low-light scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a flow chart of the first embodiment of the video data processing method of the present application;

[0040] Figure 2 This is a schematic diagram of a signal conversion process involved in an embodiment of the video data processing method of the present application;

[0041] Figure 3 Schematic diagram of a noise reduction model processing flow involved in an embodiment of the video data processing method of the present application;

[0042] Figure 4 This is a schematic diagram of an enhanced model processing flow involved in an embodiment of the video data processing method of the present application;

[0043] Figure 5 A schematic diagram of the overall flow of video data processing involved in an embodiment of the video data processing method of the present application;

[0044] Figure 6 This is a schematic diagram of the module structure of the video data processing device of this application;

[0045] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the video data processing method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of the present application is: respectively perform noise reduction processing on the original image data of each frame in the original video data to obtain multiple frames of noise-reduced image data, and respectively perform image enhancement processing on the original image data of each frame in the original video data to obtain multiple frames of enhanced image data; respectively perform pixel-by-pixel gamma transformation on the noise-reduced image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data to obtain each target image data; and perform fusion processing on each of the target image data to obtain target video data.

[0050] Currently, capturing high-quality video data in low-light scenarios, such as at night or in dimly lit indoor environments, typically requires expensive hardware. However, most video capture devices often produce suboptimal video data in low-light scenarios, potentially exhibiting poor clarity, contrast, and color saturation.

[0051] This application performs noise reduction and image enhancement processing on each frame of original image data in the original video data to reduce noise interference in the image and enhance features such as details and contrast of the image. Then, based on the enhanced image data obtained after the image enhancement processing, the corresponding noise reduction image data is subjected to pixel-by-pixel gamma transformation to combine the respective advantageous features of the enhanced image data and the noise reduction image data to obtain target image data, so that the target image data retains the clarity after noise reduction and has the rich details and contrast after enhancement, that is, the image quality of each frame of original image data is improved. Then, the target image data corresponding to each frame of original image data are fused to obtain noise-reduced and enhanced video data. In this way, the video data obtained based on the fusion of the image data with improved quality also has better video effects. In other words, this application can improve the video quality of video data shot in low-light scenes.

[0052] It should be noted that the execution entity of the methods in each embodiment of the video data processing method of this application can be a video data processing system, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or a video data processing device capable of implementing the above functions, etc. This embodiment does not specifically limit this. The following uses the video data processing system as the execution entity as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, this application proposes a video data processing method of the first embodiment, please refer to Figure 1 , the video data processing method includes steps S10 to S30:

[0054] Step S10, performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data;

[0055] It should be noted that the above original video data is video data shot in a low-light scene.

[0056] Noise reduction processing is performed on each frame of image data in the original video data (hereinafter referred to as original image data for distinction) to obtain multiple frames of noise-reduced image data (hereinafter referred to as noise-reduced image data for distinction), and image enhancement processing is performed on each frame of original image data in the original video data to obtain multiple frames of enhanced image data (hereinafter referred to as enhanced image data for distinction). It should be understood that one piece of original image data corresponds to one piece of noise-reduced image data and one piece of enhanced image data.

[0057] Step S20, performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data, to obtain each target image data;

[0058] Based on the enhanced image data corresponding to each of the original image data, the denoised image data corresponding to each of the original image data is subjected to a pixel-by-pixel gamma transform to obtain transformed image data (hereinafter referred to as target image data for distinction). That is, for each denoised image data, a pixel-by-pixel gamma transform is performed on it based on the enhanced image data corresponding to the denoised image data to combine the clarity of the denoised image data after denoising with the enhanced image details, contrast, and color saturation of the enhanced image data, thereby obtaining higher quality target image data.

[0059] In this embodiment, step S20 may include:

[0060] Step S201, determining a pixel-by-pixel gamma matrix of each enhanced image data;

[0061] Step S202 : performing pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the pixel-by-pixel gamma matrix corresponding to each of the original image data to obtain each target image data.

[0062] Determine the pixel-by-pixel gamma matrix of each enhanced image data, and then perform pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each original image data based on the pixel-by-pixel gamma matrix corresponding to each original image data to obtain each target image data.

[0063] For example, for any enhanced image data, the pixel-by-pixel gamma matrix of the enhanced image data is represented as γ, and the noise reduction image data corresponding to the enhanced image data is represented as X. Then, the pixel-by-pixel gamma transformation formula can be expressed as: Y = 1-(1-X) γ , where Y is the target image data obtained after transformation.

[0064] Step S30: performing fusion processing on the target image data to obtain target video data.

[0065] The target image data are fused to obtain processed video data (hereinafter referred to as target video data for distinction).

[0066] In this embodiment, step S30 may include:

[0067] Step S301: inputting each target image data into a fusion network, and fusing each target image data through the fusion network to obtain a video frame sequence;

[0068] Step S302: convert the video frame sequence into RGB domain to obtain target video data.

[0069] It should be noted that the fusion network is used to fuse multiple frames of image data and also to convert raw domain image data into RGB domain.

[0070] After obtaining multiple frames of target image data, each target image data is input into a fusion network, and each target image data is fused through the fusion network to obtain a video frame sequence, which is then converted into video data that conforms to the RGB domain color standard, namely the target video data.

[0071] Exemplarily, after receiving each frame of target image data, the fusion network extracts and fuses the image data of adjacent frames based on the order of the target image data to obtain a video frame sequence, and then converts the video frame sequence from the raw domain to the RGB domain to output target video data that meets the RGB domain color standard. It should be understood that the embodiment of the present application can obtain video data captured by the camera in real time, input it into the video data processing system frame by frame, perform noise reduction and enhancement respectively, obtain enhanced and noise-reduced raw data, convert it into RGB domain video data, and push the target video data to the SoC (System on Chip).

[0072] The embodiment of the present application performs noise reduction and image enhancement processing on each frame of original image data in the original video data to reduce noise interference in the image and enhance features such as details and contrast of the image. Then, based on the enhanced image data obtained after the image enhancement processing, the corresponding noise reduction image data is subjected to pixel-by-pixel gamma transformation to combine the respective advantageous features of the enhanced image data and the noise reduction image data to obtain target image data, so that the target image data retains the clarity after noise reduction and has the rich details and contrast after enhancement, that is, the image quality of each frame of original image data is improved. Then, the target image data corresponding to each frame of original image data are fused to obtain video data after noise reduction and enhancement. In this way, the video data obtained based on the fusion of the image data with improved quality also has better video effects. That is, the embodiment of the present application can improve the video quality of video data shot in low-light scenes.

[0073] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated hereafter. On this basis, the step S10 may include:

[0074] In step S101, the original image data of each frame in the original video data is input into the denoising model respectively to obtain the denoised image data corresponding to each of the original image data, wherein the denoising model is trained using clean image data as the model training label and using the noisy image data obtained by adding noise to the clean image data as the model input data, and the clean image data is image data with a noise content less than a preset threshold.

[0075] It should be noted that the denoising model is used to perform denoising on image data, and the denoising model is trained using clean image data as the model training label and the image data obtained after adding noise to the clean image data (hereinafter referred to as noise image data for distinction) as the model input data. Among them, a noise content threshold is set in advance, that is, the above-mentioned preset threshold. The embodiment of the present application does not limit the specific size of the preset threshold. The smaller the preset threshold, the less noise content in the clean image data. Clean image data refers to image data with a noise content less than the preset threshold. Noisy image data is image data after adding noise to the clean image data. In this way, it should be understood that the denoising model obtained by training the above-mentioned sample data can reduce the noise content in the input image data to below the preset threshold.

[0076] The original image data of each frame in the original video data is input into the noise reduction model respectively to obtain the noise reduction image data corresponding to each original image data.

[0077] In this embodiment, the video data processing method further includes:

[0078] Step A10, acquiring a plurality of clean image data, and performing noise processing on each of the clean image data based on a pre-calibrated noise parameter to obtain each noisy image data;

[0079] Step A20, constructing sample data based on each of the clean image data and the noise image data corresponding to each of the clean image data;

[0080] Step A30: constructing a denoising dataset based on the plurality of sample data, and training an initial denoising model based on the denoising dataset to obtain the denoising model.

[0081] It should be noted that the pre-calibrated noise parameters include but are not limited to noise parameters in the noise model, black level error and fixed pattern noise parameters.

[0082] Multiple clean image data sets are acquired, and then each clean image data set is subjected to noise processing based on pre-calibrated noise parameters to obtain each noisy image data set after noise processing. Multiple sample data sets are then constructed based on each clean image data set and the noise image data corresponding to each clean image data set. It should be understood that each sample data set includes a clean image data set and the noise image data corresponding to the clean image data set. A denoised dataset is then constructed based on the multiple sample data sets, and an initial noise model is trained based on the denoised dataset until the corresponding loss function converges, thereby obtaining a denoising model for denoising the image data.

[0083] For example, a camera is used to capture a series of high-quality, static raw image data using a low ISO (sensitivity) and long exposure. The multiple frames of raw image data are averaged to obtain clean image data, which is used as the label data. For each pixel in the clean image, all corresponding pixels in the multiple frames of raw data are found. The variance is then calculated, and a straight line is fitted with the mean on the horizontal axis and the variance on the vertical axis. The slope of the line is k, and the intercept is σ² variance. The k and σ parameters of the noise model are calibrated using the above method at different ISO (sensitivity) settings. The k and σ parameters at different ISO settings are then fitted separately.

[0084] Specifically, a camera sensor that needs to be calibrated is used to shoot multiple frames of grayscale cards in a dark light laboratory according to the simulated gain, and the noise parameters of the sensor are calibrated using the Poisson-Gaussian noise model. Figure 2 The figure shows the signal conversion process. The imaging process of noise image data is as follows: photon u is photoelectrically converted to electron based on the conversion efficiency α, and dark noise n is converted to electron based on the dark noise n. d The electrons are converted into voltage, and then the voltage is processed based on the analog gain g to obtain the processed voltage, and then based on the read noise n r and quantization noise n q The processed voltage is converted to digital-to-analog to obtain the pixel value x. Thus, the entire imaging process introduces dark noise, read noise, and quantization noise, as well as Poisson noise caused by the wave-particle duality of photons. The calculation formula for the pixel value x obtained by fitting the Poisson-Gaussian noise model is:

[0085] x=ɡ(αu+nd)+nr

[0086]

[0087] Among them, P represents Poisson distribution, N represents normal distribution, u obeys Poisson distribution, σ d and

[0088]

[0089] Among them, x * is the theoretical pixel value, x is the output pixel value, and we can further obtain:

[0090] E(x)=x *

[0091] Var(x)=kx * +σ 2

[0092] Among them, E(x) represents the pixel mean and Var(x) represents the pixel variance.

[0093] In addition to the Poisson and Gaussian noise mentioned above, black level error and fixed pattern noise also need to be calibrated. Fixed pattern noise occurs both before and after analog gain. First, calibrate the black level error. At the analog gain and possible exposure time, capture a black frame and calculate the average value as the black level error. Fit a linear relationship between the black level error, analog gain, and exposure time to complete the black level error calibration. At the same ISO, capture multiple black frames and add and average them as the sum of the black level error and fixed pattern noise. Since the black level error calibration has been completed, continue fitting the linear relationship between fixed pattern noise and ISO to complete the calibration. During model training, take clean image data and noisy image data obtained by adding noise to the clean image data based on k and σ at the corresponding ISO, as well as the black level error and fixed pattern noise, to form data pairs for training.

[0094] In this way, the embodiment of the present application performs deep fitting on the noise model of the sensor to improve the noise reduction effect, and uses a parallel approach of the noise reduction network to perform dark light enhancement, significantly reducing video processing delay.

[0095] In this embodiment, step S101 may include:

[0096] Step S1011: for any first image data in each of the original image data, down-sample the first image data using the noise reduction model to obtain second image data of multiple resolutions;

[0097] Step S1012: Perform noise reduction processing on the first image data and each of the second image data using the noise reduction model to obtain third image data of multiple resolutions, and upsample each of the third image data to obtain noise-reduced image data corresponding to the first image data.

[0098] For any original image data among the original image data (hereinafter referred to as the first image data for distinction), the first image data is downsampled in multiple dimensions through the denoising model to obtain multiple image data with different resolutions (hereinafter referred to as the second image data for distinction). Then, the first image data and each second image data are denoised respectively through the denoising model to obtain multiple denoised image data (hereinafter referred to as the third image data for distinction), and each third image data has a different resolution. Then, multiple frames of third image data with different resolutions are upsampled to obtain denoised image data corresponding to the first image data.

[0099] It should be noted that the denoising model includes a multi-scale network structure for acquiring image information of different scales. Downsampling can reduce the image size to improve the signal-to-noise ratio of the image, while high-resolution image size is conducive to retaining image details. Therefore, the embodiment of the present application performs multiple rounds of downsampling processing on the output original image data based on the multi-scale network structure to obtain image data of different sizes, and then performs denoising processing on the original image data and the image data obtained after downsampling to obtain image data of various sizes after denoising, and then performs upsampling processing on each denoised image data in the direction from small to large size to obtain image data of original resolution, that is, denoised image data. For example, if Figure 3 The figure shows a schematic diagram of the denoising model processing flow. For each frame of raw image data, the denoising model downsamples the input raw image data to obtain image a. It then downsamples image a to obtain image b, and then downsamples image b to obtain image c. Denoising is performed on the image data of the four branches, respectively, to obtain the original denoised image, denoised image a, denoised image b, and denoised image c. Denoised image c is then upsampled to denoised image b, denoised image b is upsampled to denoised image a, and denoised image a is upsampled to the original denoised image, to obtain the denoised image data of the current raw image data. It is also worth emphasizing that the downsampling factor is adjusted based on the hardware computing power and denoising effect, and is fixed during deployment.

[0100] In this embodiment, step S10 may include:

[0101] Step S102, performing downsampling processing on each frame of original image data in the original video data to obtain fourth image data;

[0102] Step S103: Inputting each of the fourth image data into an enhancement model to obtain each of the fifth image data, wherein the enhancement model is trained using the correctly exposed images taken in dim light as model training labels and the incorrectly exposed image data corresponding to the correctly exposed images as model input data;

[0103] Step S104 : performing up-sampling processing on each of the fifth image data to obtain enhanced image data corresponding to each of the original image data.

[0104] It should be noted that the enhancement model is used to perform image enhancement processing on image data. This enhancement mode is obtained by using correctly exposed images taken in low light as model training labels and the incorrectly exposed image data corresponding to the correctly exposed images as model input data for training. The correctly exposed images refer to image data with normal exposure taken in low light, while the incorrectly exposed images corresponding to the correctly exposed images refer to image data taken in low light after the exposure is adjusted (overexposed or underexposed image data). Specifically, in one feasible implementation, images of different exposures (e.g., underexposure, normal exposure, and overexposure) are taken in dark light in advance and used to synthesize high-quality label images. Image synthesis is mainly based on contrast, brightness, color saturation, and objective quality evaluation indicators. The image is downsampled to different sizes at different multiples and the image area is cropped at a certain size. The contrast, brightness, color saturation, and objective quality evaluation indicators are calculated to obtain a reference score. The image and area with the highest score are used to synthesize label data as model training labels, and the image and area with the lowest score are used as model input data. An enhanced data set is constructed based on each model training label and the model input data corresponding to each model training label, so as to train the initial enhancement model based on the enhanced data set until the corresponding loss function converges, thereby obtaining an enhanced model for enhancing image data.

[0105] First, the original image data of each frame in the original video data is down-sampled to obtain each processed image data (hereinafter referred to as the fourth image data for distinction), and each fourth image data is input into the enhancement model to obtain each enhanced image data (hereinafter referred to as the fifth image data for distinction). Finally, each fifth image data is up-sampled to obtain the enhanced image data corresponding to each original image data.

[0106] Exemplarily, the enhancement model uses a structural re-parameterized convolution module to build a network structure. Since the input original image data has not been denoised, downsampling is beneficial to improving the signal-to-noise ratio of the image, and the brightness and contrast enhancement focuses on global and local information. Therefore, before the original image data is input into the network, the input original image data is downsampled. During the enhancement network processing, no downsampling is performed, the feature map size remains unchanged, and the final output feature map is upsampled to obtain a pixel-by-pixel γ matrix, and a gamma transform is performed on the denoised image. Specifically, Figure 4The figure shows a schematic diagram of the enhancement model processing flow. For any original image data, before the original image data is input into the enhancement model, the original image data is downsampled to obtain an image d with a smaller resolution. The image d is then input into the enhancement model to obtain an enhanced image d after enhancement processing. The enhanced image d is then upsampled to obtain enhanced image data corresponding to the current original image data.

[0107] In addition, if Figure 5 The figure shows a schematic diagram of the overall process of video data processing. First, the original video data is input into the denoising model and the enhancement model respectively. The denoising model is used to perform denoising on the original image data of each frame in the original video data to obtain multiple frames of denoised image data. The enhancement model is used to perform image enhancement on the original image data of each frame in the original video data to obtain enhanced image data. The pixel-by-pixel gamma matrix of each enhanced image data is further determined. Then, the denoised images are gamma transformed based on the pixel-by-pixel gamma transform formula to obtain multiple target image data. Then, the target image data are fused into target video data based on the fusion network.

[0108] In this way, the embodiment of the present application obtains a denoising model for denoising image data and an enhancement model for performing image enhancement on image data by training the initial denoising model and the initial enhancement model respectively, so as to perform denoising and enhancement on each frame of original image data in the original video data to obtain denoised image data and enhanced image data, thereby providing a data basis for subsequently obtaining higher quality image data.

[0109] In addition, the current ISP (Image Signal Processor) for video data processing mostly uses traditional algorithms, and each module processes one by one, which is not enough for low-light scenes. In addition, there is also a way to use AI algorithms to replace a certain module in the ISP, but this AI algorithm can only complete a single task. For example, AI noise reduction can only be used for noise reduction, and there is a problem of low processing efficiency. The present application uses an AI algorithm to implement an end-to-end ISP video processing flow, including the process from raw domain data acquired by the sensor to outputting RGB domain data for storage, display, and transmission. First, the relevant parameters of the camera sensor are calibrated, a training data set is synthesized, and then a neural network model is designed to implement the ISP function. Thus, compared with traditional ISP processing methods and AI processing methods, the present application proposes an end-to-end data set, namely a rawtorgb data set, to calibrate the noise parameters of the sensor for data set synthesis to obtain a noise reduction data set; obtain image data of different exposures, use a fusion algorithm to construct high-quality labels, and obtain an enhanced data set; build a neural network and conduct supervised training to allow the model to learn the ISP processing flow, such as noise reduction, enhancement, demosaicing, color mapping, etc., so as to realize the use of a neural network to implement the entire ISP process. Based on the performance of the relevant hardware, a neural network is built and supervised training is conducted, and finally a video with low noise, high clarity, good contrast and color is output. It should be understood that the present application optimizes the instructions of the video data from the perspective of the algorithm, and obtains video data with good video effects while controlling costs.

[0110] The present application also provides a video data processing device, please refer to Figure 6 , the video data processing device includes:

[0111] The processing module 10 is configured to perform noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and to perform image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data;

[0112] a transform module 20 for performing pixel-by-pixel gamma transform on the noise reduction image data corresponding to each of the original image data based on the enhanced image data corresponding to each of the original image data, to obtain target image data;

[0113] The fusion module 30 is used to perform fusion processing on the target image data to obtain target video data.

[0114] Optionally, the processing module 10 is further configured to:

[0115] The original image data of each frame in the original video data is input into the denoising model respectively to obtain the denoised image data corresponding to each of the original image data, wherein the denoising model is trained using clean image data as the model training label and using the noisy image data obtained by adding noise to the clean image data as the model input data, and the clean image data is image data with a noise content less than a preset threshold.

[0116] Optionally, the video data processing apparatus further includes a noise reduction dataset construction module, wherein the noise reduction dataset construction module is configured to:

[0117] Acquire multiple clean image data, and perform noise processing on each of the clean image data based on a pre-calibrated noise parameter to obtain each noisy image data;

[0118] constructing a plurality of sample data based on each of the clean image data and the noise image data corresponding to each of the clean image data;

[0119] A denoising data set is constructed based on the plurality of sample data, and an initial denoising model is trained based on the denoising data set to obtain the denoising model.

[0120] Optionally, the processing module 10 is further configured to:

[0121] For any first image data in each of the original image data, down-sampling the first image data using the noise reduction model to obtain second image data of multiple resolutions;

[0122] The first image data and each of the second image data are subjected to denoising processing respectively by the denoising model to obtain third image data of multiple resolutions, and each of the third image data is subjected to upsampling processing to obtain denoised image data corresponding to the first image data.

[0123] Optionally, the processing module 10 is further configured to:

[0124] down-sampling the original image data of each frame in the original video data to obtain fourth image data;

[0125] Inputting each of the fourth image data into an enhancement model to obtain each of the fifth image data, wherein the enhancement model is trained using correctly exposed images taken in dim light as model training labels and incorrectly exposed image data corresponding to the correctly exposed images as model input data;

[0126] Up-sampling is performed on each of the fifth image data to obtain enhanced image data corresponding to each of the original image data.

[0127] Optionally, the transformation module 20 is further configured to:

[0128] Determining a pixel-by-pixel gamma matrix for each of the enhanced image data;

[0129] Based on the pixel-by-pixel gamma matrix corresponding to each of the original image data, pixel-by-pixel gamma transformation is performed on the noise reduction image data corresponding to each of the original image data to obtain each target image data.

[0130] Optionally, the fusion module 30 is further configured to:

[0131] Inputting each of the target image data into a fusion network, and fusing each of the target image data through the fusion network to obtain a video frame sequence;

[0132] The video frame sequence is converted into RGB domain to obtain target video data.

[0133] The video data processing device provided in the embodiments of the present application, employing the video data processing method of the above-described embodiments, can solve the technical problem of improving the video quality of video data captured in low-light scenarios. Compared to the prior art, the beneficial effects of the video data processing device provided in the embodiments of the present application are the same as those of the video data processing method provided in the above-described embodiments. Other technical features of the video data processing device are the same as those disclosed in the above-described embodiments and are not further described here.

[0134] The present application provides a video data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video data processing method in the above-mentioned embodiment 1.

[0135] Reference below Figure 7 , which shows a schematic diagram of the structure of a video data processing device suitable for implementing the embodiments of the present application. The video data processing device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7The video data processing device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0136] like Figure 7 As shown, the video data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the video data processing device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the video data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a video data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0137] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0138] The video data processing device provided in this application, employing the video data processing method in the above-described embodiment, can solve the technical problem of improving the video quality of video data captured in low-light scenarios. Compared to the prior art, the beneficial effects of the video data processing device provided in this application are the same as those of the video data processing method provided in the above-described embodiment. Other technical features of the video data processing device are the same as those disclosed in the above-described embodiment and are not further described here.

[0139] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0140] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0141] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the video data processing method in the above embodiment.

[0142] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0143] The computer-readable storage medium may be included in the video data processing device, or may exist independently without being incorporated into the video data processing device.

[0144] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the video data processing device, the video data processing device: performs noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performs image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data; performs pixel-by-pixel gamma transformation on the noise-reduced image data corresponding to each original image data based on the enhanced image data corresponding to each original image data to obtain each target image data; and performs fusion processing on each target image data to obtain target video data.

[0145] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0146] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0147] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0148] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described video data processing method, and can solve the technical problem of how to improve the video quality of video data captured in low-light scenes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the video data processing method provided in the above-described embodiment, and are not further elaborated here.

[0149] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned video data processing method when executed by a processor.

[0150] The computer program product provided in this application can improve the video quality of video data captured in low-light scenes. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as the beneficial effects of the video data processing method provided in the above embodiments, and will not be repeated here.

[0151] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A video data processing method, characterized in that: The video data processing method comprises: Performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data; Determining a pixel-by-pixel gamma matrix for each of the enhanced image data; Based on the pixel-by-pixel gamma matrix corresponding to each of the original image data, the noise reduction image data corresponding to each of the original image data is subjected to pixel-by-pixel gamma transformation to obtain each target image data, wherein the pixel-by-pixel gamma transformation formula is expressed as: Y = 1-(1-X) γ , X represents the denoised image data corresponding to the enhanced image data, γ represents the pixel-by-pixel gamma matrix of the enhanced image data, and Y represents the target image data; The target image data are fused to obtain target video data.

2. The method according to claim 1, wherein The step of performing noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data includes: The original image data of each frame in the original video data is input into the denoising model respectively to obtain the denoised image data corresponding to each of the original image data, wherein the denoising model is trained using clean image data as the model training label and using the noisy image data obtained by adding noise to the clean image data as the model input data, and the clean image data is image data with a noise content less than a preset threshold.

3. The method according to claim 2, wherein The method further comprises: Acquire multiple clean image data, and perform noise processing on each of the clean image data based on a pre-calibrated noise parameter to obtain each noisy image data; constructing a plurality of sample data based on each of the clean image data and the noise image data corresponding to each of the clean image data; A denoising data set is constructed based on the plurality of sample data, and an initial denoising model is trained based on the denoising data set to obtain the denoising model.

4. The method according to claim 2, wherein The step of obtaining the noise-reduced image data corresponding to each of the original image data comprises: For any first image data in each of the original image data, down-sampling the first image data using the noise reduction model to obtain second image data of multiple resolutions; The first image data and each of the second image data are subjected to denoising processing respectively by the denoising model to obtain third image data of multiple resolutions, and each of the third image data is subjected to upsampling processing to obtain denoised image data corresponding to the first image data.

5. The method according to claim 1, wherein The step of performing image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data includes: down-sampling the original image data of each frame in the original video data to obtain fourth image data; Inputting each of the fourth image data into an enhancement model to obtain each of the fifth image data, wherein the enhancement model is trained using correctly exposed images taken in dim light as model training labels and incorrectly exposed image data corresponding to the correctly exposed images as model input data; Up-sampling is performed on each of the fifth image data to obtain enhanced image data corresponding to each of the original image data.

6. The method according to any one of claims 1 to 5, characterized in that The step of fusing the target image data to obtain target video data includes: Inputting each of the target image data into a fusion network, and fusing each of the target image data through the fusion network to obtain a video frame sequence; The video frame sequence is converted into RGB domain to obtain target video data.

7. A video data processing device, characterized in that: The video data processing device comprises: a processing module, configured to perform noise reduction processing on each frame of original image data in the original video data to obtain multiple frames of noise-reduced image data, and to perform image enhancement processing on each frame of original image data in the original video data to obtain multiple frames of enhanced image data; The transformation module is configured to determine a pixel-by-pixel gamma matrix for each of the enhanced image data; and perform pixel-by-pixel gamma transformation on the noise reduction image data corresponding to each of the original image data based on the pixel-by-pixel gamma matrix corresponding to each of the original image data to obtain each target image data, wherein the pixel-by-pixel gamma transformation formula is expressed as: Y = 1-(1-X) γ , X represents the denoised image data corresponding to the enhanced image data, γ represents the pixel-by-pixel gamma matrix of the enhanced image data, and Y represents the target image data; The fusion module is used to perform fusion processing on the target image data to obtain target video data.

8. A video data processing device, characterized in that: The video data processing device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video data processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the video data processing method according to any one of claims 1 to 6 are implemented.

Images