Video processing method and device

Abstract

The invention discloses a video processing method and device, and relates to the technical field of video HDR. The method comprises the following steps: acquiring a to-be-processed HDR video; dividing an image in the HDR video to be processed according to a preset moment node to obtain a frame image at each moment; and inputting the frame image at each moment into the trained residual branch model for image processing to obtain a video processing result. Compared with the prior art, the method has the advantages that the short-frame image and the long-frame image are subjected to noise reduction and integration through the residual branch model, and the output of the previous branch is used as the input of the next branch, so that the robustness is higher, the noise reduction effect is ensured to the greatest extent, and the problem of ghosting in the fusion process is avoided.

Description

Technical Field

[0001] This application relates to the technical field of video HDR, and in particular, to a video processing method and apparatus. Background Art

[0002] Video HDR (High Dynamic Range) can expand the display brightness range, show more details in bright and dark areas, bring richer colors and more vivid and natural detail performance to the picture. Specifically, it includes the noise reduction and fusion tasks of different exposure frame images in each video frame. Traditional algorithms and existing AI algorithms (such as AHDRNet, ADNet) all adopt the alignment + fusion processing strategy, and using this processing strategy will inevitably bring the "ghosting" problem. Therefore, how to ensure the noise reduction ability of the model and the restoration of details, and at the same time be able to well process the fusion of moving objects to avoid the ghosting problem is a difficulty in video HDR. Summary of the Invention

[0003] In view of this, this application provides a video processing method and apparatus, and the main purpose is how to avoid the problem of ghosting while ensuring the noise reduction effect of the model and restoring the image details.

[0004] In a first aspect, this application provides a video processing method, including:

[0005] Obtain a to-be-processed HDR video;

[0006] Divide the images in the to-be-processed HDR video according to a preset time node to obtain frame images at each moment; the frame images include short frame images and long frame images;

[0007] Input the frame images at each moment into a trained residual branch model for image processing to obtain a video processing result;

[0008] Wherein, each branch in the residual branch model adopts a residual structure, and the branch at least includes a first processing branch for denoising the short frame image, a second processing branch for denoising the long frame image based on the processing result of the first processing branch, and a third processing branch for integrating the frame image based on the processing result of the second processing branch.

[0009] Optionally, in the case where the residual branch model includes three processing branches, the step of obtaining the trained residual branch model includes: obtaining a training HDR video; the training video includes training frame images divided into each moment; setting three processing branches for the initial model, and training the initial model based on the training frame images to obtain the trained residual branch model.

[0010] Optionally, the training frame images include short-frame images and long-frame images corresponding to each moment; training the initial model based on the training frame images to obtain the trained residual branch model includes: training a first processing branch through the short-frame images to obtain first branch weights; training a second processing branch according to the first branch weights and the long-frame images to obtain second branch weights; training a third processing branch according to the second branch weights to obtain third branch weights; and respectively allocating the first branch weights, the second branch weights, and the third branch weights to corresponding processing branches to obtain the trained residual branch model.

[0011] Optionally, the training a first processing branch through the short-frame images to obtain first branch weights includes: inputting the short-frame images into the first processing branch for noise reduction processing to obtain a first processing result; the first processing result includes a first image and the first branch weights; the training a second processing branch according to the first branch weights and the long-frame images to obtain second branch weights includes: fixing the first weights, inputting the first image and the long-frame images into the second processing branch for noise reduction processing to obtain a second processing result; the second processing result includes a second image and the second branch weights; the training a third processing branch according to the second branch weights to obtain third branch weights includes: fixing the second weights, inputting the second image and historical frame images into the third processing branch for noise reduction processing to obtain a third processing result; the third processing result includes a third image and the third branch weights, and the historical frame images are third images of frame images at the previous moment with the current moment as a reference; the third image at the current moment is a processing result of a frame image at the current moment obtained by fusing the first image and the second image with the short-frame image at the current moment as a reference frame.

[0012] Optionally, after obtaining the processing result of the frame image at the current moment, the method further includes: integrating the third images at each moment according to the third branch weights to obtain a video processing result of the training frame images.

[0013] Optionally, before training the first processing branch through the short-frame images, the method further includes: performing brightness alignment processing on the short-frame images by using an exposure ratio between the short-frame images and the long-frame images; and / or denoising the short-frame images by using a preset non-blind denoising method.

[0014] In a second aspect, the present application provides a video processing device, including:

[0015] An acquisition unit configured to acquire a to-be-processed HDR video;

[0016] A dividing unit, configured to divide the images in the to-be-processed HDR video according to preset time nodes, so as to obtain frame images at each moment; the frame images include short frame images and long frame images;

[0017] A processing unit, configured to input the frame images at each moment into a trained residual branch model for image processing to obtain a video processing result;

[0018] Wherein, each branch in the residual branch model adopts a residual structure, and the branch at least includes a first processing branch for denoising the short frame image, a second processing branch for denoising the long frame image based on the processing result of the first processing branch, and a third processing branch for integrating the frame images based on the processing result of the second processing branch.

[0019] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the video processing method described in the first aspect is implemented.

[0020] In a fourth aspect, the present application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. When the processor executes the computer program, the video processing method described in the first aspect is implemented.

[0021] In a fifth aspect, the present application provides a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send the signals to the processor, and the signals include computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is enabled to execute the video processing method described in the first aspect.

[0022] With the above technical solution, a video processing method and apparatus provided by the present application first obtain a to-be-processed HDR video. Then, at a preset time node, the images in the to-be-processed HDR video are divided to obtain frame images at each moment, where the frame images include short-frame images and long-frame images. Inputting the frame images at each moment into a trained residual branch model for image processing can obtain a video processing result. Among them, each branch in the residual branch model adopts a residual structure, and the branch at least includes a first processing branch for denoising the short-frame image, a second processing branch for denoising the long-frame image based on the processing result of the first processing branch, and a third processing branch for integrating the frame image based on the processing result of the second processing branch. Compared with the related art, by respectively denoising and integrating the short-frame and long-frame images through the residual branch model, and using the output of the previous branch as the input of the next branch, the robustness is stronger, thus ensuring the denoising effect to the greatest extent and avoiding the ghost problem during the fusion process.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0025] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0026] Figure 1 Shows a schematic flowchart of a video processing method provided by an embodiment of the present application;

[0027] Figure 2 Shows a schematic flowchart of the training steps of a residual branch model provided by an embodiment of the present application;

[0028] Figure 3 Shows a schematic diagram of the processing logic of a residual branch model provided by an embodiment of the present application;

[0029] Figure 4 Shows a schematic structural diagram of a video processing apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] Some embodiments of the present disclosure will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatuses, and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those set forth herein, but may be changed as will be apparent after understanding the present disclosure, except for operations that must be performed in a specific order. Additionally, descriptions of features known in the art may be omitted for the sake of clarity and conciseness.

[0031] The embodiments described in some embodiments of the present disclosure below do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] In order to improve the problem of avoiding ghosting while ensuring the noise reduction effect of the model and restoring image details. This embodiment provides a video processing method, as Figure 1 shown, the method includes:

[0033] S101. Obtain the HDR video to be processed.

[0034] An HDR video refers to a high dynamic range video, which uses a wider color and brightness range than ordinary videos and can present a more realistic picture effect in scenes with strong lighting contrast. It has the following characteristics: 1. Color and brightness range: The HDR video can capture and display a wider color and brightness range than ordinary videos, making the picture more vivid; 2. Detail display: The HDR video can display more details, including dark and highlight areas, making the picture clearer; 3. Realism: The HDR video can better reflect the visual effects in the real environment, making the audience feel a more real and vivid picture.

[0035] S102. Divide the images in the HDR video to be processed according to preset time nodes to obtain frame images at each moment.

[0036] Among them, the frame images include short frame images and long frame images.

[0037] "According to a preset time node" means that when processing a video, images in the video are extracted or processed according to a certain specific preset time point or time period. For example, the video is divided into images at multiple time points in seconds, and then further refined into frame images. A frame image refers to that in video processing, a video is usually decomposed into a series of frame images. Each frame is a static picture in the video, similar to a photo. After division, the images corresponding to each preset time point are extracted to form "frame images". These frame images represent the static state of the video at specific time points.

[0038] For HDR videos, it mainly includes noise reduction and fusion of frame images with different exposures. For each frame of video image in a dual-exposure system, it includes long-frame images and short-frame images with two different exposure parameters. The main differences between short-frame images and long-frame images lie in the exposure time and exposure gain. The main differences between long and short frames are in the exposure time (the exposure time of the long frame is t_l, and the exposure time of the short frame is t_s) and the exposure gain (the gain of the long frame is gain_l, and the gain of the short frame is gain_s). The long frame includes more dark information, while the short frame includes more bright details.

[0039] S103: Input the frame image of each moment into the trained residual branch model for image processing to obtain the video processing result.

[0040] Among them, each branch in the residual branch model adopts a residual structure. The branch at least includes a first processing branch for denoising the short-frame image, a second processing branch for denoising the long-frame image based on the processing result of the first processing branch, and a third processing branch for integrating the frame image based on the processing result of the second processing branch.

[0041] The Residual Network (ResNet) is a Convolutional Neural Network (CNN) structure in deep learning. It solves the problems of gradient disappearance and representation bottleneck in deep networks by introducing "residual blocks". The residual block allows the network to learn the residual between the input and the output, which helps the network better capture details and features in the image. The residual branch model refers to one or more residual blocks in the residual network. In the residual network, data usually passes through and is processed between multiple residual blocks. Each residual block receives the input data and outputs the processed data. These data may be directly passed to the next residual block, or combined with the input data in a certain form (such as addition) and then passed to the next residual block. The frame image of each moment is input into the trained residual branch model, and the model will perform a series of processing and analysis on these images. After being processed by the residual branch model, each frame image will obtain a corresponding result.

[0042] In this embodiment, first, an HDR video to be processed is obtained. Then, according to preset time nodes, the images in the HDR video to be processed are divided to obtain frame images at each moment, where the frame images include short-frame images and long-frame images. The frame images at each moment are input into a trained residual branch model for image processing, and the video processing result can be obtained. Among them, each branch in the residual branch model adopts a residual structure, and the branches at least include a first processing branch for denoising the short-frame images, a second processing branch for denoising the long-frame images based on the processing result of the first processing branch, and a third processing branch for integrating the frame images based on the processing result of the second processing branch. Compared with the related art, by using the residual branch model to denoise and integrate the short-frame and long-frame images respectively, and taking the output of the previous branch as the input of the next branch, the robustness is stronger, thus ensuring the denoising effect to the greatest extent and avoiding the ghosting problem during the fusion process.

[0043] Optionally, when the residual branch model includes three processing branches, the steps of obtaining the trained residual branch model include: obtaining a training HDR video; the training video includes training frame images that have been divided into each moment; setting three processing branches for the initial model, and training the initial model based on the training frame images to obtain the trained residual branch model.

[0044] In this embodiment, first, an HDR video for training is obtained. The HDR video to be trained is also divided into frame images at each moment according to preset time nodes. Essentially, a video is composed of a series of consecutive frame images. To train the model, it is usually necessary to split the video into individual frame images, and these frame images will be used as the input data of the model.

[0045] The residual branch model is a deep learning model structure that usually contains one or more "residual blocks", which allow the model to learn the residuals (differences) between the input and the output. In this embodiment, three processing branches are set in the initial model. The frame images previously split from the training HDR video are used as training data and input into the initial model, and the loss is calculated based on the model output and the expected output. After multiple iterative trainings, the model will gradually learn how to extract and transform information from the input frames to produce the expected output, thus becoming a trained residual branch model.

[0046] Optionally, the training frame images include short-frame images and long-frame images corresponding to each moment; based on the training frame images, the initial model is trained to obtain a trained residual branch model, including: training the first processing branch with the short-frame images to obtain the first branch weights; training the second processing branch according to the first branch weights and the long-frame images to obtain the second branch weights; training the third processing branch according to the second branch weights to obtain the third branch weights; and respectively assigning the first branch weights, the second branch weights, and the third branch weights to the corresponding processing branches to obtain the trained residual branch model.

[0047] In this embodiment, the training frame images include short-frame images and long-frame images corresponding to each moment. First, the frame images are used as inputs to train the initial model. The training process involves separate and sequential training of three processing branches. Finally, a trained residual branch model is obtained.

[0048] In the first stage of training, short-frame images are used as inputs to train the first processing branch. Through this process, this branch will learn how to extract features or information from the short-frame images. After training is completed, the weight parameters of this branch, that is, the first weight parameters, will be used for subsequent model inference and further training.

[0049] In the second stage, the weights of the first branch and the long-frame images are used to train the second processing branch. The output of the first branch is used as the input of the second branch. In this way, the second branch can further learn how to extract features or information from the long-frame images based on the learning results of the first branch.

[0050] In the third stage, the weights of the second branch are used to train the third processing branch. Similar to the second stage, the output of the second branch is used as the input of the third branch. In this way, the third branch can further extract and transform information based on the learning results of the previous two branches. The weight parameters of the three trained branches are respectively applied to their corresponding processing branches, thereby obtaining a complete trained residual branch model.

[0051] Optionally, the first processing branch is trained with short-frame images to obtain the first branch weights, including: inputting the short-frame images into the first processing branch for noise reduction processing to obtain a first processing result; the first processing result includes a first image and first branch weights; training the second processing branch with the first branch weights and long-frame images to obtain second branch weights, including: fixing the first weights, inputting the first image and the long-frame images into the second processing branch for noise reduction processing to obtain a second processing result; the second processing result includes a second image and second branch weights; training the third processing branch with the second branch weights to obtain third branch weights, including: fixing the second weights, inputting the second image and historical frame images into the third processing branch for noise reduction processing to obtain a third processing result; the third processing result includes a third image and third branch weights, and the historical frame images are the third images of the frame images at the previous moment based on the current moment; the third image at the current moment is the processing result of the frame image at the current moment obtained by fusing the first image and the second image with the short-frame image at the current moment as the reference frame.

[0052] In this embodiment, for the model training of the three processing branches, each branch performs noise reduction processing on different types of frame images and gradually optimizes its weights during the training process.

[0053] The specific details of the training of the first processing branch are as follows:

[0054] (1) Input: Short-frame images.

[0055] (2) Processing: Input the short-frame images into the first processing branch for noise reduction processing.

[0056] (3) Output: The first processing result, including the first image (the noise-reduced short-frame image) and the first branch weights (the weight parameters learned by this branch during training).

[0057] The specific details of the training of the second processing branch are as follows:

[0058] (1) Prerequisite: When training the second processing branch, the weights of the first processing branch are fixed.

[0059] (2) Input: The first image (i.e., the noise-reduced short-frame image output by the first processing branch) and long-frame images.

[0060] (3) Processing: Input the first image and the long-frame images into the second processing branch for noise reduction processing.

[0061] (4) Output: The second processing result, including the second image (the noise-reduced result) and the second branch weights.

[0062] The specific details of the training of the third processing branch are as follows:

[0063] (1) Prerequisite: When training the third processing branch, the weights of the first and second processing branches are fixed.

[0064] (2) Input: The second image (i.e., the denoised image output by the second processing branch) and the historical frame image. The historical frame image refers to the frame image at a certain moment before the current moment. Here, the historical frame image is the third image of the frame image at the previous moment (i.e., the output of the third processing branch at the previous moment).

[0065] (3) Processing: Input the second image and the historical frame image into the third processing branch for denoising processing.

[0066] (4) Output: The third processing result, including the third image (the denoised frame image at the current moment) and the third branch weight (the weight parameter learned by this branch during the training process). The "third image at the current moment" mentioned here is obtained by fusing the first image and the second image based on the short frame image.

[0067] Among them, non-blind denoising is adopted for the denoising processing in the model training of the three processing branches above, but the standard non-blind denoising transformation is not used (here the standard non-blind denoising refers to converting the noise distribution into a Gaussian distribution with a variance of 1), but the modified non-blind denoising is used. The modified non-blind denoising not only considers the original value of the image (multiplied by the exposure time and shot noise), but also takes into account the square term of the shot noise and the readout noise, and is adjusted using the square root and the proportional coefficient. This is actually an enhancement processing based on the image signal and the known noise characteristics, aiming to reduce the noise impact while retaining the image details.

[0068] By gradually combining different types of frame images, the model can learn the best method for denoising at different time scales, thereby improving the quality of the finally output image. At the same time, by fixing the weights of the previous branches, the model can continue to learn new features without destroying the already learned features.

[0069] Optionally, after obtaining the processing result of the frame image at the current moment, the method further includes: integrating the third image at each moment according to the third branch weight to obtain the video processing result of the training frame image.

[0070] In this embodiment, the function of the third processing branch is, in addition to continuing the noise reduction based on the output of the second processing branch, to integrate the noise-reduced images into the final video. When using the residual branch model to perform noise reduction or other image processing on each frame of the image, a series of processed frame images (referred to as the third images here) will be obtained. Just having these individual, processed frame images is not sufficient to form a complete and continuous video. In video processing, each frame of the image is an independent but continuous unit in the video. In order to obtain the processing result of the entire video sequence, these frame images need to be recombined in the time order in which they appear in the original video. Although the processing of each frame image depends on their respective inputs (short frame images, long frame images, and historical frame images) and the weights of the corresponding processing branches (first branch weight, second branch weight, and third branch weight), when integrating these frame images to form the video processing result, the weight of the third branch is mainly used. Using the weight of the third branch can ensure that the obtained video processing result not only has high quality for each frame image, but also is coherent and consistent in both time and space for the entire video sequence.

[0071] Optionally, before training the first processing branch with the short frame image, the method further includes: performing brightness alignment processing on the short frame image using the exposure ratio of the short frame image and the long frame image; and / or, performing denoising on the short frame image by a preset non-blind denoising method.

[0072] In this embodiment, the short frame is subjected to exposure alignment before input, that is, the brightness is matched with that of the long frame, and non-blind denoising is adopted in data processing. The exposure ratio is a parameter that describes the brightness difference between images under different exposure times. The long frame image has higher brightness, while the short frame image has lower brightness. Due to the brightness difference between the long frame and the short frame images, directly using them to train the model may lead to a decline in model performance because the model needs to process this unnecessary brightness change. Therefore, performing brightness alignment processing is to ensure that the brightness of the short frame image is similar to or matches that of the long frame image, ensuring the consistency of the image brightness under different exposure conditions, so that the model can focus more on learning other features of the image rather than the brightness difference.

[0073] Non-blind denoising is a common step in image processing, used to reduce or eliminate noise in images. Non-blind denoising means that the denoising process does not depend on the specific type or characteristics of the noise, but is based on some preset algorithms or methods to process. In this embodiment, instead of using the standard non-blind denoising transformation (where the standard non-blind denoising refers to transforming the noise distribution into a Gaussian distribution with a variance of 1), a modified non-blind denoising is used. The modified non-blind denoising not only considers the original value of the image (multiplied by the exposure time and shot noise), but also takes into account the square term of the shot noise and the read noise, and is adjusted using the square root and proportionality coefficient, making it more robust to the full scene. For example, the modified non-blind denoising transformation formula is as shown below:

[0074] img_vst = vst(img*expo,read*expo,shot*sqrt(expo))

[0075] = 2 / (shot*sqrt(expo))*sqrt(shot*sqrt(expo)*img*expo + 3 / 8*shot^2*expo + read*expo)

[0076] = 2 / shot*sqrt(shot*img*sqrt(expo)+3 / 8*shot^2+read)

[0077] In the formula, \(\text{img}\) represents the pixel value of the original short frame. \(\text{expo}\) is the exposure time or exposure compensation factor, used to adjust the brightness of the image. \(\text{shot}\) represents the shot noise in the photoelectric conversion process, which is related to the number of photons. \(\text{read}\) represents the read noise, which is the inherent electronic noise in the process of the sensor reading the signal.

[0078] The preprocessing step is to improve the efficiency and accuracy of subsequent model training. By performing brightness alignment and denoising on the short-frame images, unnecessary differences and interferences can be reduced, enabling the model to better focus on learning the key features of the images.

[0079] Furthermore, as Figure 2 shown, a schematic flow diagram of the training steps of a residual branch model provided in this embodiment is shown. The specific details are as follows:

[0080] S201, Obtain the training frame images of each moment that have been divided in the training HDR video.

[0081] Divide the images in the HDR video to be processed according to preset time nodes to obtain frame images for each moment. Among them, the frame images include short-frame images and long-frame images. The main differences between the short-frame and long-frame images lie in the exposure time (the exposure time of the long frame is \(t_l\), and the exposure time of the short frame is \(t_s\)) and the exposure gain (the gain of the long frame is \(gain_l\), and the gain of the short frame is \(gain_s\)).

[0082] S202. Use the exposure ratio of the short-frame image and the long-frame image to perform brightness alignment processing on the short-frame image.

[0083] Due to the differences in exposure time and exposure gain, the long frame includes more dark part information, while the short frame includes more bright part details. The brightness of the long frame and the short frame can be aligned through the exposure ratio. The short frame is subjected to exposure alignment before input, that is, the brightness is made consistent with that of the long frame. The exposure ratio is shown in the following formula:

[0084] expo = (gain_l * t_l) / (gain_s * t_s)

[0085] In the formula, \(\text{expo}\) is the exposure time or exposure compensation factor, which is used to adjust the brightness of the image.

[0086] S203. Denoise the short-frame image through a preset non-blind denoising method.

[0087] In data processing, non-blind denoising is adopted, but the standard non-blind denoising transformation (here, the standard non-blind denoising refers to converting the noise distribution into a Gaussian distribution with a variance of 1) is not used, but a modified non-blind denoising transformation formula is used, as shown in the following formula:

[0088] img_vst = vst(img * expo, read * expo, shot * sqrt(expo))

[0089] = 2 / (shot * sqrt(expo)) * sqrt(shot * sqrt(expo) * img * expo + 3 / 8 * shot^2 * expo + read * expo)

[0090] = 2 / shot * sqrt(shot * img * sqrt(expo) + 3 / 8 * shot^2 + read)

[0091] In the formula, \(\text{img}\) represents the pixel values of the original short frame. \(\text{expo}\) is the exposure time or exposure compensation factor, which is used to adjust the brightness of the image. \(\text{shot}\) represents the shot noise in the photoelectric conversion process, which is related to the number of photons. \(\text{read}\) represents the read noise, which is the inherent electronic noise in the process of the sensor reading the signal.

[0092] S204. Input the short frame image at the current moment into the first processing branch for noise reduction processing to obtain the first image and the first branch weight.

[0093] S205. Fix the first weight, and input the first image and the long frame image into the second processing branch for noise reduction processing to obtain the second image and the second branch weight.

[0094] S206. Fix the second weight, and input the second image and the historical frame image into the third processing branch for noise reduction processing to obtain the third image and the third branch weight.

[0095] S207. Integrate the third image at each moment according to the third branch weight to obtain the video processing result of the training frame image.

[0096] S204 to S207 are to set three processing branches for the initial model under the residual branch model, and train the initial model based on the training frame image to obtain the trained residual branch model. The training process involves the separate and sequential training of the three processing branches, and the specific details are as Figure 3 , which shows a schematic diagram of the processing logic of a residual branch model provided in this embodiment. The specific details are as follows:

[0097] Taking the short frame as the reference frame for each frame output of the video, the model architecture adopts the 3unet architecture, that is, it includes the first processing branch Short_unet, the second processing branch Long_unet, and the third processing branch Time_unet.

[0098] In terms of training, 3-stage decoupled training is adopted, that is, in the first stage, Short_unet short frame noise reduction is trained;

[0099] In the second stage of training, the weight of Short_unet is fixed, the weight of Long_unet is updated, and the input of Long_unet is the output of Short_unet and the original long frame image;

[0100] During the three-stage training, fix the weights of Short_unet and Long_unet, and update the weights of Time_unet. The input of Time_unet is the output of Long_unet and the historical frames (the historical frames are the outputs of the previous frame of Time_uent), and the output of Time_unet is the final output of the current frame.

[0101] Each unet can also use other network structures, such as transform, etc., as long as it has the ability of feature extraction and non-linear mapping. Each unet adopts a residual structure, that is, the original short frame plus three learned residuals to obtain the final output.

[0102] The inputs of the three unets include the original short frame, the original long frame, and the historical frames, which together constitute the complete information that can be obtained for processing the current frame.

[0103] Further, as Figures 1 to 3 a specific implementation of the method shown, this embodiment provides a video processing device, as Figure 4 shown, the device includes: an acquisition unit 41, a division unit 42, and a processing unit 43.

[0104] The acquisition unit 41 is configured to acquire the HDR video to be processed;

[0105] The division unit 42 is configured to divide the images in the HDR video to be processed according to preset time nodes to obtain frame images at each moment; the frame images include short frame images and long frame images;

[0106] The processing unit 43 is configured to input the frame images at each moment into the trained residual branch model for image processing to obtain a video processing result;

[0107] Wherein, each branch in the residual branch model adopts a residual structure, and the branch at least includes a first processing branch for denoising the short frame image, a second processing branch for denoising the long frame image based on the processing result of the first processing branch, and a third processing branch for integrating the frame image based on the processing result of the second processing branch.

[0108] In a specific application scenario, the acquisition unit 41 is specifically configured to acquire a training HDR video; the training video includes training frame images that have been divided into each moment; set three processing branches for the initial model, and based on the training frame images, train the initial model to obtain the trained residual branch model.

[0109] In a specific application scenario, the processing unit 43 is further specifically configured to train a first processing branch through the short-frame image to obtain a first branch weight; train a second processing branch according to the first branch weight and the long-frame image to obtain a second branch weight; train a third processing branch according to the second branch weight to obtain a third branch weight; and allocate the first branch weight, the second branch weight, and the third branch weight to the corresponding processing branches respectively to obtain the trained residual branch model.

[0110] In a specific application scenario, the processing unit 43 is further specifically configured to input the short-frame image into a first processing branch for noise reduction processing to obtain a first processing result; the first processing result includes a first image and the first branch weight;

[0111] The training of the second processing branch according to the first branch weight and the long-frame image to obtain a second branch weight includes: fixing the first weight, inputting the first image and the long-frame image into the second processing branch for noise reduction processing to obtain a second processing result; the second processing result includes a second image and the second branch weight;

[0112] The training of the third processing branch according to the second branch weight to obtain a third branch weight includes: fixing the second weight, inputting the second image and a historical frame image into the third processing branch for noise reduction processing to obtain a third processing result; the third processing result includes a third image and the third branch weight, and the historical frame image is the third image of the frame image at the previous moment based on the current moment;

[0113] The third image at the current moment takes the short-frame image at the current moment as a reference frame, and the first image and the second image are fused with the short-frame image to obtain the processing result of the frame image at the current moment.

[0114] In a specific application scenario, the processing unit 43 is further specifically configured to integrate the third image at each moment according to the third branch weight to obtain the video processing result of the training frame image.

[0115] In a specific application scenario, the processing unit 43 is further specifically configured to perform brightness alignment processing on the short-frame image by using the exposure ratio of the short-frame image and the long-frame image; and / or perform denoising on the short-frame image by a preset non-blind denoising method.

[0116] It should be noted that for other corresponding descriptions of each functional unit involved in the video processing method provided in this embodiment, reference can be made to the corresponding description in Figures 1 to 3 and details are not described herein again.

[0117] Based on the above as Figures 1 to 3 shown in the method, correspondingly, this embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method shown above as Figures 1 to 3 shown is implemented.

[0118] Based on such an understanding, the technical solution of this application can be embodied in the form of a software product, and the software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of this application.

[0119] Based on the above as Figures 1 to 3 shown in the method, and Figure 4 shown in the virtual device embodiment, in order to achieve the above object, this embodiment of the application also provides an electronic device, such as intelligent terminals such as smart phones, tablet computers, drones, intelligent robots, etc., and the device includes a storage medium and a processor; the storage medium is used for storing a computer program; the processor is used for executing the computer program to implement the method shown above as Figures 1 to 3 shown.

[0120] Optionally, the above-mentioned physical device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, sensors, an audio circuit, a WI-FI module, and so on. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc., and optionally the user interface may further include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0121] Those skilled in the art can understand that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine some components, or have different component arrangements.

[0122] The storage medium may further include an operating system and a network communication module. The operating system is a program for managing the hardware and software resources of the above-mentioned physical device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to implement the communication between the components inside the storage medium, as well as the communication between other hardware and software in the information processing physical device.

[0123] Based on the above as Figures 1 to 3 shown in the method, and Figure 4The virtual device embodiment shown. This embodiment also provides a chip, including one or more interface circuits and one or more processors; the interface circuit is configured to receive signals from the memory of the electronic device and send the signals to the processor, and the signals include computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is caused to execute the method as described above such as Figures 1 to 3 shown.

[0124] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or can also be implemented by hardware. By applying the solution of this embodiment, compared with the related art, in this embodiment, the residual branch model is used to perform noise reduction and integration on short-frame and long-frame images respectively, and the output of the previous branch is used as the input of the next branch, with stronger robustness, thus ensuring the noise reduction effect to the greatest extent and avoiding the problem of ghosting during the fusion process.

[0125] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the said element.

[0126] The above are only specific embodiments of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments described herein, but will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A video processing method, characterized in that, Including: Obtain the HDR video to be processed; Divide the images in the HDR video to be processed according to preset time nodes to obtain frame images for each moment; The frame images include short frame images and long frame images; Input the frame images for each moment into the trained residual branch model for image processing to obtain a video processing result; Wherein, each branch in the residual branch model adopts a residual structure, and the branch at least includes a first processing branch for denoising the short frame image, a second processing branch for denoising the long frame image based on the processing result of the first processing branch, and a third processing branch for integrating the frame image based on the processing result of the second processing branch.

2. The method according to claim 1, characterized in that When the residual branch model includes three processing branches, the steps of obtaining the trained residual branch model include: Obtain a training HDR video; the training video includes training frame images that have been divided into each moment; Set three processing branches for the initial model, and train the initial model based on the training frame images to obtain the trained residual branch model.

3. The method according to claim 2, characterized in that, The training frame images include short frame images and long frame images corresponding to each moment; Training the initial model based on the training frame images to obtain the trained residual branch model includes: Train the first processing branch through the short frame image to obtain the first branch weight; Train the second processing branch according to the first branch weight and the long frame image to obtain the second branch weight; Train the third processing branch according to the second branch weight to obtain the third branch weight; Assign the first branch weight, the second branch weight, and the third branch weight to the corresponding processing branches respectively to obtain the trained residual branch model.

4. The method according to claim 3, characterized in that The training the first processing branch through the short frame image to obtain the first branch weight includes: Input the short frame image into the first processing branch for noise reduction processing to obtain a first processing result; the first processing result includes a first image and the first branch weight; The training the second processing branch according to the first branch weight and the long frame image to obtain the second branch weight includes: Fix the first weight, input the first image and the long frame image into the second processing branch for noise reduction processing to obtain a second processing result; the second processing result includes a second image and the second branch weight; The training the third processing branch according to the second branch weight to obtain the third branch weight includes: Fix the second weight, input the second image and the historical frame image into the third processing branch for noise reduction processing to obtain a third processing result; the third processing result includes a third image and the third branch weight, and the historical frame image is the third image of the frame image at the previous moment based on the current moment; The third image at the current moment is based on the short frame image at the current moment as a reference frame, and the first image and the second image are fused with the short frame image to obtain the processing result of the frame image at the current moment.

5. The method according to claim 4, characterized in that After obtaining the processing result of the frame image at the current moment, the method further includes: Integrating the third image at each moment according to the third branch weight to obtain the video processing result of the training frame image.

6. The method according to claim 3, characterized in that, Before training the first processing branch with the short frame image, the method further includes: Performing brightness alignment processing on the short frame image by using the exposure ratio between the short frame image and the long frame image; and / or Denosing the short frame image by a preset non-blind denoising method.

7. A video processing device, characterized in that, It includes: An acquisition unit configured to acquire a to-be-processed HDR video; A partitioning unit configured to partition the images in the to-be-processed HDR video according to preset time nodes to obtain frame images at each moment; the frame images include short frame images and long frame images; A processing unit configured to input the frame images at each moment into a trained residual branch model for image processing to obtain a video processing result; Wherein, each branch in the residual branch model adopts a residual structure, and the branches at least include a first processing branch for denoising the short frame image, a second processing branch for denoising the long frame image based on the processing result of the first processing branch, and a third processing branch for integrating the frame images based on the processing result of the second processing branch.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 6.

9. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 6.

10. A chip, characterized in that, It includes one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, and the signal includes computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is caused to execute the method according to any one of claims 1 to 6.

Images