A video deblurring processing method, device and electronic equipment

Abstract

The present disclosure relates to the technical field of video processing, and provides a video deblurring processing method and device and electronic equipment. The method comprises: acquiring a blurred first video; performing image deblurring processing on N first video frames of the first video respectively to obtain second video frames corresponding to the first video frames one by one; sorting N second video frames according to time sequence to obtain a second video; and performing video stabilization processing on the second video to obtain a clear third video. The effect of deblurring processing on a blurred video can be achieved, the complexity of video deblurring processing is reduced, and the video deblurring capability has strong generalization.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and more specifically, to a video deblurring method, apparatus, and electronic device. Background Technology

[0002] With the widespread use of digital imaging devices (such as smartphones, surveillance cameras, and drones), video has become a core medium for information transmission and scene recording. However, during video shooting, factors such as device shake (e.g., hand tremors during handheld shooting), rapid movement of the subject (e.g., a speeding vehicle), and insufficient ambient light can easily cause video images to become blurry (including motion blur and out-of-focus blur), severely reducing the visual quality and readability of the video. Therefore, video deblurring technology has become an important research direction in the field of computer vision. Its core objective is to eliminate blur interference in videos and restore clear images through algorithmic models.

[0003] Video deblurring models suffer from difficulties in obtaining samples and limited public datasets, resulting in a much smaller number of training samples compared to image deblurring models. This leads to problems such as weak model generalization ability, poor inter-frame temporal consistency, and insufficient detail restoration, ultimately making it difficult for video deblurring effects to meet the needs of practical applications. Summary of the Invention

[0004] One objective of this disclosure is to provide a new technical solution for video deblurring.

[0005] According to a first aspect of the present disclosure, a video deblurring method is provided, comprising:

[0006] Get the blurry first video;

[0007] The first video is deblurred for each of the N first video frames to obtain the second video frames that correspond one-to-one with the first video frames.

[0008] The second video is obtained by sorting the N second video frames according to their time sequence.

[0009] The second video is then stabilized to obtain a clear third video.

[0010] Optionally, the step of stabilizing the second video to obtain a clear third video includes:

[0011] Obtain the target video frame from the second video, perform image stabilization processing on the target video frame, and obtain the corresponding third video frame;

[0012] The N third video frames are sorted according to time sequence to obtain the third video.

[0013] Optionally, the target video frame is the i-th second video frame, where i is a positive integer less than or equal to NK and greater than K, and K is a positive integer less than N;

[0014] The step of performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes:

[0015] Based on the K preceding second video frames adjacent to the i-th second video frame and the K following second video frames adjacent to the i-th second video frame, image stabilization processing is performed on the i-th second video frame to obtain the i-th third video frame.

[0016] Optionally, the step of performing image stabilization processing on the i-th second video frame based on the K preceding second video frames adjacent to the i-th second video frame and the K following second video frames adjacent to the i-th second video frame to obtain the i-th third video frame includes:

[0017] Obtain the i-th first video frame of the first video, the K first video frames adjacent to the i-th first video frame, and the K last first video frames adjacent to the i-th first video frame;

[0018] The image stabilization process is performed on the i-th second video frame based on the i-th first video frame, the K first video frames adjacent to the i-th first video frame, the K last first video frames adjacent to the i-th first video frame, the K first second video frames adjacent to the i-th second video frame, and the K last second second video frames adjacent to the i-th second video frame to obtain the i-th third video frame.

[0019] Optionally, the target video frame is the j-th second video frame, where j is a positive integer less than or equal to K, or j is a positive integer greater than NK and less than N;

[0020] The step of performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes:

[0021] The j-th second video frame is taken as the j-th third video frame.

[0022] Optionally, the method further includes:

[0023] Obtain a first training sample set, which includes multiple first training samples, each of which includes a matching blurred image and a sharp image;

[0024] The image deblurring model is trained using the first training sample set, and the model parameters of the image deblurring model are updated by minimizing the loss between the output input and the input data of the image deblurring model as the training objective.

[0025] The image deblurring model is used to deblur video frames.

[0026] Optionally, the method further includes:

[0027] Obtain a second training sample set; the second training sample set includes multiple second training samples, each of which includes a matching blurred video and a clear video;

[0028] Based on the pre-trained image deblurring model, each blurred video frame of the blurred video of the second training sample is deblurred to obtain a deblurred video frame that corresponds one-to-one with the blurred video frame; the image deblurring model is used to perform image deblurring on the video frame.

[0029] The deblurred video frames corresponding to each second training sample are sorted according to time order to obtain the corresponding deblurred video.

[0030] Based on the deblurred video and clear video corresponding to the second training sample, a third training sample is generated;

[0031] The video stabilization model is trained using the third training sample, and the training objective is to minimize the loss between the output and input of the video stabilization model and the input data. The model parameters of the video stabilization model are then updated.

[0032] The video stabilization model is used to perform video stabilization processing on videos.

[0033] Optionally, the method further includes:

[0034] A fourth training sample is generated based on the first video and the third video;

[0035] The video deblurring model is trained using the fourth training sample, and the training objective is to minimize the loss between the output and input data of the video deblurring model. The model parameters of the video deblurring model are then updated.

[0036] The video deblurring model is used to perform video deblurring processing.

[0037] According to a second aspect of this disclosure, a video deblurring processing apparatus is provided, comprising:

[0038] The video acquisition module is used to acquire the blurry first video.

[0039] The image deblurring module is used to perform image deblurring on N first video frames of the first video to obtain second video frames that correspond one-to-one with the first video frames.

[0040] The video frame sorting module is used to sort N second video frames according to time order to obtain the second video.

[0041] The video stabilization module is used to stabilize the second video to obtain a clear third video.

[0042] According to a third aspect of this disclosure, an electronic device is provided, including a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the method as described in the first aspect of this disclosure under the control of the computer program.

[0043] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect of this disclosure.

[0044] Through the embodiments of this disclosure, the first video frames of the blurred first video are first deblurred to obtain second video frames that correspond one-to-one with the first video frames. Then, the second video frames are sorted according to the time sequence to obtain the second video. Finally, the second video is stabilized to obtain a clear third video. This achieves the effect of deblurring the blurred video, reduces the complexity of video deblurring, and has a strong generalizable video deblurring capability.

[0045] When using image deblurring models and video stabilization models to achieve video deblurring, the number of labeled samples for "blurred image - clear image" is much greater than the number of labeled samples for "blurred video - clear video". Therefore, it has a better video deblurring effect than using video deblurring models that are used to deblur the video.

[0046] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0048] Figure 1 This is a block diagram illustrating the hardware configuration of an electronic device that can implement embodiments of the present disclosure;

[0049] Figure 2This is a flowchart of a video deblurring method according to an embodiment of the present disclosure;

[0050] Figure 3 This is a schematic diagram of image stabilization processing according to an embodiment of the present disclosure;

[0051] Figure 4 This is a block diagram of a video deblurring processing apparatus according to an embodiment of the present disclosure;

[0052] Figure 5 This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0053] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0054] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0055] Techniques, methods, and apparatus known to those skilled in the art in the relevant field may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

[0056] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0057] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0058] <Hardware Configuration>

[0059] Figure 1 This is a block diagram illustrating the hardware configuration of an electronic device 1000 that can implement embodiments of the present disclosure.

[0060] Electronic device 1000 can be a portable computer, desktop computer, mobile phone, tablet computer, AR glasses, etc. For example... Figure 1As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc. The processor 1100 may be a CPU, a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, etc. The communication device 1400 may be capable of wired or wireless communication, specifically including Wi-Fi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, etc. The display device 1500 may be, for example, an LCD screen, a touch screen, etc. The input device 1600 may include, for example, a touch screen, a keyboard, motion input, etc. Users can input / output voice information through the speaker 1700 and the microphone 1800.

[0061] Figure 1 The electronic devices shown are merely illustrative and in no way intended to limit this disclosure, its application, or use. In embodiments applied to this disclosure, the memory 1200 of the electronic device 1000 is used to store instructions for controlling the processor 1100 to operate and perform any of the methods provided in the embodiments of this disclosure. Those skilled in the art will understand that, although... Figure 1 The electronic device 1000 is shown with multiple devices shown; however, this disclosure may relate only to some of these devices. For example, electronic device 1000 may only relate to processor 1100 and memory 1200. Those skilled in the art can design instructions based on the schemes disclosed herein. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0062] <Method Implementation>

[0063] This disclosure provides a video deblurring method, which can be implemented by an electronic device. Specifically, the electronic device can be as follows: Figure 1 The electronic device shown is 1000.

[0064] Figure 2 This is a flowchart of a video deblurring method according to an embodiment of the present disclosure.

[0065] like Figure 2 As shown, the video deblurring method includes the following steps S2100 to S2400:

[0066] Step S2100: Obtain the blurred first video.

[0067] In this embodiment, the blurry first video may be captured by an electronic device performing the method of this embodiment, or it may be obtained by the electronic device from other electronic devices.

[0068] Step S2200: Perform image deblurring on N first video frames of the first video to obtain second video frames that correspond one-to-one with the first video frames.

[0069] In this embodiment, the first video may include N first video frames. The N first video frames may be all the video frames of the first video, or they may be keyframes of the first video.

[0070] In this embodiment, image deblurring is performed on each of the N first video frames to obtain a second video frame corresponding to each first video frame, resulting in N second video frames. The timestamp of each second video frame is the same as the timestamp of its corresponding first video frame.

[0071] In some embodiments, image deblurring is performed on N first video frames of a first video to obtain a deblurred second video, including: traversing the N first video frames of the first video, performing image deblurring on the currently traversed first video frame, and obtaining the corresponding second video frame.

[0072] In this embodiment, the N first video frames of the first video can be traversed in ascending or descending order of time, or the key frames in the N first video frames can be traversed first, and then the non-key frames in the N first video frames can be traversed as needed.

[0073] In this embodiment, the image deblurring process can be performed on the first video frame being traversed, based on a pre-trained image deblurring model, to obtain the corresponding second video frame. The image deblurring model learns a large number of clear and blurred image pairs, allowing it to automatically learn the mapping relationship from blurred to clear images, thereby achieving image deblurring.

[0074] Specifically, the first video frame being traversed can be input into the image deblurring model, and the image deblurring model can output the second video frame corresponding to the first video frame being traversed.

[0075] In some embodiments, the method further includes: acquiring a first training sample set, the first training sample set including multiple first training samples, each first training sample including a matching blurred image and a clear image; training an image deblurring model using the first training sample set, taking the minimum loss between the output input of the image deblurring model and the input data as the training objective, and updating the model parameters of the image deblurring model; the image deblurring model is used to perform image deblurring processing on the first video frame of the first video frame.

[0076] In existing technologies, there are multiple large-scale public image deblurring datasets (such as GoPro Dataset, HIDEDataset, RealBlur Dataset, etc.) containing tens of thousands to hundreds of thousands of labeled samples of "blurred image - clear image", covering a variety of blur types (motion blur, Gaussian blur), scenes (indoor, outdoor, night scene) and resolutions, which can meet the training needs of different models.

[0077] Based on sufficient initial training samples, image deblurring models (such as DeblurGAN based on convolutional neural networks (CNN) and Restormer based on the Transformer architecture) can fully learn the mapping relationship between blurred features and clear features, and show excellent deblurring effect in practical applications: they can not only effectively eliminate the blur interference of single-frame images, but also restore image details (such as text and textures), and have strong generalization ability, which can adapt to the blur image processing needs in different scenarios.

[0078] This embodiment demonstrates how to train an image deblurring model with excellent deblurring performance.

[0079] Step S2300: Sort the N second video frames according to the time sequence to obtain the second video.

[0080] In an embodiment where the N first video frames can be all video frames of the first video, the N second video frames can be sorted according to time order, and the resulting video frame sequence can be used as the second video.

[0081] In an embodiment where the N first video frames are keyframes of the first video, the N second video frames and the non-keyframes in the first video can be sorted according to time order, and the resulting video frame sequence can be used as the second video.

[0082] Step S2400: Stabilize the second video to obtain a clear third video.

[0083] The second video is stabilized by means of physical compensation or algorithm correction to offset the image displacement caused by step S2200, and finally outputs a smooth and stable third video.

[0084] In some embodiments, performing video stabilization on the second video to obtain a high-quality third video includes: performing video stabilization on the second video based on a pre-trained video stabilization model to obtain the third video.

[0085] Specifically, the second video can be input into the video stabilization model, which will then output a smooth and stable third video.

[0086] Furthermore, both the first and second videos can be input into the video stabilization model, which will then output a smooth and stable third video.

[0087] In this embodiment, the video stabilization model can be a deep learning-based video stabilization model. Its principle includes training a convolutional neural network (CNN) using a dataset containing stable and unstable video pairs, enabling it to learn the mapping from blurry, jittery frames to sharp, stable frames. The video stabilization model can learn to output deformable meshes or dense deformable fields to correct jittery frames through deformation.

[0088] Furthermore, the video stabilization model can also fuse information from multiple frames, warp the feature codes of multiple adjacent frames to the virtual camera space, fuse the corresponding features and decode them, and generate the output frame by weighted averaging.

[0089] In some embodiments, performing video stabilization processing on a video stabilization model to obtain a high-quality third video includes: acquiring a target video frame from a second video, performing image stabilization processing on the target video frame to obtain a corresponding third video frame; and sorting the obtained N third video frames according to time order to obtain a third video.

[0090] In one embodiment, the N second video frames can be used as the target video frames sequentially, either in ascending or descending order of time.

[0091] In another embodiment, it can also involve traversing N second video frames, using the currently traversed second video frame as the target video frame. Specifically, it can traverse the N second video frames in ascending or descending chronological order, or it can prioritize traversing the Kth to NKth second video frames, then traversing the 1st to K-1th second video frames, and the N-K+1th to Nth second video frames.

[0092] In the embodiment where the target video frame is the i-th second video frame of the second video, i is a positive integer less than or equal to NK and greater than K, and K is a positive integer less than N. Then, performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes: performing image stabilization processing on the i-th second video frame based on the K preceding second video frames adjacent to the i-th second video frame and the K following second video frames adjacent to the i-th second video frame, to obtain the i-th third video frame.

[0093] In this embodiment, K can be preset according to the application scenario or specific needs. The larger K is, the better the video stabilization effect of the obtained third video. For example, K can be 1 or 2.

[0094] The first K second video frames adjacent to the i-th second video frame can include the iK-th second video frame to the (i-1)-th second video frame. The last K second video frames adjacent to the i-th second video frame can include the (i+1)-th second video frame to the (i+K)-th second video frame.

[0095] In this embodiment, by fusing the first K second video frames adjacent to the i-th second video frame and the last K second video frames adjacent to the i-th second video frame, the image stabilization effect of the i-th second video frame can be improved.

[0096] In the embodiment where the target video frame is the i-th second video frame of the second video, where i is a positive integer less than or equal to NK and greater than K, and K is a positive integer less than N, image stabilization processing is performed on the target video frame to obtain the corresponding third video frame. This includes: based on the i-th first video frame of the first video, the K first video frames adjacent to the i-th first video frame, the K last first video frames adjacent to the i-th first video frame, the K first second video frames adjacent to the i-th second video frame, and the K last second second video frames adjacent to the i-th second video frame, image stabilization processing is performed on the i-th second video frame to obtain the i-th third video frame. Specifically, it can be as follows: Figure 3 As shown.

[0097] In this embodiment, the first K first video frames adjacent to the i-th first video frame may include the iK-th to i-1-th first video frames. The last K first video frames adjacent to the i-th first video frame may include the i+1-th to i+K-th first video frames.

[0098] In this embodiment, by fusing the iKth first video frame to the (i+Kth)th first video frame to perform image stabilization processing on the ith second video frame, the image stabilization effect of the obtained third video can be improved.

[0099] In the embodiment where the target video frame is the j-th second video frame of the second video, j is a positive integer less than K, or j is a positive integer greater than NK and less than N; then, performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes: using the j-th second video frame as the j-th third video frame.

[0100] In this embodiment, for the first to the Kth second video frames and the N-K+1 to the Nth second video frames, there are no adjacent first K second video frames or adjacent last K second video frames, so the probability of image deviation is small. Therefore, the jth second video frame can be directly used as the jth third video frame, that is, the jth second video frame is not image stabilization processed.

[0101] Through the embodiments of this disclosure, the first video frames of the blurred first video are first deblurred to obtain second video frames that correspond one-to-one with the first video frames. Then, the second video frames are sorted according to the time sequence to obtain the second video. Finally, the second video is stabilized to obtain a clear third video. This achieves the effect of deblurring the blurred video, reduces the complexity of video deblurring, and has a strong generalizable video deblurring capability.

[0102] When using image deblurring models and video stabilization models to achieve video deblurring, the number of labeled samples for "blurred image - clear image" is much greater than the number of labeled samples for "blurred video - clear video". Therefore, it has a better video deblurring effect than using video deblurring models that are used to deblur the video.

[0103] In some embodiments, the method further includes a step of training a video stabilization model. Specifically, after training the image deblurring model, the image deblurring model can be frozen, and the video stabilization model can be trained. In this case, during the training of the video stabilization model, only the model parameters of the video stabilization model are adjusted, and the model parameters of the image deblurring model are not adjusted. Alternatively, after training the image deblurring model, both the image deblurring model and the video stabilization model can be trained simultaneously. In this case, during the simultaneous training of the image deblurring model and the video stabilization model, the model parameters of both the video stabilization model and the image deblurring model are adjusted.

[0104] In one embodiment, the step of training a video stabilization model may include: obtaining a second training sample set; the second training sample set includes multiple second training samples, each second training sample including a matching blurred video and a clear video; deblurring each blurred video frame of the blurred video in the second training sample based on a pre-trained image deblurring model, to obtain deblurred video frames corresponding one-to-one with the blurred video frames; the image deblurring model is used to perform image deblurring on the video frames; sorting the deblurred video frames corresponding to each second training sample according to time order to obtain the corresponding deblurred video; generating a third training sample based on the deblurred video and clear video corresponding to the second training sample; training the video stabilization model using the third training sample, taking the minimum loss between the output input and input data of the video stabilization model as the training objective, and updating the model parameters of the video stabilization model; wherein, the video stabilization model is used to perform video stabilization on the video.

[0105] In another embodiment, the step of training a video stabilization model may include: obtaining a second training sample set; the second training sample set includes multiple second training samples, each second training sample including a matching blurred video and a clear video; deblurring each blurred video frame of the blurred video in the second training sample based on a pre-trained image deblurring model, to obtain deblurred video frames corresponding one-to-one with the blurred video frames; the image deblurring model is used to perform image deblurring on the video frames; sorting the deblurred video frames corresponding to each second training sample according to time order to obtain the corresponding deblurred video; generating a third training sample based on the blurred video, deblurred video, and clear video corresponding to the second training sample; training the video stabilization model using the third training sample, taking the minimum loss between the output input and input data of the video stabilization model as the training objective, and updating the model parameters of the video stabilization model; wherein, the video stabilization model is used to perform video stabilization on the video.

[0106] In this embodiment, a third training sample is generated using the second training sample set and the trained image deblurring model. The video stabilization model is then trained based on the third training sample. This increases the number of samples used to train the video stabilization model, thereby enabling the trained video stabilization model to have strong generalization ability and better video stabilization effect. This, in turn, improves the video deblurring effect achieved by combining the image deblurring model and the video stabilization model.

[0107] In some embodiments, the method further includes: generating a fourth training sample based on the first video and the third video; training a video deblurring model using the fourth training sample, taking the minimum loss between the output input of the video deblurring model and the input data as the training objective, and updating the model parameters of the video deblurring model; wherein the video deblurring model is used to perform video deblurring processing on the video.

[0108] In this embodiment, the third video is used as the annotation of the first video, and a fourth training sample is generated based on the first video and the third video to distill and train the video deblurring model. This increases the number of training samples for the video deblurring model, thereby improving the video deblurring effect and enhancing the generalization of the video deblurring model.

[0109] <Device Embodiment>

[0110] This embodiment provides a video deblurring processing device, such as... Figure 4 As shown, the video deblurring processing device 4000 may include a video acquisition module 4100, an image deblurring processing module 4200, a video frame sorting module 4300, and a video stabilization processing module 4400.

[0111] The video acquisition module 4100 is used to acquire the blurry first video.

[0112] The image deblurring module 4200 is used to perform image deblurring on N first video frames of the first video to obtain second video frames that correspond one-to-one with the first video frames.

[0113] The video frame sorting module 4300 is used to sort N second video frames according to time order to obtain the second video.

[0114] The video stabilization module 4400 is used to perform video stabilization on the second video to obtain a clear third video.

[0115] In some embodiments, the step of performing video stabilization on the second video to obtain a clear third video includes:

[0116] Obtain the target video frame from the second video, perform image stabilization processing on the target video frame, and obtain the corresponding third video frame;

[0117] The N third video frames are sorted according to time sequence to obtain the third video.

[0118] In some embodiments, the target video frame is the i-th second video frame, where i is a positive integer less than or equal to NK and greater than K, and K is a positive integer less than N;

[0119] The step of performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes:

[0120] Based on the K preceding second video frames adjacent to the i-th second video frame and the K following second video frames adjacent to the i-th second video frame, image stabilization processing is performed on the i-th second video frame to obtain the i-th third video frame.

[0121] In some embodiments, the step of performing image stabilization processing on the i-th second video frame based on the K preceding second video frames adjacent to the i-th second video frame and the K following second video frames adjacent to the i-th second video frame to obtain the i-th third video frame includes:

[0122] Obtain the i-th first video frame of the first video, the K first video frames adjacent to the i-th first video frame, and the K last first video frames adjacent to the i-th first video frame;

[0123] The image stabilization process is performed on the i-th second video frame based on the i-th first video frame, the K first video frames adjacent to the i-th first video frame, the K last first video frames adjacent to the i-th first video frame, the K first second video frames adjacent to the i-th second video frame, and the K last second second video frames adjacent to the i-th second video frame to obtain the i-th third video frame.

[0124] In some embodiments, the target video frame is the j-th second video frame, where j is a positive integer less than or equal to K, or j is a positive integer greater than NK and less than N;

[0125] The step of performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes:

[0126] The j-th second video frame is taken as the j-th third video frame.

[0127] In some embodiments, the video deblurring processing apparatus 4000 further includes:

[0128] A module for obtaining a first training sample set, the first training sample set including multiple first training samples, each first training sample including a matching blurred image and a sharp image;

[0129] A module for training an image deblurring model using a first training sample set, taking the minimum loss between the output input and input data of the image deblurring model as the training objective, and updating the model parameters of the image deblurring model.

[0130] The image deblurring model is used to deblur video frames.

[0131] In some embodiments, the video deblurring processing apparatus 4000 further includes:

[0132] A module for acquiring a second training sample set; the second training sample set includes multiple second training samples, each of which includes a matching blurred video and a clear video;

[0133] This module is used to deblur each blurred video frame of the blurred video of the second training sample based on a pre-trained image deblurring model, to obtain a deblurred video frame that corresponds one-to-one with the blurred video frame; the image deblurring model is a module used to perform image deblurring on the video frame.

[0134] This module is used to sort the deblurred video frames corresponding to each second training sample according to the time sequence to obtain the corresponding deblurred video.

[0135] A module for generating a third training sample based on the deblurred video and clear video corresponding to the second training sample;

[0136] A module for training a video stabilization model using the third training sample, taking the minimum loss between the output input of the video stabilization model and the input data as the training objective, and updating the model parameters of the video stabilization model;

[0137] The video stabilization model is used to perform video stabilization processing on videos.

[0138] In some embodiments, the video deblurring processing apparatus 4000 further includes:

[0139] A module for generating a fourth training sample based on the first video and the third video;

[0140] A module for training a video deblurring model using the fourth training sample, taking the minimum loss between the output input of the video deblurring model and the input data as the training objective, and updating the model parameters of the video deblurring model;

[0141] The video deblurring model is used to perform video deblurring processing.

[0142] <Electronic Device Examples>

[0143] This embodiment provides an electronic device, which in one aspect may include the aforementioned video deblurring processing device 4000.

[0144] On the other hand, such as Figure 5 As shown, the electronic device 1000 may include a processor 1100 and a memory 1200. The memory 1200 is used to store computer programs, and the processor 1100 is used to control the electronic device to execute the methods of any embodiment of this disclosure under the control of the computer programs.

[0145] <Example of a readable storage medium>

[0146] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the methods described in any of the method embodiments of this disclosure.

[0147] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0148] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0149] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0150] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of the present invention.

[0151] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0152] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0153] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0155] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A video deblurring method, characterized in that, include: Get the blurry first video; The first video is deblurred for each of the N first video frames to obtain the second video frames that correspond one-to-one with the first video frames. The second video is obtained by sorting the N second video frames according to their time sequence. Based on a pre-trained video stabilization model, the second video is stabilized to obtain a clear third video; wherein, the video stabilization model is used to correct inter-frame jitter in the video. The step of performing image deblurring on N first video frames of the first video to obtain a deblurred second video includes: traversing the N first video frames of the first video; inputting the currently traversed first video frame into a pre-trained image deblurring model for image deblurring to obtain the corresponding second video frame; The step of stabilizing the second video to obtain a clear third video includes: Obtain the target video frame from the second video, perform image stabilization processing on the target video frame, and obtain the corresponding third video frame; The N third video frames are sorted according to time sequence to obtain the third video. The target video frame is the i-th second video frame, where i is a positive integer less than or equal to NK and greater than K, and K is a positive integer less than N; The step of performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes: Obtain the i-th first video frame of the first video, the K first video frames adjacent to the i-th first video frame, and the K last first video frames adjacent to the i-th first video frame; Based on the i-th first video frame, the K first video frames adjacent to the i-th first video frame, the K last first video frames adjacent to the i-th first video frame, the K first second video frames adjacent to the i-th second video frame, and the K last second second video frames adjacent to the i-th second video frame, image stabilization processing is performed on the i-th second video frame to obtain the i-th third video frame; The target video frame is the j-th second video frame, where j is a positive integer less than or equal to K, or j is a positive integer greater than NK and less than N; The step of performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes: The j-th second video frame is taken as the j-th third video frame.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a first training sample set, which includes multiple first training samples, each of which includes a matching blurred image and a sharp image; The image deblurring model is trained using the first training sample set, and the model parameters of the image deblurring model are updated by minimizing the loss between the output input and the input data of the image deblurring model as the training objective. The image deblurring model is used to deblur video frames.

3. The method according to claim 1, characterized in that, The method further includes: Obtain a second training sample set; the second training sample set includes multiple second training samples, each of which includes a matching blurred video and a clear video; Based on the pre-trained image deblurring model, each blurred video frame of the blurred video of the second training sample is deblurred to obtain a deblurred video frame that corresponds one-to-one with the blurred video frame; the image deblurring model is used to perform image deblurring on the video frame. The deblurred video frames corresponding to each second training sample are sorted according to time order to obtain the corresponding deblurred video. Based on the deblurred video and clear video corresponding to the second training sample, a third training sample is generated; The video stabilization model is trained using the third training sample, and the training objective is to minimize the loss between the output and input of the video stabilization model and the input data. The model parameters of the video stabilization model are then updated. The video stabilization model is used to perform video stabilization processing on videos.

4. The method according to claim 1, characterized in that, The method further includes: A fourth training sample is generated based on the first video and the third video; The video deblurring model is trained using the fourth training sample, and the training objective is to minimize the loss between the output and input data of the video deblurring model. The model parameters of the video deblurring model are then updated. The video deblurring model is used to perform video deblurring processing.

5. A video deblurring processing device, characterized in that, include: The video acquisition module is used to acquire the blurry first video. The image deblurring module is used to perform image deblurring on N first video frames of the first video to obtain second video frames that correspond one-to-one with the first video frames. The video frame sorting module is used to sort N second video frames according to time order to obtain the second video. The video stabilization module is used to perform video stabilization processing on the second video based on a pre-trained video stabilization model to obtain a clear third video; wherein, the video stabilization model is used to correct inter-frame jitter in the video; The step of performing image deblurring on N first video frames of the first video to obtain a deblurred second video includes: traversing the N first video frames of the first video; inputting the currently traversed first video frame into a pre-trained image deblurring model for image deblurring to obtain the corresponding second video frame; The step of stabilizing the second video to obtain a clear third video includes: Obtain the target video frame from the second video, perform image stabilization processing on the target video frame, and obtain the corresponding third video frame; The N third video frames are sorted according to time sequence to obtain the third video. The target video frame is the i-th second video frame, where i is a positive integer less than or equal to NK and greater than K, and K is a positive integer less than N; The step of performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes: Obtain the i-th first video frame of the first video, the K first video frames adjacent to the i-th first video frame, and the K last first video frames adjacent to the i-th first video frame; Based on the i-th first video frame, the K first video frames adjacent to the i-th first video frame, the K last first video frames adjacent to the i-th first video frame, the K first second video frames adjacent to the i-th second video frame, and the K last second second video frames adjacent to the i-th second video frame, image stabilization processing is performed on the i-th second video frame to obtain the i-th third video frame; The target video frame is the j-th second video frame, where j is a positive integer less than or equal to K, or j is a positive integer greater than NK and less than N; The step of performing image stabilization processing on the target video frame to obtain the corresponding third video frame includes: The j-th second video frame is taken as the j-th third video frame.

6. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, and the processor being used, under the control of the computer program, to execute the method as described in any one of claims 1 to 4.

Images