Digital video super-resolution processing method, system, equipment and medium

Abstract

The invention discloses a digital video super-resolution processing method, system and device and a medium. The method comprises the following steps: decomposing a video file into independent image frames; calculating a threshold value for selecting the key frame; dividing the image frame into a key frame and a non-key frame based on a threshold value; carrying out super-resolution processing on the key frame by adopting a RealBaicVSR technology to improve the resolution of the key frame; calculating the motion displacement between the non-key frame and the previous adjacent nearest key frame by adopting a feature-oriented variational optical flow method; projecting the nearest key frame before the current non-key frame to the current non-key frame based on the motion displacement; repairing an image obtained by projection by adopting a fast advancing repairing method to obtain a super-resolution non-key frame; and recombining the processed super-resolution key frame and non-key frame into a video sequence to generate a final super-resolution video. According to the method, key frame super-resolution processing can be carried out on the video in a short time, the video picture quality is improved, and the video is clearer and more vivid.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a digital video super-resolution processing method, system, equipment and medium. Background Art

[0002] With the development of digital media technology, video has become an indispensable part of people's daily life and work. However, due to the limitations of shooting equipment or compression during transmission, the resolution of many videos cannot meet the needs of current high-definition display devices. Traditional video magnification technology often leads to problems such as blurred images and loss of details, which seriously affects the viewing experience. Therefore, how to improve the clarity of low-resolution videos has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a digital video super-resolution processing method, system, device and medium.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A digital video super-resolution processing method comprises the following steps:

[0006] Decompose video files into individual image frames;

[0007] Calculate the threshold for selecting key frames;

[0008] Classifying image frames into key frames and non-key frames based on a threshold;

[0009] Use RealBasicVSR technology to perform super-resolution processing on key frames to improve the resolution of key frames;

[0010] The feature-guided variational optical flow method is used to calculate the motion displacement between the non-key frame and the previous closest key frame; based on the motion displacement, the previous closest key frame of the current non-key frame is projected to the current non-key frame;

[0011] The projected image is repaired using a fast marching repair method to obtain a super-resolution non-key frame.

[0012] The processed super-resolution key frames and non-key frames are reassembled into a video sequence to generate the final super-resolution video.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] Furthermore, decomposing the video file into independent image frames specifically includes:

[0015] Input the path of the video file, and use the FFmpeg tool to extract the image frames in the video frame by frame, and store these image frames in the specified folder. The image frames are named and stored in chronological order.

[0016] Furthermore, the threshold for selecting key frames is specifically:

[0017] Calculate the motion error e between every two adjacent frames i ,i +1 :

[0018]

[0019] where I i (p) and I i +1(p) represent the gray values of the i-th frame and the (i + 1)-th frame at pixel p respectively; P represents the total number of pixels in a frame of image; α represents the error coefficient;

[0020] Calculate the average error E of all frames in the entire video:

[0021]

[0022] where e i ,i+1 is the motion error between the i-th frame and the (i + 1)-th frame, and N is the total number of frames in the entire video;

[0023] Set the lower limit c1 and upper limit c2 of the threshold;

[0024] If E < c1, let the threshold T = E, select the key frames, and count the proportion of key frames in the total number of frames. Adjust the threshold T up and down in steps of 0.1 according to the proportion of key frames in the total number of frames until the proportion of key frames meets the requirements to obtain the final threshold T; if c1 ≤ E < c2, directly obtain the threshold T = E.

[0025] Furthermore, classifying the image frames into key frames and non-key frames based on the threshold is specifically:

[0026] Define the first frame of the video as the initial key frame; for the video frames after the first frame, calculate the motion error e between the current video frame and the nearest previous key frame:

[0027]

[0028] where y and y pre are the current frame and the nearest previous key frame respectively, P() represents an operation to convert an RGB image into a grayscale image, and t represents the number of non-zero terms in the error matrix ∑(P(y) - P(y pre ));

[0029] If the motion error e is greater than the threshold T, the current frame is selected as a key frame; otherwise, the current frame is selected as a non-key frame.

[0030] Furthermore, the RealBasicVSR technology is used to perform super-resolution processing on key frames as follows:

[0031] The mapping relationship between low-resolution images and high-resolution images is learned by training the RealBasicVSR model, the features of key frames are extracted using the trained RealBasicVSR model, and these features are used to reconstruct high-resolution key frames.

[0032] Furthermore, the feature-guided variational optical flow method is used to calculate the motion displacement between the non-key frame and the previous closest key frame as follows:

[0033]

[0034] Among them, E(θ) is the motion displacement, θ is the optical flow to be calculated, I represents the non-key frame, and K represents the nearest key frame before the non-key frame. and are the gradients of the keyframe and non-keyframe at pixel p, respectively, and the function c = 0.001, θ1(p) represents the corresponding optical flow vector obtained by feature matching at point p, θ(p) represents whether the key frame has features available at point p, and its value is 0 or 1, F key (p) and F non (p+θ1(p)) represents the feature vectors in keyframes and non-keyframes respectively;

[0035] The method of projecting the nearest key frame adjacent to the current non-key frame to the current non-key frame based on motion displacement is specifically as follows:

[0036] x k，l =y m，n , k=m+θ1,l=n+θ2

[0037] Among them, y represents the previous nearest key frame, x represents the high-resolution version of the current non-key frame, θ1 and θ2 represent the first dimension and second dimension of the optical flow θ respectively, (k, l) represents the coordinates of the pixel point in the non-key frame, and (m, n) represents the coordinates of the pixel point in the key frame.

[0038] Furthermore, the fast marching repair method is used to repair the projected image in detail as follows:

[0039]

[0040] In the formula, I repaired(d) represents the brightness of the restored pixel d, VL(f) is the brightness gradient at the pixel f, d∈D is the pixel to be restored, Q(d) is a neighborhood of the pixel d, each point f in the neighborhood is used to restore the point d, W(d, f) is the weight function, which is used to define the contribution of the point f in the neighborhood to the restoration of the current point d to be restored. The weight function is defined as follows:

[0041] W(d,f)=P(d,f)·Q(d,f)·R(d,f)

[0042] Where P(d, f) represents the direction parameter, Q(d, f) represents the distance parameter, and R(d, f) represents the level set distance parameter.

[0043] The present invention also provides a digital video super-resolution processing system, comprising:

[0044] A decomposition module, used for decomposing a video file into independent image frames;

[0045] A threshold calculation module, used for calculating a threshold for selecting a key frame;

[0046] A classification module, for classifying image frames into key frames and non-key frames based on a threshold;

[0047] The key frame repair module is used to perform super-resolution processing on key frames using RealBasicVSR technology to improve the resolution of key frames;

[0048] The non-key frame restoration module is used to calculate the motion displacement between the non-key frame and the previous nearest key frame using the feature-guided variational optical flow method; project the nearest key frame before the current non-key frame to the current non-key frame based on the motion displacement; and restore the projected image using the fast marching restoration method to obtain a super-resolution non-key frame;

[0049] The synthesis module is used to recombine the processed super-resolution key frames and non-key frames into a video sequence to generate the final super-resolution video.

[0050] The present invention further proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the digital video super-resolution processing method as described above is implemented.

[0051] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the digital video super-resolution processing method as described above.

[0052] The beneficial effects of the present invention are:

[0053] The present invention adopts the most advanced RealBasicVSR technology and FFmpeg operation tools, intelligently identifies key frames in the video and performs super-resolution processing on them, while appropriately enlarging non-key frames to ensure the uniformity and smoothness of the entire video picture. The present invention can perform key frame super-resolution processing on the video in a short time, improve the video picture quality, and make the video clearer and more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall flow chart of the present invention;

[0055] Figure 2 Schematic diagram of the video frame extraction process;

[0056] Figure 3 Schematic diagram of image super-resolution processing;

[0057] Figure 4 Schematic diagram of the process of synthesizing video for high-resolution images;

[0058] Figure 5 Comparison chart before and after video super-resolution processing. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] Embodiment 1

[0061] The present invention proposes a digital video super-resolution processing method, the overall process is as follows Figure 1 As shown, the following steps are included:

[0062] Decompose the video file into individual image frames; e.g. Figure 2 As shown, specifically:

[0063] Input the video file path, use the FFmpeg tool to extract the image frames in the video frame by frame, and store these image frames in the specified folder. The image frames are named and stored in chronological order.

[0064] Calculate the threshold for selecting keyframes; specifically:

[0065] Calculate the motion error e between each two adjacent frames i，i+1 :

[0066]

[0067] Among them, I i (p) and I i+1 (p) respectively represent the gray values of the i-th frame and the (i + 1)-th frame at pixel p; P represents the total number of pixels in an image frame; α represents the error coefficient;

[0068] Calculate the average error E of all frames in the entire video:

[0069]

[0070] Among them, e i，i+1 is the motion error between the i-th frame and the (i + 1)-th frame, and N is the total number of frames in the entire video;

[0071] Set the lower threshold c1 and the upper threshold c2;

[0072] If E < c1, let the threshold T = E, select the key frames, and count the proportion of key frames in the total number of frames. According to the proportion of key frames in the total number of frames, adjust the threshold T up and down in steps of 0.1 until the proportion of key frames meets the requirements to obtain the final threshold T; if c1 ≤ E < c2, directly obtain the threshold T = E.

[0073] Based on the threshold, divide the image frames into key frames and non-key frames; specifically:

[0074] Define the first frame of the video as the initial key frame; for the video frames after the first frame, calculate the motion error e between the current video frame and the nearest previous key frame:

[0075]

[0076] Among them,.y and yp r e are the current frame and the nearest previous key frame respectively, P() represents an operation to convert an RGB image into a grayscale image, and t represents the number of non-zero terms in the error matrix ∑(P(y) - P(ypre));

[0077] If the motion error e is greater than the threshold T, the current frame is selected as a key frame; otherwise, the current frame is selected as a non-key frame.

[0078] Use the RealBasicVSR technology to perform super-resolution processing on the key frames to improve the resolution of the key frames; specifically, by training the RealBasicVSR model to learn the mapping relationship between the low-resolution image and the high-resolution image, using the trained RealBasicVSR model to extract the features of the key frames, and using these features to reconstruct the high-resolution key frames. As Figure 3 shown.

[0079] The feature-guided variational optical flow method is used to calculate the motion displacement between the non-key frame and the previous closest key frame; based on the motion displacement, the previous closest key frame of the current non-key frame is projected to the current non-key frame; specifically:

[0080]

[0081] Among them, E(θ) is the motion displacement, θ is the optical flow to be calculated, I represents the non-key frame, and K represents the nearest key frame before the non-key frame. and are the gradients of the key frame and non-key frame at pixel p, respectively, and the function φ c = 0.001, θ1(p) represents the corresponding optical flow vector obtained by feature matching at point p, θ(p) represents whether the key frame has features available at point p, and its value is 0 or 1, F key (p) and F non (p+θ1(p)) represents the feature vectors in keyframes and non-keyframes respectively;

[0082] The method of projecting the nearest key frame adjacent to the current non-key frame to the current non-key frame based on motion displacement is specifically as follows:

[0083] x k，l =y m.n , k=m+θ1,l=n+θ2

[0084] Among them, y represents the previous nearest key frame, x represents the high-resolution version of the current non-key frame, θ1 and θ2 represent the first dimension and second dimension of the optical flow θ respectively, (k, l) represents the coordinates of the pixel point in the non-key frame, and (m, n) represents the coordinates of the pixel point in the key frame.

[0085] The projected image is repaired using a fast marching repair method to obtain a super-resolution non-key frame; specifically:

[0086]

[0087] In the formula, I repaired (d) represents the brightness of the restored pixel d, VL(f) is the brightness gradient at the pixel f, d∈D is the pixel to be restored, Q(d) is a neighborhood of the pixel d, each point f in the neighborhood is used to restore the point d, W(d, f) is the weight function, which is used to define the contribution of the point f in the neighborhood to the restoration of the current point d to be restored. The weight function is defined as follows:

[0088] W(d,f)=P(d,f)·Q(d,f)·R(d,f)

[0089] Where P(d, f) represents the direction parameter, Q(d, f) represents the distance parameter, and R(d, f) represents the level set distance parameter.

[0090] The processed super-resolution key frames and non-key frames are reassembled into a video sequence to generate the final super-resolution video. Figure 4 As shown in the figure, all processed high-resolution image frames are reassembled into a video file in chronological order using the FFmpeg tool to ensure the coherence and consistency of video playback.

[0091] Embodiment 2

[0092] The present invention proposes a digital video super-resolution processing system corresponding to the method of embodiment 1, comprising:

[0093] A decomposition module, used for decomposing a video file into independent image frames;

[0094] A threshold calculation module, used for calculating a threshold for selecting a key frame;

[0095] A classification module, for classifying image frames into key frames and non-key frames based on a threshold;

[0096] The key frame repair module is used to perform super-resolution processing on key frames using RealBasicVSR technology to improve the resolution of key frames;

[0097] The non-key frame restoration module is used to calculate the motion displacement between the non-key frame and the previous nearest key frame using the feature-guided variational optical flow method; project the nearest key frame before the current non-key frame to the current non-key frame based on the motion displacement; and restore the projected image using the fast marching restoration method to obtain a super-resolution non-key frame;

[0098] The synthesis module is used to recombine the processed super-resolution key frames and non-key frames into a video sequence to generate the final super-resolution video.

[0099] The implementation methods of each module and module function in the system are completely consistent with the steps of the method in Example 1, so they will not be repeated here.

[0100] Embodiment 3

[0101] The present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the digital video super-resolution processing method as described in the first embodiment is implemented.

[0102] Embodiment 4

[0103] The present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the digital video super-resolution processing method as described in the first embodiment.

[0104] The user selects the video file that he wants to process with super resolution through the "SuperResolution" software. The software decomposes the video into image frames and performs super resolution processing on each image frame independently. The processed image frames are recombined into a super resolution video and saved in a specified folder. The effect of the present invention is as follows: Figure 5 shown.

[0105] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0106] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0107] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A digital video super-resolution processing method, characterized in that, it includes the following steps: Decompose the video file into independent image frames; Calculate the threshold for selecting key frames; Based on the threshold, divide the image frames into key frames and non-key frames; Use the RealBasicVSR technology to perform super-resolution processing on the key frames to improve the resolution of the key frames; Use the feature-guided variational optical flow method to calculate the motion displacement between the non-key frame and the nearest adjacent key frame before it; project the nearest adjacent key frame before the current non-key frame onto the current non-key frame based on the motion displacement; Use the fast marching inpainting method to repair the projected image to obtain a super-resolution non-key frame; Recombine the processed super-resolution key frames and non-key frames into a video sequence to generate the final super-resolution video.

2. The digital video super-resolution processing method according to claim 1, characterized in that, the specific method of decomposing the video file into independent image frames is: Input the video file path, use the FFmpeg tool to extract the image frames in the video frame by frame, and store these image frames in a specified folder. The image frames are named and stored in chronological order.

3. The digital video super-resolution processing method according to claim 1, characterized in that, the specific method of calculating the threshold for selecting key frames is: Calculate the motion error e between each two adjacent frames i,i+1 : Among them, I i (p) and I i+1 (p) represents the grayscale value at pixel p in the i-th frame and the i+1-th frame respectively; P represents the total number of pixels in a frame image; α represents the error coefficient; Calculate the average error E of all frames of the entire video: Among them, e i,i+1 is the motion error between the i-th frame and the i+1-th frame, and N is the total number of frames in the entire video; Set the threshold lower limit c1 and upper limit c2; If E < c1, let the threshold T = E, select the key frames, and count the proportion of key frames in the total number of frames. Adjust the threshold T up and down in steps of 0.1 according to the proportion of key frames in the total number of frames until the proportion of key frames meets the requirements to obtain the final threshold T; if c1 ≤ E < c2, directly obtain the threshold T = E.

4. The digital video super-resolution processing method according to claim 1, characterized in that, the specific method of dividing the image frames into key frames and non-key frames based on the threshold is: Define the first frame of the video as the initial key frame; for the video frames after the first frame, calculate the motion error e between the current video frame and the nearest adjacent key frame before it: Among them, y and y pre are the current frame and the previous nearest neighbor key frame, P() represents an operation to convert an RGB image into a grayscale image, and t represents the error matrix ∑(P(y)-P(y pre ))The number of non-zero entries in ; If the motion error e is greater than the threshold T, the current frame is selected as a key frame; otherwise, the current frame is selected as a non-key frame.

5. The digital video super-resolution processing method according to claim 1, characterized in that, the specific method of performing super-resolution processing on the key frames using the RealBasicVSR technology is: Through training the RealBasicVSR model to learn the mapping relationship between low-resolution images and high-resolution images, use the trained RealBasicVSR model to extract the features of the key frames, and use these features to reconstruct high-resolution key frames.

6. The digital video super-resolution processing method according to claim 1, characterized in that, the specific method of calculating the motion displacement between the non-key frame and the nearest adjacent key frame before it using the feature-guided variational optical flow method is: Among them, E(θ) is the motion displacement, θ is the optical flow to be calculated, I represents the non-key frame, and K represents the nearest key frame before the non-key frame. and are the gradients of the keyframe and non-keyframe at pixel p, respectively, and the function θ 1 (p) represents the corresponding optical flow vector obtained by feature matching at point p, θ(p) represents whether the key frame has features available at point p, and its value is 0 or 1. key (p) and F non (p+θ 1 (p)) represent the feature vectors in keyframes and non-keyframes respectively; the specific method of projecting the nearest adjacent key frame before the current non-key frame onto the current non-key frame based on the motion displacement is: x k,l =y m,n ,k=m+θ 1 ,l=n+θ 2 Among them, y represents the nearest key frame, x represents the high-resolution version of the current non-key frame, and θ 1 and θ 2 They represent the first and second dimensions of the optical flow θ respectively, (k, l) represents the coordinates of the pixel points in the non-key frame, and (m, n) represents the coordinates of the pixel points in the key frame.

7. The digital video super-resolution processing method according to claim 1, characterized in that, The method of using the fast marching repair method to repair the projected image is specifically as follows: In the formula, I repaired (d) represents the brightness of the restored pixel d, VL(f) is the brightness gradient at the pixel f, d∈D is the pixel to be restored, Q(d) is a neighborhood of the pixel d, and each point f in the neighborhood is used to restore the point d. W(d,f) is a weight function that defines the contribution of the point f in the neighborhood to the restoration of the current point d to be restored. The weight function is defined as follows: W(d,f)=P(d,f)·Q(d,f)·R(d,f) Where P(d, f) represents the direction parameter, Q(d, f) represents the distance parameter, and R(d, f) represents the level set distance parameter.

8. A digital video super-resolution processing system, It is characterized in that include: A decomposition module, used for decomposing a video file into independent image frames; A threshold calculation module, used for calculating a threshold for selecting a key frame; A classification module, for classifying image frames into key frames and non-key frames based on a threshold; The key frame repair module is used to perform super-resolution processing on key frames using RealBasicVSR technology to improve the resolution of key frames; The non-key frame restoration module is used to calculate the motion displacement between the non-key frame and the previous nearest key frame using the feature-guided variational optical flow method; project the nearest key frame before the current non-key frame to the current non-key frame based on the motion displacement; and restore the projected image using the fast marching restoration method to obtain a super-resolution non-key frame; The synthesis module is used to recombine the processed super-resolution key frames and non-key frames into a video sequence to generate the final super-resolution video.

9. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the digital video super-resolution processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, It is characterized in that The computer program enables a computer to execute the digital video super-resolution processing method as described in any one of claims 1 to 7.

Images