Video restoration method and device

Abstract

The embodiment of the invention provides a video restoration method and device, and relates to the technical field of video restoration. Comprising the following steps: acquiring a first reference frame and a second reference frame of a to-be-repaired video frame; based on the first reference frame and the second reference frame, obtaining a first repairing result of the to-be-repaired video frame through an interpolation algorithm; acquiring a forward optical flow from the first reference frame to the second reference frame and a reverse optical flow from the second reference frame to the first reference frame; obtaining a second repair result of the video frame to be repaired according to the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow; obtaining a first reliability weight of the first repair result and a second reliability weight of the second repair result according to an optical flow consistency error between the forward optical flow and the reverse optical flow; and performing weighted fusion on the second repair result and the first repair result according to the first reliability weight and the second reliability weight to obtain a target repair result of the to-be-repaired video frame. The embodiment of the invention is used for improving the video repairing effect.

Description

TECHNICAL FIELD

[0001] The embodiments of the present application relate to the technical field of video restoration. More particularly, the present application relates to a video restoration method and device. BACKGROUND

[0002] With the acceleration of the process of media digitization, a large number of valuable historical video materials are being converted from traditional analog media (such as film and magnetic tape) to digital format. These historical video materials include early television dramas, documentaries, news materials and artistic performances, etc., which have important historical, cultural and artistic value.

[0003] Due to the long-term storage of film carriers, the material ages, the film may have cracks, scratches or emulsion layer deterioration, and the magnetic powder of the magnetic tape decays, resulting in signal reading distortion. In addition, analog signals are easily affected by electromagnetic interference, mechanical parts of the playing equipment are worn out, and the storage environment is affected by temperature and humidity fluctuations, light pollution and other factors. Old films generally have signal transmission and carrier loss problems, which are specifically manifested in the phenomenon of flickering stripes, color block disorder or snowflake noise during playing. With the advent of the digital era, it is necessary to convert these classic video works into digital films and restore the converted digital films, which is a key path to inherit classic video culture. Unfortunately, the restoration effect of the converted digital films is not good, which is difficult to meet the growing viewing needs of the audience, which not only greatly affects the viewing experience of the audience, but also seriously restricts the development and extensive use of classic video resources in the new era. SUMMARY

[0004] The exemplary embodiments of the present application provide a video restoration method and device for improving the restoration effect of the video.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] In a first aspect, the embodiments of the present application provide a video restoration method, comprising:

[0007] obtaining a first reference frame and a second reference frame of a to-be-restored video frame; in the time domain, the first reference frame is located before the to-be-restored video frame, and the second reference frame is located after the to-be-restored video frame;

[0008] obtaining a first restoration result of the to-be-restored video frame based on the first reference frame and the second reference frame by an interpolation algorithm;

[0009] obtaining a forward optical flow from the first reference frame to the second reference frame and a reverse optical flow from the second reference frame to the first reference frame;

[0010] obtaining a second repair result of the video frame to be repaired according to the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow;

[0011] obtaining a first reliability weight of the first repair result and a second reliability weight of the second repair result according to an optical flow consistency error between the forward optical flow and the reverse optical flow, the optical flow consistency error being positively correlated with the first reliability weight, and the optical flow consistency error being negatively correlated with the second reliability weight;

[0012] fusing the second repair result and the first repair result according to the first reliability weight and the second reliability weight to obtain a target repair result of the video frame to be repaired.

[0013] As an optional implementation of the embodiment of the application, the obtaining of the first reliability weight of the first repair result and the second reliability weight of the second repair result according to the optical flow consistency error between the forward optical flow and the reverse optical flow comprises:

[0014] the second reliability weight is obtained according to the optical flow consistency error and a reliability weight calculation formula; the reliability weight calculation formula is:

[0015]

[0016] wherein, W(x, y) is the second reliability weight, E flow (x, y) is the optical flow consistency error, and σ is a constant.

[0017] As an optional implementation of the embodiment of the application, before the obtaining of the first reliability weight of the first repair result and the second reliability weight of the second repair result according to the optical flow consistency error between the forward optical flow and the reverse optical flow, the method further comprises:

[0018] obtaining forward predicted pixel coordinates of each pixel point according to the forward optical flow;

[0019] obtaining reverse predicted pixel coordinates of each pixel point according to the reverse optical flow;

[0020] obtaining the optical flow consistency error according to the forward predicted pixel coordinates and the reverse predicted pixel coordinates of each pixel point.

[0021] As an optional implementation of the embodiment of the application, the obtaining of the first repair result of the video frame to be repaired based on the first reference frame and the second reference frame through an interpolation algorithm comprises:

[0022] obtaining a first pixel value and a second pixel value corresponding to each pixel point of the first repair result; the first pixel value corresponding to any pixel point is a pixel value of a pixel point with the same pixel coordinate in the first reference frame, and the second pixel value corresponding to any pixel point is a pixel value of a pixel point with the same pixel coordinate in the second reference frame;

[0023] performing weighted summation on the first pixel value and the second pixel value corresponding to each pixel point of the first repair result based on a preset weight coefficient, to obtain a pixel value of each pixel point of the first repair result;

[0024] obtaining the first repair result according to the pixel value of each pixel point of the first repair result.

[0025] As an optional implementation of the embodiment of the application, the obtaining the second repair result of the video frame to be repaired according to the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow comprises:

[0026] obtaining forward motion compensation information and reverse motion compensation information of the video frame to be repaired according to the forward optical flow and the reverse optical flow respectively;

[0027] performing motion compensation on the first reference frame based on the forward motion compensation information to obtain a forward compensation result;

[0028] performing motion compensation on the second reference frame based on the reverse motion compensation information to obtain a reverse compensation result;

[0029] fusing the forward compensation result and the reverse compensation result to obtain the second repair result.

[0030] As an optional implementation of the embodiment of the application, the obtaining forward motion compensation information and reverse motion compensation information of the video frame to be repaired according to the forward optical flow and the reverse optical flow respectively comprises:

[0031] obtaining a first coefficient according to an interval between the video frame to be repaired and the first reference frame;

[0032] calculating a product of the forward optical flow and the first coefficient to obtain the forward motion compensation information;

[0033] obtaining a second coefficient according to an interval between the video frame to be repaired and the second reference frame;

[0034] calculating a product of the reverse optical flow and the second coefficient to obtain the reverse motion compensation information.

[0035] As an optional implementation of the embodiment of the present application, the fusing the forward compensation result and the reverse compensation result to obtain a second repair result comprises:

[0036] calculating an average value of pixel values of pixel points with same pixel coordinates in the forward compensation result and the reverse compensation result to obtain pixel values of each pixel point in the second repair result;

[0037] obtaining the second repair result according to the pixel values of each pixel point in the second repair result.

[0038] In a second aspect, the embodiment of the present application provides a video repair device, comprising:

[0039] a reference frame obtaining unit configured to obtain a first reference frame and a second reference frame of a video frame to be repaired; in a time domain, the first reference frame is located before the video frame to be repaired, and the second reference frame is located after the video frame to be repaired;

[0040] a first repair unit configured to obtain a first repair result of the video frame to be repaired by an interpolation algorithm based on the first reference frame and the second reference frame;

[0041] an optical flow obtaining unit configured to obtain a forward optical flow from the first reference frame to the second reference frame and a reverse optical flow from the second reference frame to the first reference frame;

[0042] a second repair unit configured to obtain a second repair result of the video frame to be repaired according to the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow;

[0043] a weight determining unit configured to obtain a first reliability weight of the first repair result and a second reliability weight of the second repair result according to an optical flow consistency error between the forward optical flow and the reverse optical flow, the optical flow consistency error being positively correlated with the first reliability weight, and the optical flow consistency error being negatively correlated with the second reliability weight;

[0044] a weighted fusion unit configured to weightedly fuse the second repair result and the first repair result according to the first reliability weight and the second reliability weight to obtain a target repair result of the video frame to be repaired.

[0045] As an optional implementation of the embodiment of the present application, the weight determining unit is specifically configured to obtain the second reliability weight according to the optical flow consistency error and a reliability weight calculation formula; the reliability weight calculation formula is:

[0046]

[0047] wherein, W(x, y) is the second reliability weight, E flow (x, y) is the optical flow consistency error, and σ is a constant.

[0048] As an optional implementation of the embodiment of the present application, the weight determination unit is further configured to: before obtaining the first reliability weight of the first repair result and the second reliability weight of the second repair result according to the optical flow consistency error between the forward optical flow and the reverse optical flow, obtain the forward predicted pixel coordinates of each pixel point according to the forward optical flow; obtain the reverse predicted pixel coordinates of each pixel point according to the reverse optical flow; and obtain the optical flow consistency error according to the forward predicted pixel coordinates and the reverse predicted pixel coordinates of each pixel point.

[0049] As an optional implementation of the embodiment of the present application, the first repair unit is specifically configured to: obtain the first pixel value and the second pixel value corresponding to each pixel point of the first repair result; the first pixel value corresponding to any pixel point is the pixel value of the pixel point with the same pixel coordinates in the first reference frame, and the second pixel value corresponding to any pixel point is the pixel value of the pixel point with the same pixel coordinates in the second reference frame; and perform weighted summation on the first pixel value and the second pixel value corresponding to each pixel point of the first repair result based on a preset weight coefficient, to obtain the pixel value of each pixel point of the first repair result; and obtain the first repair result according to the pixel value of each pixel point of the first repair result.

[0050] As an optional implementation of the embodiment of the present application, the second repair unit is specifically configured to: obtain the forward motion compensation information and the reverse motion compensation information of the video frame to be repaired according to the forward optical flow and the reverse optical flow, respectively; perform motion compensation on the first reference frame based on the forward motion compensation information to obtain a forward compensation result; perform motion compensation on the second reference frame based on the reverse motion compensation information to obtain a reverse compensation result; and fuse the forward compensation result and the reverse compensation result to obtain a second repair result.

[0051] As an optional implementation of the embodiment of the present application, the second repair unit is specifically configured to: obtain a first coefficient according to the interval between the video frame to be repaired and the first reference frame; calculate the product of the forward optical flow and the first coefficient to obtain the forward motion compensation information; obtain a second coefficient according to the interval between the video frame to be repaired and the second reference frame; and calculate the product of the reverse optical flow and the second coefficient to obtain the reverse motion compensation information.

[0052] As an optional implementation of the embodiment of the present application, the second repairing unit is specifically configured to calculate an average value of pixel values of pixel points with same pixel coordinates in the forward compensation result and the reverse compensation result, to obtain pixel values of pixel points in the second repairing result; and obtain the second repairing result according to the pixel values of the pixel points in the second repairing result.

[0053] In a third aspect, the embodiments of the present application provide an electronic device, comprising: a memory configured to store a computer program; and a processor configured to execute the computer program to implement the video repairing method in any optional implementation of the first aspect.

[0054] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, having a computer program stored thereon, and when the computer program is executed by a computing device, the computing device implements the video repairing method in any optional implementation of the first aspect.

[0055] In a sixth aspect, the embodiments of the present application provide a computer program product, when the computer program product runs on a computer, the computer implements the video repairing method in any optional implementation of the first aspect.

[0056] The video repairing method provided by the embodiments of the present application first obtains a first reference frame located before a to-be-repaired video frame in a time domain and a second reference frame located after the to-be-repaired video frame in the time domain, then obtains a first repairing result of the to-be-repaired video frame based on the first reference frame and the second reference frame by an interpolation algorithm, further obtains a forward optical flow from the first reference frame to the second reference frame and a reverse optical flow from the second reference frame to the first reference frame, and obtains a second repairing result of the to-be-repaired video frame according to the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow. After obtaining the first repairing result and the second repairing result, a first reliability weight of the first repairing result and a second reliability weight of the second repairing result are obtained according to an optical flow consistency error between the forward optical flow and the reverse optical flow, and the second repairing result and the first repairing result are fused by weighting according to the first reliability weight and the second reliability weight, to obtain a target repairing result of the to-be-repaired video frame. Since the video repairing method provided by the embodiments of the present application can obtain a first repairing result of a to-be-repaired video frame based on an interpolation algorithm, obtain a second repairing result of the to-be-repaired video frame based on an optical flow compensation algorithm, and obtain a first reliability weight of the first repairing result and a second reliability weight of the second repairing result according to an optical flow consistency error between the forward optical flow and the reverse optical flow, the embodiments of the present application can combine the advantages of two repairing methods, and realize more accurate fusion repairing by dynamically adjusting the weights, thereby improving the repairing effect of the video. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the implementation manners in the related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0058] Figure 1 A step flow chart of a video repairing method in some embodiments is shown;

[0059] Figure 2 A step flow chart of a video repairing method in some embodiments is shown;

[0060] Figure 3 A corresponding relationship diagram of a pixel point in some embodiments is shown;

[0061] Figure 4 A structure diagram of a video repairing device provided by the embodiments of the present application is shown;

[0062] Figure 5 A hardware structure diagram of an electronic device provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0063] In order to make the purpose and implementation manners of the present application more clear, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary implementation manners of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all the embodiments.

[0064] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the implementation manners of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0065] The terms "include" and "have" and any variations thereof, are intended to cover but not exclusively include, for example, a product or device containing a series of components, and it does not have to be limited to all the components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0066] Figure 1An exemplary step flow diagram of a video repairing method provided by an embodiment of the present application is shown in FIG. 1. The video repairing method can be executed by a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart watch, a smart bracelet, or other electronic device, or a video repairing apparatus integrated in the electronic device. Figure 1 As shown in FIG. 1, the video repairing method provided by an embodiment of the present application includes the following steps.

[0067] S11, acquiring a first reference frame and a second reference frame of the video frame to be repaired.

[0068] In the time domain, the first reference frame is located before the video frame to be repaired, and the second reference frame is located after the video frame to be repaired.

[0069] In some embodiments, the first reference frame and the second reference frame are intact video frames without screen phenomena such as present flickering stripes, color block disorder, or snowflake noise.

[0070] It should be noted that the first reference frame and the second reference frame can be adjacent video frames of the video frame to be repaired, or can be video frames that are not adjacent to the video frame to be repaired. The embodiments of the present application do not limit this, and the first reference frame is located before the video frame to be repaired and the second reference frame is located after the video frame to be repaired in the time domain.

[0071] In some embodiments, the implementation of acquiring the first reference frame and the second reference frame of the video frame to be repaired can include:

[0072] Determining whether a previous video frame of the video frame to be repaired in the time domain is an intact video frame. If yes, the previous video frame of the video frame to be repaired is determined as the first reference frame. If no, whether a next video frame is an intact video frame is determined until an intact video frame located before the video frame to be repaired is found and determined as the first reference frame.

[0073] Determining whether a next video frame of the video frame to be repaired in the time domain is an intact video frame. If yes, the next video frame of the video frame to be repaired is determined as the second reference frame. If no, whether a next video frame is an intact video frame is determined until an intact video frame located after the video frame to be repaired is found and determined as the second reference frame.

[0074] The screen artifact is usually caused by local or global image distortion. In some embodiments, the implementation of determining the video frame to be repaired (i.e., the video frame with the screen artifact) is as follows: first, feature extraction is performed on the current frame to obtain key features such as the proportion of abnormal pixels, color stability, and difference from adjacent frames; then, the key features are quantized and normalized to unify the feature dimension and value range; subsequently, an independent feature focusing on the screen artifact is decoupled from the processed features, which can effectively eliminate other interference factors and accurately reflect the screen-related information; finally, by comparing the independent feature with the feature distribution of the normal frame, it is determined whether it exceeds the preset threshold, and if it exceeds the threshold, it is determined that the current frame has a screen artifact and is determined as the video frame to be repaired.

[0075] The video frame with the screen artifact usually lasts for only 1-2 frames in the time domain, and the frames before and after it are usually intact, and the video sequence has high redundancy in the time dimension, and the content between adjacent frames is highly similar. Using this characteristic, the embodiments of the present application obtain a first video frame located before the video frame to be repaired in the time domain and a second video frame located after the video frame to be repaired in the time domain, and obtain information from the first video frame and the second video frame to repair the video frame to be repaired.

[0076] S12, based on the first reference frame and the second reference frame, obtaining a first repair result of the video frame to be repaired through an interpolation algorithm.

[0077] In some embodiments, according to the first reference frame and the second reference frame, the first repair result of the video frame to be repaired is obtained, including:

[0078] Based on a linear difference value algorithm, frames are inserted between the first reference frame and the second reference frame to obtain the first repair result of the video frame to be repaired.

[0079] S13, obtaining a forward optical flow (Forward Optical Flow) from the first reference frame to the second reference frame and a backward optical flow (Backward Optical Flow) from the second reference frame to the first reference frame.

[0080] In some embodiments, the forward optical flow from the first reference frame to the second reference frame and the backward optical flow from the second reference frame to the first reference frame can be obtained based on an optical flow estimation algorithm.

[0081] S14, according to the first reference frame, the second reference frame, the forward optical flow and the backward optical flow, obtaining a second repair result of the video frame to be repaired.

[0082] In some embodiments, the second repair result of the video frame to be repaired is obtained according to the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow, comprising: the second repair result of the video frame to be repaired is obtained by a motion compensation algorithm based on the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow.

[0083] S14, a first reliability weight of the first repair result and a second reliability weight of the second repair result are obtained according to an optical flow consistency error between the forward optical flow and the reverse optical flow.

[0084] Wherein, the optical flow consistency error is positively correlated with the first reliability weight, and the optical flow consistency error is negatively correlated with the second reliability weight.

[0085] The relationship between the optical flow consistency error and the first reliability weight of the first repair result is: the greater the optical flow consistency error, the greater the first reliability weight; on the contrary, the smaller the optical flow consistency error, the smaller the first reliability weight.

[0086] The relationship between the optical flow consistency error and the second reliability weight of the second repair result is: the greater the optical flow consistency error, the smaller the second reliability weight; on the contrary, the smaller the optical flow consistency error, the greater the second reliability weight.

[0087] In some embodiments, before the first reliability weight of the first repair result and the second reliability weight of the second repair result are obtained according to the optical flow consistency error between the forward optical flow and the reverse optical flow, the video repair method further comprises: obtaining the optical flow consistency error between the forward optical flow and the reverse optical flow, and the implementation manner of obtaining the optical flow consistency error between the forward optical flow and the reverse optical flow can comprise the following steps a to c:

[0088] Step a, the forward predicted pixel coordinates of each pixel point are obtained according to the forward optical flow.

[0089] The forward predicted position of the pixel point with pixel coordinates (x, y) is (u t-1→t+1 , v t-1→t+1 ).

[0090] Wherein, u t-1→t+1 , v t-1→t+1 are the horizontal and vertical components of the pixel point with pixel coordinates (x, y) in the forward optical flow, respectively.

[0091] Step b, the reverse predicted pixel coordinates of each pixel point are obtained according to the reverse optical flow.

[0092] The inverse prediction position of the pixel point with pixel coordinates (x, y) is (x+u t-1→t+1 , y+v t-1→t+1 ).

[0093] Step c, obtaining the optical flow consistency error according to the forward prediction pixel coordinates and the inverse prediction pixel coordinates of each pixel point.

[0094] In some embodiments, the optical flow consistency error is denoted as E flow (x,y), and the calculation formula of the optical flow consistency error obtained according to the forward prediction pixel coordinates and the inverse prediction pixel coordinates of each pixel point can be shown as the following formula (1):

[0095] E flow (x,y)=||V for→back (x,y)+V back→for (x+u t-1→t+1 , y+v t-1→t+1 )|| (1)

[0096] In some embodiments, the first reliability weight of the first repair result and the second reliability weight of the second repair result are obtained according to the optical flow consistency error between the forward optical flow and the inverse optical flow, including: the second reliability weight is obtained according to the optical flow consistency error and a reliability weight calculation formula. The reliability weight calculation formula is shown as the following formula (2):

[0097]

[0098] wherein, W(x,y) is the second reliability weight, E flow (x,y) is the optical flow consistency error, and σ is a constant.

[0099] The σ can control the sensitivity of the second reliability weight to the optical flow consistency error.

[0100] In some embodiments, the first reliability weight of the first repair result and the second reliability weight of the second repair result are obtained according to the optical flow consistency error between the forward optical flow and the inverse optical flow, including: the difference between 1 and the second reliability weight is calculated to obtain the first reliability weight.

[0101] S16, the second repair result and the first repair result are fused according to the first reliability weight and the second reliability weight to obtain the target repair result of the video frame to be repaired.

[0102] In some embodiments, the weighting and fusing the second repair result and the first repair result according to the first reliability weight and the second reliability weight to obtain the target repair result of the video frame to be repaired can be represented as the following formula (3):

[0103] F repaired (x,y)=W(x,y)*F motion (x,y)+[1-W(x,y)]*F simple (x,y) (3)

[0104] wherein, F repaired (x,y) is the target repair result of the video frame to be repaired, W(x,y) is the second reliability weight, 1-W(x,y) is the first reliability weight, F motion (x,y) is the second repair result, F simple (x,y) is the first repair result.

[0105] The video repair method provided by the embodiments of the present application first obtains a first reference frame located before the video frame to be repaired in the time domain and a second reference frame located after the video frame to be repaired in the time domain, then obtains a first repair result of the video frame to be repaired based on the first reference frame and the second reference frame through an interpolation algorithm, further obtains a forward optical flow from the first reference frame to the second reference frame and a reverse optical flow from the second reference frame to the first reference frame, and obtains a second repair result of the video frame to be repaired according to the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow. After obtaining the first repair result and the second repair result, a first reliability weight of the first repair result and a second reliability weight of the second repair result are obtained according to the optical flow consistency error between the forward optical flow and the reverse optical flow, and the second repair result and the first repair result are weighted and fused according to the first reliability weight and the second reliability weight to obtain a target repair result of the video frame to be repaired. Since the video repair method provided by the embodiments of the present application can obtain a first repair result of the video frame to be repaired based on an interpolation algorithm, obtain a second repair result of the video frame to be repaired based on an optical flow compensation algorithm, and obtain a first reliability weight of the first repair result and a second reliability weight of the second repair result according to the optical flow consistency error between the forward optical flow and the reverse optical flow, the embodiments of the present application can combine the advantages of two repair methods, realize more accurate fusion repair through dynamic weight adjustment, and improve the repair effect of the video.

[0106] In other embodiments, the implementation manner of obtaining the target repair result of the video frame to be repaired can further include:

[0107] The second repair result, the first repair result, and the to-be-repaired video frame are fused according to the first reliability weight and the second reliability weight to obtain a target repair result of the to-be-repaired video frame.

[0108] Compared with fusing the second repair result and the first repair result according to the first reliability weight and the second reliability weight to obtain a target repair result of the to-be-repaired video frame, fusing the second repair result, the first repair result, and the to-be-repaired video frame according to the first reliability weight and the second reliability weight to obtain a target repair result of the to-be-repaired video frame can effectively utilize effective information of the to-be-repaired video frame, thereby further improving the video repair effect.

[0109] As an extension and refinement of the above-mentioned embodiments, the embodiments of the present application provide another video repair method, referring to FIG. 1, the video repair method comprises the following steps: Figure 2 The video repair method comprises the following steps:

[0110] S201, acquiring a first reference frame and a second reference frame of the to-be-repaired video frame.

[0111] In the time domain, the first reference frame is located before the to-be-repaired video frame, and the second reference frame is located after the to-be-repaired video frame.

[0112] S202, acquiring a first pixel value and a second pixel value corresponding to each pixel point of the first repair result.

[0113] The first pixel value corresponding to any pixel point is the pixel value of a pixel point with the same pixel coordinates in the first reference frame, and the second pixel value corresponding to any pixel point is the pixel value of a pixel point with the same pixel coordinates in the second reference frame.

[0114] For example, referring to FIG. 3, the resolutions of the first reference frame 31, the first repair result 32, and the second reference frame 33 are the same, the pixel coordinates of the pixel point A in the first repair result 32 are (1, 2), the pixel coordinates of the pixel point B in the first reference frame 31 are (1, 2), and the pixel coordinates of the pixel point C in the second reference frame 33 are (1, 2), so the first pixel value corresponding to the pixel point A in the first repair result is the pixel value of the pixel point B, and the second pixel value corresponding to the pixel point A in the first repair result is the pixel value of the pixel point C. Figure 3 S203, performing weighted summation on the first pixel value and the second pixel value corresponding to each pixel point of the first repair result based on a preset weight coefficient to obtain the pixel value of each pixel point of the first repair result.

[0115]

[0116] ​Suppose that the weight values of the first pixel value and the second pixel value are both 0.5, and the initial repair value of the pixel point with the pixel coordinate (x, y) is I simple (x,y), the first pixel value and the second pixel value corresponding to the pixel point with the pixel coordinate (x, y) are I forward (x,y) and I backward (x,y), the process of obtaining the initial repair value of the pixel point with the pixel coordinate (x, y) according to the first pixel value and the second pixel value corresponding to the pixel point with the pixel coordinate (x, y) can be represented as the following formula (4):

[0117] I simple (x,y)=0.5*I forward (x,y)+0.5*I backward (x,y) (4)

[0118] S204, obtaining the first repair result according to the pixel values of each pixel point in the first repair result.

[0119] S205, obtaining the forward optical flow from the first reference frame to the second reference frame and the reverse optical flow from the second reference frame to the first reference frame.

[0120] Let the forward optical flow be V for→back , the reverse optical flow be V back→for , the first reference frame be F forward , the second reference frame be F backward , and the optical flow estimation algorithm function be OpticalFlow(), then obtaining the forward optical flow from the first reference frame to the second reference frame can be represented as the following formula (5), and obtaining the forward optical flow from the second reference frame to the first reference frame can be represented as the following formula (6):

[0121] V for→back =OpticalFlow(F forward , F backward ) (5)

[0122] V back→for =OpticalFlow(F backward , F forward ) (6)

[0123] S206, obtaining a first coefficient according to the interval between the video frame to be repaired and the first reference frame.

[0124] In some embodiments, obtaining a first coefficient according to the interval between the video frame to be repaired and the first reference frame comprises: determining the first coefficient as 1 / (1+n). Wherein, n is the interval between the video frame to be repaired and the first reference frame.

[0125] For example, if the interval between the video frame to be repaired and the first reference frame is 0 (adjacent), the first coefficient is determined as 0.5, and if the interval between the video frame to be repaired and the first reference frame is 1, the first coefficient is determined as 1 / 3.

[0126] S207, calculating the product of the forward optical flow and the first coefficient to obtain the forward motion compensation information.

[0127] The first coefficient is denoted as a1, and the forward motion compensation information is denoted as V for→now The process of obtaining the forward motion compensation information can be represented by the following formula (7):

[0128] V for→now =a1*V for→back (7)

[0129] S208, obtaining a second coefficient according to the interval between the video frame to be repaired and the second reference frame.

[0130] In some embodiments, obtaining a second coefficient according to the interval between the video frame to be repaired and the second reference frame includes: determining the second coefficient as 1 / (1+m). Wherein, m is the interval between the video frame to be repaired and the second reference frame.

[0131] For example, if the interval between the video frame to be repaired and the second reference frame is 0 (adjacent), the first coefficient is determined as 0.5, and if the interval between the video frame to be repaired and the first reference frame is 1, the first coefficient is determined as 1 / 3.

[0132] S209, calculating the product of the reverse optical flow and the second coefficient to obtain the reverse motion compensation information.

[0133] The second coefficient is denoted as a2, and the reverse motion compensation information is denoted as V now→back The process of obtaining the reverse motion compensation information can be represented by the following formula (8):

[0134] V now→back =a2*V back→for (8)

[0135] S210, performing motion compensation on the first reference frame based on the forward motion compensation information to obtain a forward compensation result, and performing motion compensation on the second reference frame based on the reverse motion compensation information to obtain a reverse compensation result.

[0136] In some embodiments, the motion compensation on the first reference frame based on the forward motion compensation information to obtain a forward compensation result comprises: shifting the position of each pixel point in the first reference frame based on the forward motion compensation information to obtain the forward compensation result.

[0137] In some embodiments, the implementation of the motion compensation on the first reference frame based on the forward motion compensation information to obtain a forward compensation result can be shown in the following formula (9):

[0138] F MotionFor (x, y) = F forward (x - u t-1→t , y - v t-1→t ) (9)

[0139] wherein, F MotionFor (x, y) is the forward compensation result, u t-1→t , v t-1→t are horizontal and vertical components of forward optical flow respectively.

[0140] In some embodiments, the motion compensation on the second reference frame based on the backward motion compensation information to obtain a backward compensation result comprises: shifting the position of each pixel point in the second reference frame based on the backward motion compensation information to obtain the backward compensation result.

[0141] In some embodiments, the implementation of the motion compensation on the second reference frame based on the backward motion compensation information to obtain a backward compensation result can be shown in the following formula (10):

[0142] F MotionBack (x, y) = F backward (x - u t+1→t , y - v t+1→t ) (10)

[0143] wherein, F MotionBack (x, y) is the backward compensation result, u t+1→t , v t+1→t are horizontal and vertical components of backward optical flow respectively.

[0144] S211, calculating the average value of the pixel values of the pixel points with the same pixel coordinates in the forward compensation result and the backward compensation result to obtain the pixel value of each pixel point in the second repair result.

[0145] The pixel value of the pixel point with pixel coordinates (x, y) in the second repair result is represented as I motion(x, y), the average of the pixel values of the pixel points with the same pixel coordinates in the forward compensation result and the backward compensation result can be calculated to obtain the pixel value of each pixel point in the second repair result, and the process can be represented by the following formula (11):

[0146]

[0147] S212, obtaining the second repair result according to the pixel value of each pixel point in the second repair result.

[0148] S213, obtaining the optical flow consistency error between the forward optical flow and the backward optical flow.

[0149] S214, obtaining the first reliability weight of the first repair result and the second reliability weight of the second repair result according to the optical flow consistency error between the forward optical flow and the backward optical flow.

[0150] The optical flow consistency error is positively correlated with the first reliability weight, and the optical flow consistency error is negatively correlated with the second reliability weight.

[0151] S215, weighting and fusing the second repair result and the first repair result according to the first reliability weight and the second reliability weight to obtain the target repair result of the to-be-repaired video frame.

[0152] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a video repair device, which corresponds to the above-mentioned method embodiment. For ease of reading, the details of the above-mentioned method embodiment will not be described one by one, but it should be clear that the video repair device in the embodiment can correspondingly implement all the contents in the above-mentioned method embodiment.

[0153] An embodiment of the present application provides a video repair device, Figure 4 The structural diagram of the video repair device is shown in FIG. 4, and the video repair device 400 includes: Figure 4

[0154] The reference frame acquisition unit 41 is configured to acquire a first reference frame and a second reference frame of a to-be-repaired video frame. In the time domain, the first reference frame is located before the to-be-repaired video frame, and the second reference frame is located after the to-be-repaired video frame.

[0155] The first repair unit 42 is configured to acquire a first repair result of the to-be-repaired video frame by an interpolation algorithm based on the first reference frame and the second reference frame.

[0156] ​an optical flow obtaining unit 43, configured to obtain a forward optical flow from the first reference frame to the second reference frame and a reverse optical flow from the second reference frame to the first reference frame;

[0157] a second repairing unit 44, configured to obtain a second repairing result of the to-be-repaired video frame according to the first reference frame, the second reference frame, the forward optical flow and the reverse optical flow;

[0158] a weight determining unit 45, configured to obtain a first reliability weight of the first repairing result and a second reliability weight of the second repairing result according to an optical flow consistency error between the forward optical flow and the reverse optical flow, the optical flow consistency error being positively correlated with the first reliability weight, and the optical flow consistency error being negatively correlated with the second reliability weight;

[0159] a weighted fusion unit 46, configured to weightedly fuse the second repairing result and the first repairing result according to the first reliability weight and the second reliability weight, to obtain a target repairing result of the to-be-repaired video frame.

[0160] As an optional implementation of the embodiment of the present application, the weight determining unit 45 is specifically configured to obtain the second reliability weight according to the optical flow consistency error and a reliability weight calculation formula; the reliability weight calculation formula is:

[0161]

[0162] wherein, W(x, y) is the second reliability weight, E flow (x, y) is the optical flow consistency error, and σ is a constant.

[0163] As an optional implementation of the embodiment of the present application, the weight determining unit 45 is further configured to, before obtaining the first reliability weight of the first repairing result and the second reliability weight of the second repairing result according to the optical flow consistency error between the forward optical flow and the reverse optical flow, obtain a forward predicted pixel coordinate of each pixel point according to the forward optical flow; obtain a reverse predicted pixel coordinate of each pixel point according to the reverse optical flow; and obtain the optical flow consistency error according to the forward predicted pixel coordinate and the reverse predicted pixel coordinate of each pixel point.

[0164] As an optional implementation of the embodiment of the present application, the first repairing unit 42 is specifically configured to acquire the first pixel value and the second pixel value corresponding to each pixel point of the first repairing result; the first pixel value corresponding to any pixel point is the pixel value of a pixel point with the same pixel coordinate in the first reference frame, and the second pixel value corresponding to any pixel point is the pixel value of a pixel point with the same pixel coordinate in the second reference frame; the first pixel value and the second pixel value corresponding to each pixel point of the first repairing result are weighted and summed based on a preset weight coefficient to acquire the pixel value of each pixel point of the first repairing result; and the first repairing result is acquired according to the pixel value of each pixel point of the first repairing result.

[0165] As an optional implementation of the embodiment of the present application, the second repairing unit 44 is specifically configured to acquire the forward motion compensation information and the reverse motion compensation information of the video frame to be repaired according to the forward optical flow and the reverse optical flow respectively; acquire a forward compensation result by performing motion compensation on the first reference frame based on the forward motion compensation information; acquire a reverse compensation result by performing motion compensation on the second reference frame based on the reverse motion compensation information; and acquire a second repairing result by fusing the forward compensation result and the reverse compensation result.

[0166] As an optional implementation of the embodiment of the present application, the second repairing unit 44 is specifically configured to acquire a first coefficient according to the interval between the video frame to be repaired and the first reference frame; acquire the forward motion compensation information by calculating the product of the forward optical flow and the first coefficient; acquire a second coefficient according to the interval between the video frame to be repaired and the second reference frame; and acquire the reverse motion compensation information by calculating the product of the reverse optical flow and the second coefficient.

[0167] As an optional implementation of the embodiment of the present application, the second repairing unit 44 is specifically configured to acquire the pixel value of each pixel point of the second repairing result by calculating the average value of the pixel values of the pixel points with the same pixel coordinate in the forward compensation result and the reverse compensation result; and acquire the second repairing result according to the pixel value of each pixel point in the second repairing result.

[0168] The video repairing apparatus provided in the embodiments of the present application can execute the video repairing method provided in any of the above embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0169] Based on the same inventive concept, the embodiments of the present application further provide an electronic device. Figure 5 The structural schematic diagram of the electronic device provided in the embodiments of the present application is shown in Figure 5As shown, the electronic device provided by the embodiment includes a memory 501 and a processor 502, the memory 501 is used for storing a computer program, and the processor 502 is used for executing the video repairing method provided by the above embodiment when executing the computer program.

[0170] Based on the same inventive concept, the embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the video repairing method provided by any of the above embodiments.

[0171] Based on the same inventive concept, the embodiment of the present application provides a computer program product, and when the computer program product runs on a computer, the computer implements the video repairing method provided by any of the above embodiments.

[0172] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0173] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0174] The memory can include non-persistent memory in computer readable media, random access memory (RAM), and / or non-volatile memory, etc. in the form of read-only memory (ROM) or flash memory (flash RAM). The memory is an example of the computer readable medium.

[0175] Computer-readable media includes permanent and non-permanent, movable and non-movable storage media. Storage media can be implemented by any method or technology to store information, which can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0176] Finally, it should be noted that the above various embodiments are only used to illustrate the technical solutions of the present application, and are not limited to them. Although the present application has been described in detail with reference to the foregoing various embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing various embodiments, or make equivalent substitutions for part or all of the technical features, without departing from the scope of the technical solutions of the various embodiments of the present application.

Claims

1. A video restoration method, characterized in that, include: Obtain the first and second reference frames of the video frame to be repaired; In the time domain, the first reference frame is located before the video frame to be repaired, and the second reference frame is located after the video frame to be repaired; Based on the first reference frame and the second reference frame, the first repair result of the video frame to be repaired is obtained by an interpolation algorithm; Obtain the forward optical flow from the first reference frame to the second reference frame and the reverse optical flow from the second reference frame to the first reference frame; Based on the first reference frame, the second reference frame, the forward optical flow, and the reverse optical flow, obtain the second repair result of the video frame to be repaired; Based on the optical flow consistency error between the forward optical flow and the reverse optical flow, a first reliability weight of the first repair result and a second reliability weight of the second repair result are obtained. The optical flow consistency error is positively correlated with the first reliability weight and negatively correlated with the second reliability weight. The second repair result is weighted and fused with the first repair result based on the first reliability weight and the second reliability weight to obtain the target repair result of the video frame to be repaired.

2. The method according to claim 1, characterized in that, The step of obtaining a first reliability weight for the first repair result and a second reliability weight for the second repair result based on the optical flow consistency error between the forward optical flow and the reverse optical flow includes: The second reliability weight is obtained based on the optical flow consistency error and the reliability weight calculation formula; the reliability weight calculation formula is: Where W(x,y) is the second reliability weight, E flow (x,y) represents the optical flow consistency error, and σ is a constant.

3. The method according to claim 1, characterized in that, Before obtaining the first reliability weight of the first repair result and the second reliability weight of the second repair result based on the optical flow consistency error between the forward optical flow and the reverse optical flow, the method further includes: The forward predicted pixel coordinates of each pixel are obtained based on the forward optical flow; The inverse predicted pixel coordinates of each pixel are obtained based on the inverse optical flow. The optical flow consistency error is obtained based on the forward and backward predicted pixel coordinates of each pixel.

4. The method according to claim 1, characterized in that, The step of obtaining the first repair result of the video frame to be repaired based on the first reference frame and the second reference frame through an interpolation algorithm includes: Obtain the first pixel value and the second pixel value corresponding to each pixel point of the first repair result; the first pixel value corresponding to any pixel point is the pixel value of the pixel point with the same pixel coordinates in the first reference frame, and the second pixel value corresponding to any pixel point is the pixel value of the pixel point with the same pixel coordinates in the second reference frame; The first pixel value and the second pixel value corresponding to each pixel of the first repair result are weighted and summed based on a preset weight coefficient to obtain the pixel value of each pixel of the first repair result. The first repair result is obtained based on the pixel values ​​of each pixel in the first repair result.

5. The method according to claim 1, characterized in that, The step of obtaining the second repair result of the video frame to be repaired based on the first reference frame, the second reference frame, the forward optical flow, and the reverse optical flow includes: Based on the forward optical flow and the reverse optical flow, obtain the forward motion compensation information and the reverse motion compensation information of the video frame to be repaired. Based on the positive motion compensation information, motion compensation is performed on the first reference frame to obtain the positive compensation result; Based on the reverse motion compensation information, motion compensation is performed on the second reference frame to obtain the reverse compensation result; By combining the positive compensation result and the negative compensation result, a second repair result is obtained.

6. The method according to claim 5, characterized in that, The step of obtaining the forward motion compensation information and reverse motion compensation information of the video frame to be repaired based on the forward optical flow and the reverse optical flow respectively includes: The first coefficient is obtained based on the interval between the video frame to be repaired and the first reference frame; Calculate the product of the forward optical flow and the first coefficient to obtain the forward motion compensation information; The second coefficient is obtained based on the interval between the video frame to be repaired and the second reference frame; Calculate the product of the reverse optical flow and the second coefficient to obtain the reverse motion compensation information.

7. The method according to claim 5, characterized in that, The process of fusing the positive compensation result and the reverse compensation result to obtain the second repair result includes: Calculate the average pixel value of pixels with the same pixel coordinates in the positive compensation result and the negative compensation result to obtain the pixel value of each pixel in the second repair result; The second repair result is obtained based on the pixel values ​​of each pixel in the second repair result.

8. A video restoration device, characterized in that, include: The reference frame acquisition unit is used to acquire the first and second reference frames of the video frame to be repaired. In the time domain, the first reference frame is located before the video frame to be repaired, and the second reference frame is located after the video frame to be repaired; The first repair unit is used to obtain a first repair result of the video frame to be repaired based on the first reference frame and the second reference frame by using an interpolation algorithm. An optical flow acquisition unit is used to acquire the forward optical flow from the first reference frame to the second reference frame and the reverse optical flow from the second reference frame to the first reference frame. The second repair unit is used to obtain a second repair result of the video frame to be repaired based on the first reference frame, the second reference frame, the forward optical flow, and the reverse optical flow. The weight determination unit is used to obtain a first reliability weight of the first repair result and a second reliability weight of the second repair result based on the optical flow consistency error between the forward optical flow and the reverse optical flow, wherein the optical flow consistency error is positively correlated with the first reliability weight and negatively correlated with the second reliability weight; The weighted fusion unit is used to weight and fuse the second repair result with the first repair result according to the first reliability weight and the second reliability weight to obtain the target repair result of the video frame to be repaired.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the video restoration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the video restoration method according to any one of claims 1 to 7.

Images