Video restoration method, device, equipment, medium and program product

Abstract

The invention discloses a video restoration method, device and equipment, a medium and a program product, and relates to the technical field of audio and video. The video restoration method comprises the following steps: extracting an initial optical flow between continuous frames in a video to be restored through a lightweight optical flow extraction network; determining a mask sequence corresponding to each frame of image based on the position of the to-be-repaired area in each frame of image of the to-be-repaired video; inputting the initial optical flow and the mask sequence into an optical flow prediction model to obtain predicted optical flow information output by the optical flow prediction model; the optical flow prediction model comprises at least one layer of depth flow completion sub-network; and based on the to-be-repaired video, guiding the pixels to carry out pixel propagation among the video frames according to the predicted optical flow information to obtain a repaired video. According to the technical scheme of the embodiment of the invention, the video restoration efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of audio and video technology, and in particular to a video repair method, device, equipment, medium and program product. Background Art

[0002] With the rapid development of Internet technology, people can shoot videos at will and share them on the Internet. In the process of video shooting, it is inevitable to shoot some materials that you do not want to appear in the video, which requires the use of video restoration methods.

[0003] Commonly used video restoration methods include non-learning restoration and learning restoration. Non-learning video restoration methods are difficult to ensure the continuity of video frames. Learning restoration using deep learning has made some improvements in ensuring the continuity of video frames, but due to the large amount of calculation, it requires a longer waiting time. Summary of the invention

[0004] The present invention provides a video repair method, device, equipment, medium and program product to solve the problem of large amount of calculation and long time consumption in video repair.

[0005] According to one aspect of the present invention, there is provided a video repair method, comprising:

[0006] Through a lightweight optical flow extraction network, the initial optical flow between consecutive frames in the video to be repaired is extracted;

[0007] Based on the position of the to-be-repaired area in each frame of the video to be repaired, determining a mask sequence corresponding to each frame of the video to be repaired;

[0008] Inputting the initial optical flow and the mask sequence into an optical flow prediction model to obtain predicted optical flow information output by the optical flow prediction model; the optical flow prediction model comprises at least one layer of deep flow completion subnetwork;

[0009] Based on the video to be repaired, pixels are guided to perform pixel propagation between video frames according to the predicted optical flow information to obtain a repaired video.

[0010] According to another aspect of the present invention, there is provided a video restoration device, comprising:

[0011] An initial optical flow extraction module is used to extract the initial optical flow between consecutive frames in the video to be repaired through a lightweight optical flow extraction network;

[0012] A mask sequence determination module, used to determine a mask sequence corresponding to each frame of the video to be repaired based on the position of the area to be repaired in each frame of the video to be repaired;

[0013] An optical flow prediction module, used for inputting the initial optical flow and the mask sequence into an optical flow prediction model to obtain predicted optical flow information output by the optical flow prediction model; the optical flow prediction model comprises at least one layer of deep flow completion subnetwork;

[0014] The video restoration module is used to guide pixels to propagate between video frames based on the video to be restored and the predicted optical flow information to obtain a restored video.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video repair method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video repair method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the video repair method of any embodiment of the present disclosure is implemented.

[0021] The technical solution of the embodiment of the present invention extracts the initial optical flow between consecutive frames in the video to be repaired through a lightweight optical flow extraction network, and determines the mask sequence corresponding to each frame of the video to be repaired based on the position of the area to be repaired in each frame of the video to be repaired, and then inputs the initial optical flow and the mask sequence into the optical flow prediction model to obtain the predicted optical flow information output by the optical flow prediction model, and based on the video to be repaired, guides pixels to propagate between video frames according to the predicted optical flow information to obtain the repaired video, and performs initial optical flow extraction through a lightweight optical flow extraction network, which can reduce the amount of calculation, thereby improving the speed and efficiency of optical flow prediction.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a video repair method provided according to Embodiment 1 of the present invention;

[0025] Figure 2a is a schematic diagram of the structure of an optical flow prediction model provided according to Embodiment 2 of the present invention;

[0026] Figure 2b is a flow chart of a video repair method provided according to Embodiment 2 of the present invention;

[0027] Figure 3 is a structural schematic diagram of a video restoration device provided according to Embodiment 3 of the present invention;

[0028] Figure 4 It is a structural schematic diagram of an electronic device for implementing the video repair method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1A flowchart of a video restoration method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a video is restored by performing optical flow prediction through a depth flow completion subnetwork. The method can be executed by a video restoration device. The video restoration device can be implemented in the form of hardware and / or software. The video restoration device can be configured in various general-purpose computing devices. Figure 1 As shown, the method includes:

[0033] S110, extracting the initial optical flow between consecutive frames in the video to be repaired through a lightweight optical flow extraction network.

[0034] The lightweight optical flow extraction network is used to extract the initial optical flow between consecutive frames in the video to be repaired. Specifically, the lightweight optical flow extraction network is composed of two compact sub-networks for pyramid feature extraction and optical flow estimation. Exemplarily, the lightweight optical flow extraction network is LiteFlowNet, which includes two compact sub-networks NetC and NetE, where NetC is used to perform pyramid feature extraction on every two consecutive frames in the input video to obtain high-dimensional pyramid features, and NetE is used to estimate the initial optical flow based on the pyramid features.

[0035] The video to be repaired is the original business processing video, which contains people who are not related to the current business processing. In the video repair process, the irrelevant people need to be eliminated without affecting the readability of the business processing video. Since the business processing video often has a very large amount of data, video repair frame by frame often requires a huge amount of computing and a long waiting time.

[0036] In the embodiment of the present invention, in order to improve the efficiency of video restoration, a lightweight optical flow extraction network is used to extract the initial optical flow between every two consecutive frames in the video to be restored. Specifically, the video to be restored is first divided into consecutive frames to obtain image pairs consisting of every two consecutive frames. A lightweight optical flow extraction network is used to extract the optical flow of each image pair to obtain the initial optical flow corresponding to each image pair.

[0037] In a specific example, the video to be repaired is divided into continuous frames, and each two continuous frames constitute an image pair. Multiple image pairs in the video to be repaired are input into the lightweight optical flow extraction network LiteFlowNet. For each image pair, NetC in LiteFlowNet performs pyramid feature extraction on the input image pair to obtain high-dimensional pyramid features. Then NetE in LiteFlowNet estimates the initial optical flow based on the pyramid features.

[0038] S120 , determining a mask sequence corresponding to each frame of the video to be repaired based on the position of the region to be repaired in each frame of the video to be repaired.

[0039] The area to be repaired is the area where personnel irrelevant to the business processing are located contained in each frame of the business processing video.

[0040] In an embodiment of the present invention, based on the location of the area to be repaired in each frame of the video to be repaired, a mask sequence corresponding to each frame is determined. The area to be repaired in each frame can be extracted through the mask sequence. Exemplarily, a rectangular mask can be determined through the mask sequence, and the rectangular mask can be the minimum circumscribed rectangle that can cover the area to be repaired.

[0041] S130, inputting the initial optical flow and mask sequence into the optical flow prediction model to obtain the predicted optical flow information output by the optical flow prediction model; the optical flow prediction model includes at least one layer of deep flow completion subnetwork.

[0042] The optical flow prediction model is a model that includes at least one layer of deep flow completion subnetwork, wherein the deep flow completion subnetwork (Deep Flow Completion Subnetwork, referred to as DFC-S) is used to perform optical flow prediction of the area to be repaired based on the input mask sequence and optical flow.

[0043] In an embodiment of the present invention, the initial optical flow and mask sequence are first input into the first-layer deep flow completion subnetwork of the optical flow prediction model. Since the smaller the area of ​​the region to be predicted occupies in each frame of the image, the more accurate the prediction will be, therefore, the prediction range can be increased in sequence by using multiple layers of deep flow completion subnetworks in the optical flow prediction model. For example, the first layer predicts 50% of the area to be repaired, the second layer predicts 75% of the area to be repaired, and so on, until the last layer predicts the entire area to be repaired. Specifically, the first-layer deep flow completion subnetwork generates a corresponding first mask (used to extract a part of the area to be repaired) according to the mask sequence, and then makes a preliminary prediction of the area covered by the first mask according to the initial optical flow to obtain forward optical flow information and backward optical flow information.

[0044] Furthermore, the forward optical flow information and backward optical flow information output by the first layer of the deep flow completion subnetwork, as well as the mask sequence, are used as the input of the next layer of the deep flow completion subnetwork. The next layer of the deep flow completes the optical flow prediction of a larger area until the last layer of the deep flow completion subnetwork predicts the predicted optical flow information of the complete area to be repaired. The predicted optical flow information includes forward optical flow information and backward optical flow information.

[0045] S140 . Based on the video to be repaired, guide pixels to propagate between video frames according to the predicted optical flow information to obtain a repaired video.

[0046] In an embodiment of the present invention, in the video to be repaired, pixels are guided to perform forward and reverse pixel propagation between video frames according to the predicted optical flow information, and the area to be repaired in each frame of the video to be repaired is repaired to obtain a repaired video.

[0047] The technical solution of the embodiment of the present invention extracts the initial optical flow between consecutive frames in the video to be repaired through a lightweight optical flow extraction network, and determines the mask sequence corresponding to each frame of the video to be repaired based on the position of the area to be repaired in each frame of the video to be repaired, and then inputs the initial optical flow and the mask sequence into the optical flow prediction model to obtain the predicted optical flow information output by the optical flow prediction model, and based on the video to be repaired, guides pixels to propagate between video frames according to the predicted optical flow information to obtain the repaired video, and performs initial optical flow extraction through a lightweight optical flow extraction network, which can reduce the amount of calculation, thereby improving the speed and efficiency of optical flow prediction.

[0048] Embodiment 2

[0049] Figure 2a This is a flowchart of a video restoration method provided in Embodiment 2 of the present invention. This embodiment further refines the above embodiment and provides specific steps of extracting the initial optical flow between consecutive frames in the video to be restored through a lightweight optical flow extraction network, and guiding pixels to propagate between video frames based on the predicted optical flow information to obtain the restored video. Figure 2a As shown, the method includes:

[0050] S210 , performing pyramid feature extraction on consecutive frames of the video to be restored through the first sub-network in the lightweight optical flow extraction network to obtain pyramid features corresponding to the consecutive frames.

[0051] In the embodiment of the present invention, the first sub-network in the lightweight optical flow extraction network performs pyramid feature extraction on every two consecutive frames in the video to be restored to obtain pyramid features corresponding to the consecutive frames. Specifically, the first sub-network extracts two multi-scale high-dimensional pyramid features for any input image pair.

[0052] S220. Perform feature distortion on the pyramid features through the second sub-network in the lightweight optical flow extraction network to extract the initial optical flow between consecutive frames in the video to be repaired.

[0053] In the embodiment of the present invention, the second sub-network in the lightweight optical flow extraction network is used to perform feature distortion on the pyramid features to extract the initial optical flow between consecutive frames in the video to be restored. These different layers are convolved from the original image, which can save a lot of time compared to the previous multi-layer network, thereby improving the efficiency of video restoration.

[0054] S230 , determining a mask sequence corresponding to each frame of the video to be repaired based on the position of the region to be repaired in each frame of the video to be repaired.

[0055] S240, inputting the initial optical flow and the mask sequence into the optical flow prediction model to obtain the predicted optical flow information output by the optical flow prediction model; the optical flow prediction model includes at least one layer of deep flow completion subnetwork.

[0056] Optionally, the mask used by each layer of the deep flow completion subnetwork in the optical flow prediction model is determined according to the mask sequence, and the mask size increases successively, and the mask of the last layer of the deep flow completion subnetwork contains the area to be repaired.

[0057] In this optional embodiment, the mask used by each layer of the deep flow completion subnetwork in the optical flow prediction model is determined according to the mask sequence, and the mask size increases successively until the mask of the last layer of the deep flow completion subnetwork contains all the areas to be repaired. By performing optical flow prediction of the area to be repaired from a small range to a large range through multiple layers of deep flow completion subnetworks, each layer of prediction can be optimized based on the prediction of the previous layer, ensuring the reliability of the optical flow prediction.

[0058] Optionally, each layer of the depth flow completion subnetwork in the optical flow prediction model performs the following operations:

[0059] Determine the size of the currently used mask according to the mask sequence and the arrangement position in the optical flow prediction model;

[0060] Based on the forward optical flow information and backward optical flow information output by the sub-network of the previous layer of depth flow completion, forward and backward optical flow predictions are performed on the mask covered area.

[0061] In this optional embodiment, the operation of each layer of the depth flow completion subnetwork in the optical flow prediction model is provided: according to the mask sequence and the arrangement position in the optical flow prediction model, the size of the currently used mask is determined. Among them, the mask size is larger the later the arrangement, and the mask size used by the last layer of the depth flow completion subnetwork is the same as that of the entire area to be repaired.

[0062] Furthermore, based on the forward optical flow information and backward optical flow information output by the previous layer of the deep flow completion subnetwork, forward and backward optical flow predictions are performed on the mask-covered area. By using multiple layers of deep flow completion subnetworks to perform optical flow predictions using gradually larger masks, the optical flow details can be captured layer by layer. After each layer of the deep flow completion subnetwork, the detailed information of the previous deep flow completion subnetwork will be supplemented to align the flow field, and each layer of prediction is performed forward and backward simultaneously, so that forward and backward optical flow information can be obtained, achieving better optical flow prediction results.

[0063] In a specific example, the structure of the optical flow prediction model is as follows Figure 2bAs shown in the figure, it contains three layers of deep flow completion subnetwork. First, the initial optical flow between consecutive frames in the video to be repaired is And the mask sequence corresponding to each frame image {M (i-k) ,...,M i ,...,M (i+k)} Input to the first layer of the deep flow completion subnetwork of the optical flow prediction model, where Represents the initial optical flow between the i-th frame and the i+1-th frame, and so on. i Represents the mask sequence corresponding to the i-th frame image, and the mask sequence corresponds to the initial mask that can cover all the to-be-repaired areas of the current frame image.

[0064] The first-layer deep flow completion subnetwork determines that the current mask size is 1 / 2 of the initial mask based on the mask sequence and the arrangement position in the optical flow prediction model. Then, based on the initial optical flow, forward and backward optical flow predictions are performed on the mask coverage area to obtain the first forward optical flow information and the first backward optical flow information, and the first forward optical flow information is converted into the first optical flow information. And the first backward optical flow information The mask sequence is input to the second layer of the deep stream to complete the sub-network.

[0065] The second-layer deep flow completion subnetwork determines that the current mask size is 2 / 3 of the initial mask based on the mask sequence and the arrangement position in the optical flow prediction model, and then performs forward and backward optical flow prediction on the mask coverage area based on the first forward optical flow information and the first backward optical flow information to obtain the second forward optical flow information and the second backward optical flow information. The second forward optical flow information and the second backward optical flow information and the mask sequence are input into the third-layer deep flow completion network.

[0066] The third-layer deep flow completion subnetwork determines that the current mask size is the initial mask according to the mask sequence and the arrangement position in the optical flow prediction model, which can cover all the areas to be repaired, and then performs forward and backward optical flow predictions on the mask-covered area based on the second forward optical flow information and the second backward optical flow information, and obtains the third forward optical flow information and the third backward optical flow information as the final predicted optical flow information.

[0067] During the training of the optical flow field, missing areas are randomly generated in each frame and compared with the true value to find the minimum loss value L. Each time the deep flow in the optical flow prediction model completes the subnetwork, the L value is calculated and optimized continuously. The specific algorithm for the L value is as follows:

[0068]

[0069] Among them, f i is the true optical flow of the i-th frame, is the predicted optical flow of the i-th frame, M is the mask sequence, and Θ represents element-wise multiplication (Hadamard product).

[0070] This method based on optical flow and continuously optimizing similarity can handle changes in smooth areas, but if there is an area to be repaired at the boundary area, the final effect will be much worse than the ideal value, and the three-layer subnet operation will continue to expand this shortcoming. In order to overcome this problem, when predicting the optical flow of the area to be repaired, it is encouraged to generate relatively different boundary values ​​and sort them in descending order according to the loss value L. Pixels located at the edge will receive more attention, and a weight value λ is added. In order to make the final result more inclined to produce a reasonable large change value, the algorithm for the new L value is as follows:

[0071]

[0072] Among them, f i is the true optical flow of the i-th frame, is the predicted optical flow of the i-th frame, M is the mask sequence, and M h It is the mask sequence corresponding to the highest weight of the loss value, λ is used to adjust the weight values ​​of the two losses, and Θ represents element-wise multiplication.

[0073] S250 , for a set pixel point in the original frame of the video to be repaired, forward propagate the set pixel point according to the predicted optical flow information to obtain a predicted position of the set pixel point in the previous frame.

[0074] In the embodiment of the present invention, in order to verify the reliability of the predicted optical flow information, for the set pixel point in the original frame of the video to be repaired, the set pixel point is forward propagated according to the predicted optical flow information to obtain the predicted position of the set pixel point in the previous frame, so as to reduce the position error between the predicted position and the actual position of the set pixel point in the previous frame.

[0075] S260, calculating the predicted position error according to the actual position and predicted position of the set pixel point in the previous frame, if the predicted position error is less than the set threshold, executing S270, otherwise, executing S280.

[0076] In the embodiment of the present invention, the error between the actual position and the predicted position of the set pixel point in the previous frame is calculated as the predicted position error, and the predicted position error determination calculation formula is as follows:

[0077]

[0078] Among them, X i-1 is the position of the set pixel in the original frame (i.e., the i+1th frame), According to the predicted optical flow information, the set pixel point is forward propagated to obtain the predicted position of the set pixel point in the previous frame (i.e., the i-th frame), X i is the actual position of the set pixel in the previous frame, and ε is the set threshold.

[0079] The predicted position error can characterize the rationality of the predicted optical flow information. If the predicted position error is less than the set threshold, the video restoration operation in S270 is performed. By using a lightweight optical flow extraction network to extract the initial optical flow of the video to be restored, and performing an optical flow prediction layer on the area to be restored based on the initial optical flow through the optical flow prediction model of the deep flow completion subnetwork, the computational efficiency can be improved, and a balance between the video restoration effect and the video restoration time can be achieved.

[0080] S270: Based on the video to be repaired, guide pixels to propagate between video frames according to the predicted optical flow information to obtain a repaired video.

[0081] S280, extracting the initial optical flow between consecutive frames in the video to be repaired through a high-precision optical flow extraction network, and returning to execute S240.

[0082] Among them, the high-precision optical flow extraction network includes an image feature extraction subnetwork and an optical flow estimation subnetwork.

[0083] The high-precision optical flow extraction network is used to extract the initial optical flow between consecutive frames in the video to be repaired. The high-precision optical flow extraction network can extract richer optical flow information than the lightweight optical flow extraction network. The high-precision optical flow extraction network includes an image feature extraction subnetwork and an optical flow estimation subnetwork, wherein the image feature extraction subnetwork is used to extract image features of consecutive frames in the video to be repaired, and the optical flow estimation subnetwork is used to further predict optical flow information based on image features. Exemplarily, the high-precision optical flow extraction network is FlowNet2.

[0084] In the embodiment of the present invention, when the predicted position error is greater than the set threshold, it indicates that the reliability of the current optical flow prediction is low. To ensure the video restoration effect, it is necessary to continue to extract the initial optical flow between consecutive frames in the video to be restored through the high-precision optical flow extraction network. And return to execute the operation of inputting the initial optical flow and mask sequence into the optical flow prediction model to obtain the predicted optical flow information output by the optical flow prediction model, until the pixels are guided to propagate between video frames based on the predicted optical flow information based on the video to be restored, and the restored video is obtained. When the predicted position error is large, the initial optical flow extraction is performed again through the high-precision optical flow extraction network mechanical energy, so that more optical flow information can be obtained to ensure the video restoration effect.

[0085] The technical solution of the embodiment of the present invention is that before the restored video is obtained by guiding the pixels to propagate between video frames according to the predicted optical flow information based on the video to be restored, the set pixel points in the original frame of the video to be restored are forward propagated according to the predicted optical flow information to obtain the predicted position of the set pixel points in the previous frame, and the predicted position error is calculated according to the actual position and the predicted position of the set pixel points in the previous frame. When the predicted position error is less than the set threshold, it indicates that the predicted optical flow information is reasonable, and the operation of guiding the pixels to propagate between video frames according to the predicted optical flow information based on the video to be restored to obtain the restored video can be continued; when the predicted position error is greater than the set threshold, it indicates that the predicted optical flow information is unreasonable, and the initial optical flow between consecutive frames in the video to be restored is extracted through a high-precision optical flow extraction network, and the initial optical flow and the mask sequence are input into the optical flow prediction model to obtain the predicted optical flow information output by the optical flow prediction model, until the restored video is obtained, and the optical flow estimation accuracy is improved by extracting the high-precision initial optical flow, thereby ensuring the video restoration effect.

[0086] Embodiment 3

[0087] Figure 3 FIG. 1 is a schematic diagram of the structure of a video repair device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0088] An initial optical flow extraction module 310 is used to extract initial optical flows between consecutive frames in the video to be repaired through a lightweight optical flow extraction network;

[0089] A mask sequence determination module 320, configured to determine a mask sequence corresponding to each frame of the video to be repaired based on the position of the region to be repaired in each frame of the video to be repaired;

[0090] An optical flow prediction module 330, configured to input the initial optical flow and the mask sequence into an optical flow prediction model to obtain predicted optical flow information output by the optical flow prediction model; the optical flow prediction model comprises at least one layer of deep flow completion subnetwork;

[0091] The video restoration module 340 is used to guide pixels to propagate between video frames based on the video to be restored and according to the predicted optical flow information to obtain a restored video.

[0092] The technical solution of the embodiment of the present invention extracts the initial optical flow between consecutive frames in the video to be repaired through a lightweight optical flow extraction network, and determines the mask sequence corresponding to each frame of the video to be repaired based on the position of the area to be repaired in each frame of the video to be repaired, and then inputs the initial optical flow and the mask sequence into the optical flow prediction model to obtain the predicted optical flow information output by the optical flow prediction model, and based on the video to be repaired, guides pixels to propagate between video frames according to the predicted optical flow information to obtain the repaired video, and performs initial optical flow extraction through a lightweight optical flow extraction network, which can reduce the amount of calculation, thereby improving the speed and efficiency of optical flow prediction.

[0093] Optionally, the initial optical flow extraction module 310 is specifically used for:

[0094] Performing pyramid feature extraction on continuous frames of the video to be restored through the first sub-network in the lightweight optical flow extraction network to obtain pyramid features corresponding to the continuous frames;

[0095] The pyramid features are feature distorted by the second sub-network in the lightweight optical flow extraction network to extract the initial optical flow between consecutive frames in the video to be repaired.

[0096] Optionally, the video repair device also includes:

[0097] A predicted position determination module, for guiding pixels to perform pixel propagation between video frames according to the predicted optical flow information based on the video to be repaired, and before obtaining the repaired video, for a set pixel point in the original frame of the video to be repaired, forward propagating the set pixel point according to the predicted optical flow information to obtain a predicted position of the set pixel point in the previous frame;

[0098] A position error calculation module, used to calculate the predicted position error according to the actual position of the set pixel point in the previous frame and the predicted position;

[0099] The first video restoration execution module is used to execute an operation based on the video to be restored, guiding pixels to propagate between video frames according to the predicted optical flow information, to obtain a restored video when the predicted position error is less than a set threshold.

[0100] Optionally, the video repair device also includes:

[0101] a second video restoration execution module, configured to extract, when the predicted position error is greater than a set threshold, an initial optical flow between consecutive frames in the video to be restored through a high-precision optical flow extraction network, and return to execute an operation of inputting the initial optical flow and the mask sequence into an optical flow prediction model to obtain predicted optical flow information output by the optical flow prediction model, until a restored video is obtained;

[0102] Among them, the high-precision optical flow extraction network includes an image feature extraction subnetwork and an optical flow estimation subnetwork.

[0103] Optionally, the mask used by each layer of the depth flow completion subnetwork in the optical flow prediction model is determined according to the mask sequence, and the mask size increases successively, and the mask of the last layer of the depth flow completion subnetwork includes the area to be repaired.

[0104] Optionally, the optical flow prediction module 330 includes:

[0105] A mask size determination unit, used to determine the size of the currently used mask according to the mask sequence and the arrangement position in the optical flow prediction model;

[0106] The optical flow prediction unit is used to perform forward and backward optical flow prediction on the mask coverage area based on the forward optical flow information and backward optical flow information output by the sub-network of the previous layer of depth flow completion.

[0107] The video repair device provided in the embodiment of the present invention can execute the video repair method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0108] In the technical solution of the present invention, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0109] Embodiment 4

[0110] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0111] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, applicators, blade applicators, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0112] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0113] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0114] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a video restoration method.

[0115] In some embodiments, the video repair method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the video repair method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the video repair method in any other appropriate manner (e.g., by means of firmware).

[0116] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or application.

[0118] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes a backend component (e.g., as a data applier), or a computing system that includes a middleware component (e.g., an application applier), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0121] A computing system may include a client and an application server. The client and the application server are generally remote from each other and usually interact through a communication network. The relationship between the client and the application server is generated by computer programs running on corresponding computers and having a client-application server relationship with each other. The application server may be a cloud application server, also known as a cloud computing application server or a cloud host, which is a host product in a cloud computing application system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS applications.

[0122] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0123] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video repair method, characterized in that: include: Through a lightweight optical flow extraction network, the initial optical flow between consecutive frames in the video to be repaired is extracted; Based on the position of the to-be-repaired area in each frame of the video to be repaired, determining a mask sequence corresponding to each frame of the video to be repaired; Inputting the initial optical flow and the mask sequence into an optical flow prediction model to obtain predicted optical flow information output by the optical flow prediction model; the optical flow prediction model comprises at least one layer of deep flow completion subnetwork; Based on the video to be repaired, pixels are guided to perform pixel propagation between video frames according to the predicted optical flow information to obtain a repaired video.

2. The method according to claim 1, characterized in that The initial optical flow between consecutive frames in the video to be repaired is extracted through a lightweight optical flow extraction network, including: Performing pyramid feature extraction on continuous frames of the video to be restored through the first sub-network in the lightweight optical flow extraction network to obtain pyramid features corresponding to the continuous frames; The pyramid features are feature distorted by the second sub-network in the lightweight optical flow extraction network to extract the initial optical flow between consecutive frames in the video to be repaired.

3. The method according to claim 1, characterized in that Based on the video to be repaired, guiding the pixels to propagate between video frames according to the predicted optical flow information to obtain a repaired video, the method further includes: For a set pixel point in the original frame of the video to be restored, forward propagation is performed on the set pixel point according to the predicted optical flow information to obtain a predicted position of the set pixel point in the previous frame; Calculating a predicted position error according to the actual position of the set pixel point in the previous frame and the predicted position; When the predicted position error is less than a set threshold, an operation is performed based on the video to be repaired, guiding pixels to propagate between video frames according to the predicted optical flow information, to obtain a repaired video.

4. The method according to claim 3, characterized in that Also includes: When the predicted position error is greater than a set threshold, the initial optical flow between consecutive frames in the video to be repaired is extracted through a high-precision optical flow extraction network, and the initial optical flow and the mask sequence are input into the optical flow prediction model to obtain the predicted optical flow information output by the optical flow prediction model until the repaired video is obtained; Among them, the high-precision optical flow extraction network includes an image feature extraction subnetwork and an optical flow estimation subnetwork.

5. The method according to claim 1, characterized in that The mask used by each layer of the deep flow completion subnetwork in the optical flow prediction model is determined according to the mask sequence, and the mask size increases successively, and the mask of the last layer of the deep flow completion subnetwork contains the area to be repaired.

6. The method according to claim 5, characterized in that Each layer of the deep flow completion subnetwork in the optical flow prediction model performs the following operations: Determining the size of the currently used mask according to the mask sequence and the arrangement position in the optical flow prediction model; Based on the forward optical flow information and backward optical flow information output by the previous layer of depth flow completion sub-network, forward and backward optical flow predictions are performed on the mask covered area.

7. A video restoration device, characterized in that: include: An initial optical flow extraction module is used to extract the initial optical flow between consecutive frames in the video to be repaired through a lightweight optical flow extraction network; A mask sequence determination module, used to determine a mask sequence corresponding to each frame of the video to be repaired based on the position of the area to be repaired in each frame of the video to be repaired; An optical flow prediction module, used for inputting the initial optical flow and the mask sequence into an optical flow prediction model to obtain predicted optical flow information output by the optical flow prediction model; the optical flow prediction model comprises at least one layer of deep flow completion subnetwork; The video restoration module is used to guide pixels to propagate between video frames based on the video to be restored and the predicted optical flow information to obtain a restored video.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video repair method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video repair method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the video restoration method according to any one of claims 1 to 6.

Images