A video restoration method, related device, equipment and storage medium

Abstract

The application discloses a video repairing method based on artificial intelligence, and application scenarios at least include various terminals, such as mobile phones, computers, vehicle-mounted terminals and the like. The application comprises the following steps: acquiring a video sample sequence; acquiring a target mask sample sequence according to the video sample sequence; acquiring an optical flow data sequence according to the video sample sequence; performing clustering processing on pixel points included in a target mask region in each target mask frame based on each optical flow data in the optical flow data sequence, to obtain an optical flow clustering result of each target mask frame; determining an optical flow quality score according to the optical flow clustering result of each target mask frame; and adopting a video repairing mode matched with the optical flow quality score to perform repairing processing on a video to be repaired. The application also provides related devices. The application takes the optical flow quality as a basis for selecting a video repairing mode, achieves the purpose of taking the advantages and making up the shortcomings among different video repairing modes, and is thus favorable for obtaining a video picture with better repairing effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, related apparatus, equipment, and storage medium for video restoration. Background Art

[0002] Video inpainting is a task that aims to fill in missing areas in video frames with reasonable content. It mainly uses information from unobstructed areas in the video to repair obscured areas. For example, it can repair damaged videos, remove unwanted objects, reposition videos, and repair underexposed images.

[0003] Currently, there are two main types of video restoration techniques. One uses optical flow propagation and image inpainting, which first propagates available pixels to the corresponding area through optical flow, and then uses image inpainting to fill in isolated pixel blocks. The other uses an end-to-end neural network approach, using a generative model to fill in occluded areas.

[0004] However, the inventors discovered that existing solutions have at least the following issues: While optical flow infilling provides high clarity, it relies heavily on optical flow, which is susceptible to interference and can be inaccurate, making it prone to distortion and incorrect infilling. End-to-end generation methods consider semantic information and generally avoid distortion and serious errors. However, complex backgrounds can easily lead to blurred infilling. Summary of the Invention

[0005] The present invention provides a video restoration method, related apparatus, device, and storage medium. The present invention uses optical flow quality as the basis for selecting a video restoration method, achieving the goal of leveraging the strengths and weaknesses of different video restoration methods, thereby facilitating the acquisition of better restored video images.

[0006] In view of this, the present application provides a method for video restoration, comprising:

[0007] Obtain a video sample sequence for the video to be repaired, wherein the video sample sequence includes K video frame pairs, each video frame pair includes two adjacent video frames, and K is an integer greater than or equal to 1;

[0008] Obtaining a target mask sample sequence according to the video sample sequence, wherein the target mask sample sequence includes K target mask frames, each target mask frame includes a target mask area obtained by expanding an original mask area, and each target mask frame has a corresponding relationship with each video frame pair;

[0009] Obtain an optical flow data sequence according to the video sample sequence, wherein the optical flow data sequence includes K optical flow data, and each optical flow data has a corresponding relationship with each video frame pair;

[0010] Based on each optical flow data in the optical flow data sequence, clustering processing is performed on the pixel points included in the target mask area in each target mask frame to obtain an optical flow clustering result of each target mask frame;

[0011] Determine the optical flow quality score based on the optical flow clustering results of each target mask frame;

[0012] The video to be repaired is repaired using a video repair method that matches the optical flow quality score.

[0013] Another aspect of the present application provides a video restoration device, comprising:

[0014] An acquisition module, configured to acquire a video sample sequence for a video to be repaired, wherein the video sample sequence includes K video frame pairs, each video frame pair includes two adjacent video frames, and K is an integer greater than or equal to 1;

[0015] The acquisition module is further configured to acquire a target mask sample sequence according to the video sample sequence, wherein the target mask sample sequence includes K target mask frames, each target mask frame includes a target mask area obtained by expanding an original mask area, and each target mask frame has a corresponding relationship with each video frame pair;

[0016] The acquisition module is further configured to acquire an optical flow data sequence according to the video sample sequence, wherein the optical flow data sequence includes K optical flow data, and each optical flow data has a corresponding relationship with each video frame pair;

[0017] a processing module, configured to perform clustering processing on the pixel points included in the target mask area in each target mask frame based on each optical flow data in the optical flow data sequence, to obtain an optical flow clustering result of each target mask frame;

[0018] A determination module, configured to determine an optical flow quality score based on the optical flow clustering result of each target mask frame;

[0019] The restoration module is used to restore the video to be restored by using a video restoration method that matches the optical flow quality score.

[0020] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0021] an acquisition module, specifically configured to acquire a video sequence from the video to be repaired, wherein the video sequence includes T original video frames, each original video frame displays a target object, and T is an integer greater than 1;

[0022] Extracting K to-be-processed video frame pairs from a video sequence, wherein each to-be-processed video frame pair includes two adjacent original video frames;

[0023] The sizes of the original video frames in each to-be-processed video frame pair are normalized to obtain K video frame pairs, and the K video frame pairs are used as video sample sequences.

[0024] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0025] an acquisition module, specifically configured to acquire, for each video frame pair in the video sample sequence, an original mask frame according to a previous video frame or a next video frame in the video frame pair, wherein the original mask frame includes an original mask area obtained after masking the target object;

[0026] For each video frame pair in the video sample sequence, the original mask region in the original mask frame is expanded to obtain a target mask frame;

[0027] The K target mask frames are used as the target mask sample sequence.

[0028] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0029] an acquisition module, specifically configured to expand an original mask region in an original mask frame according to a first number of pixels for each video frame pair in a video sample sequence to obtain a first mask region;

[0030] For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to a second number of pixels to obtain a second mask region, wherein the second number of pixels is greater than the first number of pixels;

[0031] For each video frame pair in the video sample sequence, an XOR operation is performed on the first mask area and the second mask area to obtain a target mask frame.

[0032] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0033] an acquisition module, specifically configured to acquire, for each video frame pair in the video sample sequence, a first original mask frame based on a first video frame in the video frame pair, and acquire a second original mask frame based on a second video frame in the video frame pair, wherein the first original mask frame and the second original mask frame respectively include original mask regions obtained after masking the target object;

[0034] For each video frame pair in the video sample sequence, performing union processing on the first original mask frame and the second original mask frame to obtain an original mask frame;

[0035] For each video frame pair in the video sample sequence, the original mask region in the original mask frame is expanded to obtain a target mask frame;

[0036] The K target mask frames are used as the target mask sample sequence.

[0037] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0038] an acquisition module, specifically configured to expand an original mask region in an original mask frame according to a first number of pixels for each video frame pair in a video sample sequence to obtain a first mask region;

[0039] For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to a second number of pixels to obtain a second mask region, wherein the second number of pixels is greater than the first number of pixels;

[0040] For each video frame pair in the video sample sequence, an XOR operation is performed on the first mask area and the second mask area to obtain a target mask frame.

[0041] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0042] An acquisition module is specifically configured to determine, for each video frame pair in the video sample sequence, optical flow data based on the horizontal offset and vertical offset of each pixel point in the subsequent video frame relative to each pixel point in the previous video frame;

[0043] Take K optical flow data as the optical flow data sequence;

[0044] or,

[0045] An acquisition module is specifically configured to determine, for each video frame pair in the video sample sequence, optical flow data based on the horizontal offset and vertical offset of each pixel point in the previous video frame relative to each pixel point in the next video frame;

[0046] Take K optical flow data as the optical flow data sequence.

[0047] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0048] a processing module, specifically configured to determine, for each target mask frame, two-dimensional optical flow values ​​of X pixels in the target mask area based on corresponding optical flow data in the optical flow data sequence, where X is an integer greater than 1;

[0049] For each target mask frame, cluster the X pixels according to their two-dimensional optical flow values ​​to obtain the optical flow clustering result.

[0050] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0051] A determination module is specifically used to determine the total number of categories of each target mask frame according to the optical flow clustering result of each target mask frame;

[0052] Count the number of frames whose total number of categories is less than or equal to the category number threshold;

[0053] According to the ratio between the number of frames and the K value, the single category ratio is determined;

[0054] If the single ratio of the category is greater than the ratio threshold, the optical flow quality score is determined to be the first score;

[0055] If the single category ratio is less than or equal to the ratio threshold, the optical flow quality score is determined to be a second score.

[0056] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0057] a determination module, specifically configured to determine, for each target mask frame, an optical flow clustering result, and a moving average of each cluster according to the two-dimensional optical flow value of each pixel in each cluster, wherein the optical flow clustering result is used to determine one or more clusters;

[0058] Based on the optical flow clustering results of each target mask frame, the moving average of the target mask frame is determined according to the moving average of each cluster cluster;

[0059] Accumulate the moving average of each target mask frame to obtain the total moving distance;

[0060] If the total moving distance is greater than or equal to the distance threshold, determining the optical flow quality score to be the first score;

[0061] If the total moving distance is less than the distance threshold, the optical flow quality score is determined to be a second score.

[0062] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0063] A determination module is specifically used to determine the total number of categories of each target mask frame according to the optical flow clustering result of each target mask frame;

[0064] Count the number of frames whose total number of categories is less than or equal to the category number threshold;

[0065] According to the ratio between the number of frames and the K value, the single category ratio is determined;

[0066] For the optical flow clustering results of each target mask frame, a moving average of each cluster is determined according to the two-dimensional optical flow values ​​of each pixel in each cluster, wherein the optical flow clustering results are used to determine one or more clusters;

[0067] Based on the optical flow clustering results of each target mask frame, the moving average of the target mask frame is determined according to the moving average of each cluster cluster;

[0068] Accumulate the moving average of each target mask frame to obtain the total moving distance;

[0069] If the single ratio of the category is greater than the ratio threshold, and the total moving distance is greater than or equal to the distance threshold, then the optical flow quality score is determined to be the first score;

[0070] If the single category ratio is less than or equal to the ratio threshold, and the total movement distance is less than the distance threshold, the optical flow quality score is determined to be the second score.

[0071] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0072] The restoration module is specifically configured to use the optical flow method to restore the video to be restored if the optical flow quality score is the first score.

[0073] If the optical flow quality score is the second score, the neural network is called to perform repair processing on the video to be repaired.

[0074] In one possible design, in another implementation of another aspect of the embodiments of the present application, the video restoration apparatus further includes a display module;

[0075] A display module, configured to display the video to be repaired and a list of repair objects, wherein the list of repair objects includes at least one repairable object;

[0076] The acquisition module is further configured to respond to a selection instruction for a target object and execute a step of acquiring a video sample sequence for the video to be repaired, wherein the target object is at least one repairable object;

[0077] The display module is further configured to adopt a video restoration method that matches the optical flow quality score, repair the video to be restored, and then respond to a play instruction for the restored video to play the restored video.

[0078] On the other hand, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned methods when executing the computer program.

[0079] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned methods when the computer program is executed by a processor.

[0080] Another aspect of the present application provides a computer program product, including a computer program, which implements the above-mentioned methods when executed by a processor.

[0081] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0082] In an embodiment of the present application, a method for video restoration is provided. First, a video sample sequence for the video to be restored is obtained. Then, a target mask sample sequence can be obtained based on the video sample sequence, wherein each target mask frame includes a target mask area obtained by expanding the original mask area. Then, an optical flow data sequence is obtained based on the video sample sequence. Then, based on the individual optical flow data in the optical flow data sequence, the pixels included in the target mask area in each target mask frame are clustered to obtain an optical flow clustering result for each target mask frame. Based on this, an optical flow quality score can be determined based on the optical flow clustering result of each target mask frame, and a video restoration method that matches the optical flow quality score is used to restore the video to be restored. In the above method, the optical flow clustering result of the masked area is used to predict the optical flow quality. When the optical flow quality is good, the optical flow method can be used as a video restoration method to obtain filling content with higher clarity and credibility. When the optical flow quality is poor, a generative model can be used as a video restoration method to obtain a filling effect with higher stability. It can be seen that this application uses optical flow quality as the basis for selecting a video restoration method, so as to achieve the purpose of taking advantage of the strengths and weaknesses of different video restoration methods, thereby facilitating obtaining a video image with a better restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a schematic diagram of the architecture of the video restoration system in an embodiment of the present application;

[0084] Figure 2 This is an effect diagram of video frame filling based on the optical flow method in an embodiment of the present application;

[0085] Figure 3 This is an effect diagram of implementing video frame filling based on the model method in an embodiment of the present application;

[0086] Figure 4 A flowchart of a video repair method in an embodiment of the present application;

[0087] Figure 5 A schematic diagram of generating a target mask frame in an embodiment of the present application;

[0088] Figure 6 Another schematic diagram of generating a target mask frame in an embodiment of the present application;

[0089] Figure 7 This is another schematic diagram of generating a target mask frame in an embodiment of the present application;

[0090] Figure 8 This is another schematic diagram of generating a target mask frame in an embodiment of the present application;

[0091] Figure 9 This is a schematic diagram of determining a two-dimensional optical flow value based on forward optical flow in an embodiment of the present application;

[0092] Figure 10 This is a schematic diagram of determining a two-dimensional optical flow value based on backward optical flow in an embodiment of the present application;

[0093] Figure 11 This is a schematic diagram of the effect of removing a mark based on a video repair application in an embodiment of the present application;

[0094] Figure 12 This is a schematic diagram of the effect of removing subtitles based on a video repair application in an embodiment of the present application;

[0095] Figure 13 This is a schematic diagram of the effect of removing objects based on a video restoration application in an embodiment of the present application;

[0096] Figure 14 Schematic diagram comparing the effects of video frame restoration based on the optical flow method and the model method in an embodiment of the present application;

[0097] Figure 15 A schematic diagram of a video restoration device in an embodiment of the present application;

[0098] Figure 16 This is a schematic diagram of the structure of the terminal in the embodiment of the present application;

[0099] Figure 17 This is a structural diagram of the server in an embodiment of the present application. DETAILED DESCRIPTION

[0100] The present invention provides a video restoration method, related apparatus, device, and storage medium. The present invention uses optical flow quality as the basis for selecting a video restoration method, achieving the goal of leveraging the strengths and weaknesses of different video restoration methods, thereby facilitating the acquisition of better restored video images.

[0101] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0102] With the advent of the multimedia and artificial intelligence (AI) era, video has gradually become the mainstream way of information exchange, and the massive amount of video has posed more challenges to video quality management. Due to certain reasons, the video may have defects. For example, there are mosaic patterns in the video screen, which will affect the user's viewing experience. For example, during the video formation process, there may be station logos or advertising patterns. Based on this, the present application proposes a video repair method, which aims to remove unnecessary objects in the video or restore damaged images.

[0103] The video restoration method specifically involves AI-based computer vision (CV) and machine learning (ML). Specifically, CV technology is used to identify repairable objects (e.g., station logos, subtitles, etc.) in the video. A neural network trained using ML is then used to restore the video.

[0104] AI refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0105] It's understandable that CV is the science of making machines "see." Specifically, it refers to machine vision, which uses cameras and computers to replace the human eye in identifying and measuring objects, and then further processes images to make them more suitable for human observation or transmission to instruments for detection. As a scientific discipline, CV studies related theories and technologies, attempting to build AI systems that can extract information from images or multidimensional data. CV technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content, behavior recognition, three-dimensional object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0106] Understandably, ML is a multidisciplinary field, encompassing probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. ML is at the core of AI and the fundamental path to computer intelligence. Its applications span all areas of AI. ML and deep learning typically encompass techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0107] In order to improve the effect of video image restoration, this application proposes a video restoration method, which is applied to Figure 1The video repair system shown in the figure includes a server and a terminal, and the client is deployed on the terminal, wherein the client can be run on the terminal in the form of a browser, or in the form of an independent application (APP), etc. The specific presentation form of the client is not limited here. The server involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a car terminal, an aircraft, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this. The number of servers and terminals is also not limited. The solution provided in this application can be completed independently by the terminal, independently by the server, or by the terminal and the server in cooperation, and this application does not make specific limitations on this.

[0108] The following will be combined Figure 1 The architecture shown introduces two video restoration workflows.

[0109] For example, in one scenario, a user uploads a video to a server via a terminal. The server can then directly invoke the video restoration function. Specifically, it first determines the selected video restoration algorithm (i.e., optical flow or modeling), then uses the corresponding video restoration algorithm to restore the video. Finally, the restored video is stored in a database. When the terminal requests the server to play a video, the server retrieves the corresponding video from the database and returns it to the terminal.

[0110] For example, in another scenario, a user uploads a video to a server via a terminal, and the server stores the uploaded video in a database. When a video needs to be repaired, the corresponding video can be selected from the database and the video repair function can be called. Specifically, the selected video repair algorithm (i.e., optical flow or model method) is first determined. Based on this, the corresponding video repair algorithm is used to repair the video. Finally, the repaired video is stored in the database.

[0111] There are some differences in the effects of video restoration using optical flow and model methods. The following diagrams will explain these differences.

[0112] 1. Fill the mask area based on the optical flow method;

[0113] For ease of introduction, see Figure 2 , Figure 2 This is an effect diagram of video frame filling based on the optical flow method in the embodiment of this application. Figure 2 As shown in Figure (A), the mask object is detected in the video frame. After filling with the optical flow method, the following can be obtained: Figure 2 The video frame shown in Figure (B) is shown in the figure. As can be seen, the filling effect based on the optical flow method will be greatly affected when there are objects blocking the image or complex background motion. The erroneous pixels caused by the optical flow estimation error will gradually expand as it propagates, resulting in incorrect filling content.

[0114] 2. Fill the mask area based on the model method;

[0115] For ease of introduction, see Figure 3 , Figure 3 This is an effect diagram of implementing video frame filling based on the model method in the embodiment of this application, such as Figure 3 As shown in Figure (A), the mask object is detected in the video frame. After filling with the model method, the following can be obtained: Figure 3 The video frame shown in Figure (B) is shown in the middle. As can be seen, the padded portion is relatively blurry, and due to limitations in video memory and other factors, it is difficult to process high-resolution input. However, the overall effect is relatively stable, and it is not prone to obvious errors in areas with strong contrast.

[0116] Combined with the above introduction, limited by the restoration quality of the optical flow method and the model method, this application proposes a video restoration method that can pre-determine which video restoration method to choose for image restoration, so as to achieve a more robust filling effect. The following will introduce the video restoration method in this application, please refer to Figure 4 In the embodiment of the present application, the video repair method can be executed by a computer device, which can be a terminal or a server. The embodiment of the present application includes:

[0117] 110. Obtain a video sample sequence for the video to be repaired, wherein the video sample sequence includes K video frame pairs, each video frame pair includes two adjacent video frames, and K is an integer greater than or equal to 1;

[0118] In one or more embodiments, a video to be repaired is obtained, and then K video frame pairs are extracted from the video to be repaired as a video sample sequence, each video frame pair includes two adjacent video frames, and each video frame has a corresponding video frame number. For example, if normalization is not performed, the video sample sequence can be represented as x s , x s ={(x1,x2),((x 11 ,x 12 ),…}. The video sample sequence includes K video frame pairs, that is, the first video frame pair is represented as (x1, x2), the second video frame pair is represented as (x11 ,x 12 ), and so on. For example, if normalization has been performed, the video sample sequence is represented as x sr , x sr ={(x r1 ,x r2 ),((x r11 ,x r12 ),…}. The video sample sequence includes K video frame pairs, that is, the first video frame pair is represented by (x r1 ,x r2 ), the second video frame pair is represented as (x r11 ,x r12 ), and so on.

[0119] 120. Obtain a target mask sample sequence according to the video sample sequence, wherein the target mask sample sequence includes K target mask frames, each target mask frame includes a target mask region obtained by expanding an original mask region, and each target mask frame has a corresponding relationship with each video frame pair;

[0120] In one or more embodiments, after obtaining a video sample sequence, at least one corresponding original mask frame can be obtained for each video frame pair. For each original mask frame, the corresponding original mask region is marked, and then the original mask region is expanded by a certain number of pixels to obtain a target mask region. A target mask frame is then obtained based on the target mask region. Thus, a target mask sample sequence comprising K target mask frames is obtained.

[0121] 130. Obtain an optical flow data sequence according to the video sample sequence, wherein the optical flow data sequence includes K optical flow data, and each optical flow data has a corresponding relationship with each video frame pair;

[0122] In one or more embodiments, corresponding optical flow data is generated for each of K video frame pairs in a video sample sequence, thereby obtaining an optical flow data sequence including K optical flow data. The optical flow data can be represented as a two-channel optical flow matrix. One optical flow matrix is ​​used to record the horizontal offset of all pixel points of the video frame pair, and the other optical flow matrix is ​​used to record the vertical offset of all pixel points of the video frame pair.

[0123] 140. Based on each optical flow data in the optical flow data sequence, cluster the pixels included in the target mask area in each target mask frame to obtain an optical flow clustering result for each target mask frame;

[0124] In one or more embodiments, the optical flow data sequence is aligned with the target mask sample sequence, i.e., the optical flow data in the optical flow data sequence corresponds to the target mask frame in the target mask sample sequence. Based on this, for each target mask frame, the corresponding optical flow data is used to assign a corresponding two-dimensional optical flow value to each pixel within the target mask area. Then, based on the two-dimensional optical flow values ​​of each pixel, a clustering algorithm is used to cluster these pixels, thereby obtaining an optical flow clustering result for each target mask frame.

[0125] It is understandable that the present application may use density-based spatial clustering of applications with noise (DBSCAN), mean shift clustering, or other clustering methods to cluster pixels, which are not limited here.

[0126] 150. Determine an optical flow quality score based on the optical flow clustering result of each target mask frame;

[0127] In one or more embodiments, the optical flow quality can be comprehensively judged by combining the optical flow clustering results of each target mask frame, thereby generating a corresponding optical flow quality score. For example, the optical flow quality score in this application is a first score or a second score. A first score indicates good optical flow quality, for example, the first score can be "1." A second score indicates poor optical flow quality, for example, the second score can be "0."

[0128] It is understandable that in actual applications, other values ​​may be set for the first score and the second score respectively. This is only an illustration and should not be understood as a limitation to the present application.

[0129] 160. A video restoration method that matches the optical flow quality score is used to restore the video to be restored.

[0130] In one or more embodiments, a corresponding video restoration method may be selected based on the optical flow quality score. Specifically, if the optical flow quality score is a first value, the video to be restored is restored using an optical flow method. If the optical flow quality score is a second value, a neural network is invoked to restore the video to be restored.

[0131] Specifically, the process of using optical flow to repair the video to be repaired mainly includes: using adjacent frames to estimate optical flow, then filling the original mask area in each frame with optical flow, and applying optical flow to propagate the pixel gradients of the unmasked area to the original mask area. Poisson reconstruction is then performed on the pixel gradients to generate red, green, and blue (RGB) pixels. Finally, image repair is performed on areas that cannot be filled by optical flow. The process of calling a neural network to repair the video to be repaired is to receive frame sequence information as input and output the repaired video frame after neural network processing.

[0132] It is understandable that neural networks often use an encoder-decoder structure. The neural network used in this application can be a fine-grained visual categorization (FGVC) network, a spatial-temporal transformer network (STTN), or a decoupled spatial-temporal attention network (DSTT), etc., without limitation here.

[0133] In an embodiment of the present application, a method for video restoration is provided. Through the above-mentioned method, the optical flow clustering result of the masked area is used to predict the optical flow quality. When the optical flow quality is good, the optical flow method can be used as a video restoration method to obtain filling content with higher clarity and credibility. When the optical flow quality is poor, a generative model can be used as a video restoration method to obtain a filling effect with higher stability. It can be seen that the present application uses optical flow quality as the basis for selecting a video restoration method, so as to achieve the purpose of learning from the strengths and weaknesses of different video restoration methods, thereby facilitating the acquisition of video images with better restoration effects.

[0134] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, obtaining a video sample sequence for a video to be repaired may specifically include:

[0135] Obtaining a video sequence from the video to be repaired, wherein the video sequence includes T original video frames, each original video frame displays a target object, and T is an integer greater than 1;

[0136] Extracting K to-be-processed video frame pairs from a video sequence, wherein each to-be-processed video frame pair includes two adjacent original video frames;

[0137] The sizes of the original video frames in each to-be-processed video frame pair are normalized to obtain K video frame pairs, and the K video frame pairs are used as video sample sequences.

[0138] In one or more embodiments, a method for generating a video sample sequence is introduced. As can be seen from the above embodiments, the video sample sequence is derived from the video to be repaired, and the video to be repaired is represented by x={x t}(t=1,2,…,T), it can be seen that the video to be repaired includes T original video frames, that is, x t represents the tth original video frame.

[0139] Specifically, adjacent original video frames can be extracted at a certain interval. For example, a group of adjacent original video frames are extracted every 10 frames, and the extracted sequence is represented as x s , x s ={(x1,x2),((x 11 ,x 12 ),…}, where the sequence includes K pairs of video frames to be processed, that is, the first pair of video frames to be processed is represented as (x1, x2), the second pair of video frames to be processed is represented as (x 11 ,x 12 ), and so on. Based on this, each original video frame in the video frame pair to be processed is size normalized to obtain the corresponding video frame. Adjacent video frames constitute a video frame pair, and K video frame pairs constitute a video sample sequence, where the video sample sequence can be expressed as x sr , x sr ={(x r1 ,x r2 ),((x r11 ,x r12 ),…}.

[0140] It should be noted that the normalized video frame has a fixed size, for example, 512×288.

[0141] Secondly, embodiments of the present application provide a method for generating a video sample sequence. This method, on the one hand, extracts a number of to-be-processed video frame pairs from a video sequence for subsequent processing, thereby reducing the amount of data to be processed and conserving data processing resources. On the other hand, size normalization is performed on the original video frames, which not only aligns the statistics of each video frame but also reduces the size of the video frames, thereby improving processing efficiency.

[0142] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, obtaining a target mask sample sequence according to a video sample sequence may specifically include:

[0143] For each video frame pair in the video sample sequence, an original mask frame is obtained according to a previous video frame or a next video frame in the video frame pair, wherein the original mask frame includes an original mask area obtained after masking the target object;

[0144] For each video frame pair in the video sample sequence, the original mask region in the original mask frame is expanded to obtain a target mask frame;

[0145] The K target mask frames are used as the target mask sample sequence.

[0146] In one or more embodiments, a method for generating a target mask frame based on a single video frame is described. As can be seen from the aforementioned embodiments, if a target object is displayed in the video to be restored, a masking process is performed on the target object to obtain a corresponding original mask region. The original mask region is then expanded by a certain number of pixels to obtain a target mask region.

[0147] It should be noted that the target object can be a logo, a subtitle, an object, etc. It is understood that methods for identifying the target object include but are not limited to manual labeling and model recognition, for example, using a fully convolution network (FCN) to identify the target object.

[0148] For example, one processing method is that the video to be repaired is x={x t}(t=1,2,…,T). Mask processing can be performed on each original video frame in the video to be repaired, thereby obtaining m={m t}(t=1,2,…,T). Assume that a group of adjacent original video frames are extracted every 10 frames, then the extracted video sample sequence is represented as x s ={(x1,x2),((x 11 ,x 12 ),…}, thus, the corresponding mask frame sequence is expressed as m s ={(m1,m2),((m 11 ,m 12 ),…}. If the forward optical flow is used, then based on m s Extract the previous video frame of each video frame pair and get m sF ={m1,m 11 ,…}, then m sF Normalized, the original mask frame sequence is represented as m sr ={m r1 ,m r11 ,…}. If backward optical flow is used, then based on m s Extract the next video frame of each video frame pair and get msB ={m2,m 12 ,…}, then m sB Normalized, the original mask frame sequence is represented as m sr ={m r2 ,m r12 ,…}, wherein the original mask frame sequence includes K original mask frames.

[0149] For example, one processing method is that the video to be repaired is x={x t}(t=1,2,…,T). Assume that a set of adjacent original video frames is extracted every 10 frames, then the extracted sequence is represented as x s ={(x1,x2),((x 11 ,x 12 ),…}. For x s Each original video frame in is normalized and the video sample sequence is expressed as x sr ={(x r1 ,x r2 ),((x r11 ,x r12 ),…}. If the forward optical flow is used, then based on x sr Extract the previous video frame of each video frame pair and get x srF ={(x r1 ,x r11 ,…}, then for x srF Perform mask processing, and the original mask frame sequence obtained is represented as m sr ={m r1 ,m r11 ,…}. If backward optical flow is used, then based on x sr Extract the next video frame of each video frame pair and get x srB ={(x r2 ,x r12 ,…}, then for x srB Perform mask processing, and the original mask frame sequence obtained is represented as m sr ={m r2 ,m r12 ,…}, wherein the original mask frame sequence includes K original mask frames.

[0150] Specifically, for easier understanding, see Figure 5 , Figure 5 A schematic diagram of generating a target mask frame in an embodiment of the present application is shown in FIG. Figure 5Take the original mask frame shown in Figure (a) as an example, where 15 pixels marked as "1" constitute the original mask area. Assume that the original mask area is expanded by 2 pixels to obtain the target mask area (i.e., the gray area composed of "1"). Based on this, we can get Figure 5 The target mask frame shown in (b).

[0151] Similarly, each original mask frame is processed until the target mask sample sequence is obtained. The target mask sample sequence can be expressed as {m dst}(t=1,2,…,K). Among them, m dst represents the t-th target mask frame.

[0152] Secondly, in an embodiment of the present application, a method for generating a target mask frame based on a single video frame is provided. With this method, considering that the original mask regions of the two frames before and after a video frame pair are not much different, the region expansion process can be performed on only one of the original mask frames, thereby reducing the complexity of the operation.

[0153] Optionally, in the above Figure 4 Based on the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, for each video frame pair in the video sample sequence, the original mask area in the original mask frame is expanded to obtain a target mask frame, which may specifically include:

[0154] For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to the first number of pixels to obtain a first mask region;

[0155] For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to a second number of pixels to obtain a second mask region, wherein the second number of pixels is greater than the first number of pixels;

[0156] For each video frame pair in the video sample sequence, an XOR operation is performed on the first mask area and the second mask area to obtain a target mask frame.

[0157] In one or more embodiments, a method for expanding an original mask region is described. As can be seen from the aforementioned embodiments, for each original mask frame in the original mask frame sequence, the original mask region can be expanded to obtain a target mask region. This results in a target mask frame containing the target mask region.

[0158] Specifically, for easier understanding, see Figure 6 , Figure 6 Another schematic diagram of generating a target mask frame in an embodiment of the present application is shown in FIG. Figure 6Take the original mask frame shown in (a) as an example, where 15 pixels marked as "1" constitute the original mask area. Assume that the original mask area is expanded according to the first number of pixels (for example, 2 pixels), and the first mask area (i.e., the gray area composed of "1" is obtained). Based on this, we can get Figure 6 Assume that the original mask region is expanded according to the second number of pixels (e.g., 4 pixels) to obtain the second mask region (i.e., the gray region marked as "1"). Based on this, we can obtain Figure 6 Based on this, the first mask area and the second mask area are subjected to an XOR operation to obtain the following: Figure 6 The target mask frame shown in Figure (d) includes a target mask area (ie, a gray area marked as "1").

[0159] Similarly, each original mask frame is processed until the target mask sample sequence is obtained. The target mask sample sequence can be expressed as {m dst =m da ^m db}(t=1,2,…,K0. Among them, m dst represents the tth target mask frame, m da represents the mask frame including the first mask area, a represents the number of first pixels, m d9 represents the mask frame including the second mask area, b represents the second number of pixels, and “^” represents the exclusive OR operator.

[0160] In practical applications, the number of the first pixels may be 7, and the number of the second pixels may be 9. Thus, the target mask sample sequence can be expressed as {m dst =m d7 ^m d9}(t=1,2,…,K0. It should be noted that the first number of pixels and the second number of pixels can also be adjusted according to the situation and are not limited here.

[0161] Finally, an embodiment of the present application provides a method for expanding the original mask area. With this method, the optical flow within the original mask area is derived from the surrounding optical flow. If the surrounding optical flow is chaotic, the optical flow within the original mask area cannot be well filled. Considering that pixels close to the original mask area may contain some noise, the target mask area obtained by deviating from the original mask area has less noise, which helps improve the determination of optical flow quality.

[0162] Optionally, in the above Figure 4On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, obtaining a target mask sample sequence according to a video sample sequence may specifically include:

[0163] For each video frame pair in the video sample sequence, obtaining a first original mask frame based on a previous video frame in the video frame pair, and obtaining a second original mask frame based on a subsequent video frame in the video frame pair, wherein the first original mask frame and the second original mask frame respectively include original mask regions obtained after masking the target object;

[0164] For each video frame pair in the video sample sequence, performing union processing on the first original mask frame and the second original mask frame to obtain an original mask frame;

[0165] For each video frame pair in the video sample sequence, the original mask region in the original mask frame is expanded to obtain a target mask frame;

[0166] The K target mask frames are used as the target mask sample sequence.

[0167] In one or more embodiments, a method for generating a target mask frame based on multiple video frames is described. As can be seen from the aforementioned embodiments, if a target object is displayed in the video to be restored, a masking process is performed on the target object to obtain a corresponding original mask region. The original mask region is then expanded by a certain number of pixels to obtain a target mask region.

[0168] It should be noted that the target object may be a logo, a subtitle, an object, etc. It is understood that the method of identifying the target object includes but is not limited to manual labeling and model recognition, for example, using FCN to identify the target object.

[0169] For example, one processing method is that the video to be repaired is x={x t}(t=1,2,…,T). Mask processing can be performed on each original video frame in the video to be repaired, thereby obtaining m={m t}(t=1,2,…,T). Assume that a group of adjacent original video frames are extracted every 10 frames, then the extracted video sample sequence is represented as x s ={(x1,x2),((x 11 ,x 12 ),…}, thus, the corresponding mask frame sequence is expressed as m s ={(m1,m2),((m 11 ,m 12 ),…}. Then for m s Normalized, the original mask frame sequence is represented as m sr ={(m r1 ,mr2 ),((m r11 ,m r12 ),…}. The original mask frame sequence includes K first original mask frames (ie, {m r1 ,m r11 ,…}) and K second original mask frames (ie, m sr ={m r2 ,m r12 ,…}).

[0170] For example, one processing method is that the video to be repaired is x={x t}(t=1,2,…,T). Assume that a set of adjacent original video frames is extracted every 10 frames, then the extracted sequence is represented as x s ={(x1,x2),((x 11 ,x 12 ),…}. For x s Each original video frame in is normalized and the video sample sequence is expressed as x sr ={(x r1 ,x r2 ),((x r11 ,x r12 ),…}. Then for x sr Perform mask processing, and the original mask frame sequence obtained is represented as m sr ={(m r1 ,m r2 ),((m r11 ,m r12 ),…}. The original mask frame sequence includes K first original mask frames (ie, {m r1 ,m r11 ,…}) and K second original mask frames (ie, m sr ={m r2 ,m r12 ,…}).

[0171] Specifically, for easier understanding, see Figure 7 , Figure 7 This is another schematic diagram of generating a target mask frame in an embodiment of the present application. Figure 7 The figure (a) shows the first original mask frame, wherein 13 pixels marked as "1" constitute the original mask area of ​​the first original mask frame. Figure 7 The second original mask frame is shown in (b), wherein the 13 pixels marked as "1" constitute the original mask area of ​​the second original mask frame. After the first original mask frame and the second original mask frame are processed together, the following is obtained: Figure 7The original mask frame shown in Figure (c) has 15 pixels marked as "1" forming the original mask area of ​​the original mask frame. Assume that the original mask area is expanded by 2 pixels to obtain the target mask area (i.e., the gray area marked as "1"). Based on this, we can get Figure 7 The target mask frame shown in (d).

[0172] Similarly, each original mask frame is processed until the target mask sample sequence is obtained. The target mask sample sequence can be expressed as {m dst}(t=1,2,…,K). Among them, m dst represents the t-th target mask frame.

[0173] Secondly, an embodiment of the present application provides a method for generating a target mask frame based on multiple video frames. This method, taking into account the potential differences in the original mask regions of the two preceding and succeeding video frames in a video frame pair, can first take the union of the original mask regions of the two preceding and succeeding frames, thereby obtaining a more accurate original mask region. This improves the processing performance of the video frames.

[0174] Optionally, in the above Figure 4 Based on the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, for each video frame pair in the video sample sequence, the original mask area in the original mask frame is expanded to obtain a target mask frame, which may specifically include:

[0175] For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to the first number of pixels to obtain a first mask region;

[0176] For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to a second number of pixels to obtain a second mask region, wherein the second number of pixels is greater than the first number of pixels;

[0177] For each video frame pair in the video sample sequence, an XOR operation is performed on the first mask area and the second mask area to obtain a target mask frame.

[0178] In one or more embodiments, a method for expanding an original mask region is described. As can be seen from the aforementioned embodiments, for each original mask frame in the original mask frame sequence, the original mask region can be expanded to obtain a target mask region. This results in a target mask frame containing the target mask region.

[0179] Specifically, for easier understanding, see Figure 8 , Figure 8 This is another schematic diagram of generating a target mask frame in an embodiment of the present application. Figure 8 The figure (a) shows the first original mask frame, wherein 13 pixels marked as "1" constitute the original mask area of ​​the first original mask frame. Figure 8 The second original mask frame is shown in (b), wherein the 13 pixels marked as "1" constitute the original mask area of ​​the second original mask frame. After the first original mask frame and the second original mask frame are processed together, the following is obtained: Figure 8 In the original mask frame shown in Figure (c), 15 pixels marked as "1" constitute the original mask area of ​​the original mask frame. Assume that the original mask area is expanded according to the first number of pixels (for example, 2 pixels), and the first mask area (i.e., the gray area composed of "1" pixels) is obtained. Based on this, we can get Figure 8 The mask frame shown in Figure (d) is shown in Figure 3. Assume that the original mask area is expanded according to the second number of pixels (for example, 4 pixels) to obtain the second mask area (i.e., the gray area marked as "1"). Based on this, we can get Figure 8 Based on this, the first mask area and the second mask area are subjected to an XOR operation to obtain the following: Figure 8 The target mask frame shown in Figure (f) includes a target mask area (ie, a gray area marked as "1").

[0180] Similarly, each original mask frame is processed until the target mask sample sequence is obtained. The target mask sample sequence can be expressed as {m dst =m da ^m db}(t=1,2,…,K). Among them, m dst represents the tth target mask frame, m da represents the mask frame including the first mask area, a represents the number of first pixels, m d9 represents the mask frame including the second mask area, b represents the second number of pixels, and “^” represents the exclusive OR operator.

[0181] In practical applications, the number of the first pixels may be 7, and the number of the second pixels may be 9. Thus, the target mask sample sequence can be expressed as {m dst =m d7 ^m d9}(t=1, 2, ..., K) It should be noted that the first number of pixels and the second number of pixels can be adjusted according to circumstances and are not limited here.

[0182] Finally, an embodiment of the present application provides a method for expanding the original mask area. With this method, the optical flow within the original mask area is derived from the surrounding optical flow. If the surrounding optical flow is chaotic, the optical flow within the original mask area cannot be well filled. Considering that pixels close to the original mask area may contain some noise, the target mask area obtained by deviating from the original mask area has less noise, which helps improve the determination of optical flow quality.

[0183] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, obtaining an optical flow data sequence according to a video sample sequence may specifically include:

[0184] For each video frame pair in the video sample sequence, determine optical flow data based on the horizontal offset and vertical offset of each pixel point in the subsequent video frame relative to each pixel point in the previous video frame;

[0185] Take K optical flow data as the optical flow data sequence;

[0186] or,

[0187] Obtaining an optical flow data sequence based on a video sample sequence may specifically include:

[0188] For each video frame pair in the video sample sequence, determine optical flow data based on the horizontal offset and vertical offset of each pixel point in the previous video frame relative to each pixel point in the next video frame;

[0189] Take K optical flow data as the optical flow data sequence.

[0190] In one or more embodiments, two methods for determining optical flow data based on video frame pairs are introduced. As can be seen from the above embodiments, the video sample sequence includes K video frame pairs, each video frame pair includes two video frames. If the video frames have been size-normalized, the video sample sequence can be represented as x sr ={(x r1 ,x r2 ),((x r11 ,x r12 ),…}. Assuming the size of the video frame is 512×288, the optical flow data Fl t It represents an optical flow matrix with 2 channels and a size of 512×288. The optical flow data sequence is represented by {Fl t Thus, the two-dimensional optical flow value (w′, h′) corresponding to each pixel can be determined by combining the optical flow data, where w′ represents the horizontal offset of the pixel and h′ represents the vertical offset of the pixel.

[0191] The following will take a pixel as an example and illustrate how to determine optical flow data.

[0192] 1. Determine optical flow data based on forward optical flow;

[0193] Specifically, if forward optical flow is used, the optical flow data needs to be determined based on the horizontal and vertical offsets of each pixel in the next video frame relative to each pixel in the previous video frame. Figure 9 , Figure 9 This is a schematic diagram of determining the two-dimensional optical flow value based on the forward optical flow in an embodiment of the present application. In the previous video frame, the pixel coordinates are (3, 4). In the next video frame, the pixel coordinates are (4, 5). The horizontal offset of this pixel from the next video frame to the previous video frame is 1 (i.e., 4-3), and the vertical offset is 1 (i.e., 5-4). It can be seen that the two-dimensional optical flow value of this pixel is (1, 1).

[0194] 2. Determine optical flow data based on backward optical flow;

[0195] Specifically, if backward optical flow is used, the optical flow data needs to be determined based on the horizontal and vertical offsets of each pixel in the previous video frame relative to each pixel in the next video frame. Figure 10 , Figure 10 This is a schematic diagram of determining the two-dimensional optical flow value based on backward optical flow in an embodiment of the present application. In the previous video frame, the pixel coordinates are (1, 3). In the next video frame, the pixel coordinates are (4, 5). The horizontal offset of this pixel from the previous video frame to the next video frame is -4 (i.e., 1-4), and the vertical offset is -2 (i.e., 3-5). Therefore, the two-dimensional optical flow value of this pixel is (-4, -2).

[0196] Secondly, in the embodiment of the present application, two methods for determining optical flow data based on video frame pairs are provided. Through the above methods, optical flow data can be generated based on forward optical flow or backward optical flow, thereby improving the flexibility of the solution.

[0197] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, based on each optical flow data in the optical flow data sequence, clustering processing is performed on the pixels included in the target mask area in each target mask frame to obtain the optical flow clustering result of each target mask frame, which may specifically include:

[0198] For each target mask frame, determine the two-dimensional optical flow values ​​of X pixels in the target mask area based on the corresponding optical flow data in the optical flow data sequence, where X is an integer greater than 1;

[0199] For each target mask frame, cluster the X pixels according to their two-dimensional optical flow values ​​to obtain the optical flow clustering result.

[0200] In one or more embodiments, a method for clustering pixels within a target mask region is described. As can be seen from the aforementioned embodiments, the target mask sample sequence includes K target mask frames, and optical flow clustering is required for the pixels within the target mask region in each target mask frame. It is understood that in actual situations, the number of pixels included in the target mask region may be large. Therefore, the pixels within the target mask region may be randomly sampled in advance to obtain X pixels. Here, X is an integer greater than 1. For example, X can be set to 15,000.

[0201] Specifically, the target mask sample sequence is {m dst}(t=1,2,…,K), the optical flow data sequence is {Fl t}(t=1,…,K). Based on this, we can calculate Among them, "*" represents element-wise multiplication, thereby retaining the target mask frame The two-dimensional optical flow value corresponding to the pixel points marked as "1" in the image is set, and the rest are set to 0. Therefore, the DBSCAN algorithm can be used to map the target mask frame. The X pixels within the image are clustered based on the two-dimensional optical flow value of each pixel to obtain the optical flow clustering result of the target mask frame.

[0202] It should be noted that the optical flow clustering results of each target mask frame include the category labels corresponding to each pixel after clustering. Among them, the pixels with category label "0" are noise pixels and need to be removed. After removal, the total number of categories corresponding to the target mask frame is obtained. Taking the tth target mask frame as an example, the total number of categories corresponding to it can be expressed as C t , that is, with C t Clusters. Clusters may include N ct The number of pixels.

[0203] Secondly, in the embodiments of the present application, a method for clustering pixels within a target mask area is provided. This method allows the use of the DBSCAN algorithm to cluster pixels. This method enables adaptive clustering without requiring a pre-defined number of categories. Furthermore, the DBSCAN algorithm is able to better identify outliers and can detect clusters of any shape.

[0204] Optionally, in the above Figure 4On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, determining the optical flow quality score according to the optical flow clustering result of each target mask frame may specifically include:

[0205] According to the optical flow clustering results of each target mask frame, the total number of categories of each target mask frame is determined;

[0206] Count the number of frames whose total number of categories is less than or equal to the category number threshold;

[0207] According to the ratio between the number of frames and the K value, the single category ratio is determined;

[0208] If the single ratio of the category is greater than the ratio threshold, the optical flow quality score is determined to be the first score;

[0209] If the single category ratio is less than or equal to the ratio threshold, the optical flow quality score is determined to be a second score.

[0210] In one or more embodiments, a method for determining an optical flow quality score based on a clean rate (CR) is provided. As can be seen from the aforementioned embodiments, the optical flow clustering results for each target mask frame include the class labels corresponding to each pixel after clustering. Therefore, pixels with a class label of "0" can be eliminated to obtain the number of classes corresponding to the target mask frame.

[0211] Specifically, for the optical flow clustering results (t=1,2,…,K,c=1,2,…,C t , i=1,2,…,N ct ), the single category proportion can be calculated as follows:

[0212]

[0213] Among them, CR represents the single ratio of the category. t represents the frame number of the target mask frame, K represents the total number of target mask frames. c represents the category label, C t Indicates the total number of categories in the t-th target mask frame. i represents the pixel number, N ct Indicates the number of pixels of the t-th target mask frame corresponding to the c-th category label. Indicates an indicative function that returns 1 if the input is 1, otherwise it returns 0.

[0214] Based on this, the proportion of frames in which the total number of categories is less than or equal to the category number threshold (for example, 1) in the K target mask frames can be counted, that is, the category single ratio can be obtained.

[0215] Combined with the single ratio of each category, the criteria for determining the quality of optical flow can be defined as:

[0216]

[0217] Where Q represents the optical flow quality score. CR represents the single category ratio. threshold Represents a ratio threshold. For example, the ratio threshold can be set to 0.8, or other reasonable values, which are not limited here.

[0218] Secondly, the present embodiment provides a method for determining optical flow quality scores based on the category-uniqueness ratio. This method takes into account that a larger category-uniqueness ratio indicates a smaller total number of categories and more stable video optical flow. Therefore, using the category-uniqueness ratio can filter out videos with disturbed optical flow, thus serving as a basis for determining optical flow quality, thereby improving the feasibility and operability of the solution.

[0219] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, determining the optical flow quality score according to the optical flow clustering result of each target mask frame may specifically include:

[0220] For the optical flow clustering results of each target mask frame, a moving average of each cluster is determined according to the two-dimensional optical flow values ​​of each pixel in each cluster, wherein the optical flow clustering results are used to determine one or more clusters;

[0221] Based on the optical flow clustering results of each target mask frame, the moving average of the target mask frame is determined according to the moving average of each cluster cluster;

[0222] Accumulate the moving average of each target mask frame to obtain the total moving distance;

[0223] If the total moving distance is greater than or equal to the distance threshold, determining the optical flow quality score to be the first score;

[0224] If the total moving distance is less than the distance threshold, the optical flow quality score is determined to be a second score.

[0225] In one or more embodiments, a method for determining an optical flow quality score based on the total distance traveled is described. As can be seen from the aforementioned embodiments, the optical flow clustering results for each target mask frame include the class labels corresponding to each clustered pixel. Therefore, pixels with a class label of "0" are eliminated, thereby determining the number of classes corresponding to the target mask frame.

[0226] Specifically, for the optical flow clustering results (t=1,2,…,K,c=1,2,…,C t , i=1,2,…,N ct ), the total moving distance accumulated by K target mask frames can be calculated as follows:

[0227]

[0228] Where D represents the total distance moved. t represents the moving average of the t-th target mask frame. t represents the frame number of the target mask frame, and K represents the total number of target mask frames.

[0229] The moving average of the target mask frame can be calculated as follows:

[0230]

[0231] Among them, D t represents the moving average of the t-th target mask frame. represents the moving average of the cth cluster in the tth target mask frame. c represents the category label, C t Represents the total number of categories in the t-th target mask frame.

[0232] The moving average of the clusters can be calculated as follows:

[0233]

[0234] in, represents the moving average of the cth cluster in the tth target mask frame. Represents the two-dimensional optical flow value of the i-th pixel in the c-th cluster in the t-th target mask frame. i represents the pixel number, N ct represents the number of pixels in the t-th target mask frame corresponding to the c-th category label. ||.|| represents the Euclidean distance.

[0235] Based on this, the total moving distance of K target mask frames can be counted. Combined with the total moving distance, the judgment standard of optical flow quality can be defined as:

[0236]

[0237] Where Q represents the optical flow quality score. D represents the total distance moved. threshold Represents the distance threshold. For example, the distance threshold can be set to 4, or other reasonable values, which are not limited here.

[0238] Secondly, embodiments of this application provide a method for determining optical flow quality scores based on total movement distance. This method considers that a larger total movement distance indicates more pronounced frame motion, which facilitates optical flow estimation. Therefore, using total movement distance to filter out relatively static video can serve as a basis for determining optical flow quality, thereby improving the feasibility and operability of the solution.

[0239] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, determining the optical flow quality score according to the optical flow clustering result of each target mask frame may specifically include:

[0240] According to the optical flow clustering results of each target mask frame, the total number of categories of each target mask frame is determined;

[0241] Count the number of frames whose total number of categories is less than or equal to the category number threshold;

[0242] According to the ratio between the number of frames and the K value, the single category ratio is determined;

[0243] For the optical flow clustering results of each target mask frame, a moving average of each cluster is determined according to the two-dimensional optical flow values ​​of each pixel in each cluster, wherein the optical flow clustering results are used to determine one or more clusters;

[0244] Based on the optical flow clustering results of each target mask frame, the moving average of the target mask frame is determined according to the moving average of each cluster cluster;

[0245] Accumulate the moving average of each target mask frame to obtain the total moving distance;

[0246] If the single ratio of the category is greater than the ratio threshold, and the total moving distance is greater than or equal to the distance threshold, then the optical flow quality score is determined to be the first score;

[0247] If the single category ratio is less than or equal to the ratio threshold, and the total movement distance is less than the distance threshold, the optical flow quality score is determined to be the second score.

[0248] In one or more embodiments, a method for jointly determining an optical flow quality score based on a single category ratio and a total movement distance is introduced. As can be seen from the aforementioned embodiments, on the one hand, the proportion of frames in which the total number of categories is less than or equal to a category number threshold (e.g., 1) in the K target mask frames can be counted, that is, the single category ratio can be obtained. On the other hand, the total movement distance of the K target mask frames can be counted. It is understood that the method for determining the single category ratio and the total movement distance can be referred to the aforementioned embodiments and will not be repeated here.

[0249] Specifically, combining the single ratio of categories and the total distance moved, the criterion for determining the optical flow quality can be defined as:

[0250]

[0251] Where Q represents the optical flow quality score. D represents the total distance moved. thresholdIndicates the distance threshold. For example, the distance threshold can be set to 4, or other reasonable values, which are not limited here. CR indicates the single ratio of the category. threshold Represents a ratio threshold. For example, the ratio threshold can be set to 0.8, or other reasonable values, which are not limited here.

[0252] Secondly, embodiments of this application provide a method for determining the optical flow quality score based on both the single category ratio and the total distance traveled. This method, while utilizing the single category ratio, can filter out videos with disturbed optical flow, while utilizing the total distance traveled can filter out relatively static videos. This combination of factors can more comprehensively and accurately reflect optical flow quality, thereby improving the reliability of the optical flow quality score.

[0253] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, another optional embodiment provided by the embodiments of the present application adopts a video restoration method that matches the optical flow quality score to restore the video to be restored, which may specifically include:

[0254] If the optical flow quality score is the first score, the optical flow method is used to repair the video to be repaired.

[0255] If the optical flow quality score is the second score, the neural network is called to perform repair processing on the video to be repaired.

[0256] In one or more embodiments, a method for video restoration based on an optical flow quality score is described. As can be seen from the aforementioned embodiments, the optical flow quality score can be a first score or a second score. The following description uses the first score of "1" and the second score of "0" as an example.

[0257] Specifically, you can select the video repair method in the following ways:

[0258]

[0259] Where F1(x,m) represents the video restoration process using the optical flow method. F2(x,m) represents the video restoration process using the neural network. Q represents the optical flow quality score.

[0260] It should be noted that the goal of this application is to solve the video sequence y={y t}(t=0,1,2,…,T). The video sequence differs from the video to be repaired only in the original mask area, making the video sequence natural and consistent in time and space. Since naturalness and consistency are difficult to define formally, when training the neural network, it is hoped that the completed video sequence is consistent with the real video sequence y gt Close. Among them, y gtRepresents the true value of the video sequence without the original mask area. Based on this, by constructing the algorithm F, the solution of the video sequence y can be defined as y = F(x,m).

[0261] Secondly, an embodiment of the present application provides a method for video inpainting based on optical flow quality scores. Using this approach, before video inpainting, if the optical flow quality is judged to be good, a direct optical flow method can be used to obtain clear and reliable inpainting content. If the optical flow is unreliable, a model method is used to inpaint the content, thereby avoiding incorrect inpainting caused by inaccurate optical flow estimation and achieving a more stable inpainting effect overall.

[0262] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, another optional embodiment provided by the embodiment of the present application may further include:

[0263] Displaying the video to be repaired and a list of repair objects, wherein the list of repair objects includes at least one repairable object;

[0264] In response to a selection instruction for a target object, executing a step of acquiring a video sample sequence for a video to be repaired, wherein the target object is at least one repairable object;

[0265] After the video to be repaired is repaired using a video repair method that matches the optical flow quality score, the following steps may also be performed:

[0266] In response to a play instruction for the repaired video, the repaired video is played.

[0267] In one or more embodiments, a method for intelligently repairing videos is described. As can be seen from the aforementioned embodiments, this application can be applied to various video repair tasks, such as removing logos, removing subtitles, and removing objects. If a user wants to use a video from a certain platform, but the video has a logo that affects the viewing experience, a video repair application can be used to remove the logo. Similarly, users can erase subtitles from some videos, or remove certain moving objects from videos. The following will introduce each of these tasks separately with reference to the diagrams.

[0268] For example, see Figure 11 , Figure 11This is a schematic diagram of the effect of removing a logo based on a video repair application in an embodiment of the present application. As shown in the figure, the video to be repaired and the list of repair objects are displayed on the interface provided by the video repair application, wherein the list of repair objects displays at least one repairable object (for example, a logo, subtitles, a ship, clouds, etc.). Assume that the user selects the control corresponding to the "logo", thereby triggering a selection instruction for the target object (i.e., the logo). Then, in response to the selection instruction, the video repair function is called. Based on this, a suitable video repair method is used to repair the video, thereby obtaining a repaired video. It can be seen that there is no logo in the repaired video.

[0269] When the user triggers a play instruction for the repaired video, the repaired video can be played.

[0270] For example, see Figure 12 , Figure 12 This is a schematic diagram of the effect of removing subtitles based on a video repair application in an embodiment of the present application. As shown in the figure, the video to be repaired and the list of repair objects are displayed on the interface provided by the video repair application, wherein the list of repair objects displays at least one repairable object (for example, a logo, subtitles, a ship, clouds, etc.). Assume that the user selects the control corresponding to "subtitles", thereby triggering a selection instruction for the target object (i.e., subtitles). Then, in response to the selection instruction, the video repair function is called. Based on this, a suitable video repair method is used to repair the video, thereby obtaining a repaired video. It can be seen that there are no subtitles in the repaired video.

[0271] When the user triggers a play instruction for the repaired video, the repaired video can be played.

[0272] For example, see Figure 13 , Figure 13 This is a schematic diagram of the effect of removing objects based on a video repair application in an embodiment of the present application. As shown in the figure, the video to be repaired and the list of repair objects are displayed on the interface provided by the video repair application, wherein the list of repair objects displays at least one repairable object (for example, a logo, subtitles, a ship, clouds, etc.). Suppose the user selects the control corresponding to "ship", thereby triggering a selection instruction for the target object (i.e., ship). Then, in response to this selection instruction, the video repair function is called. Based on this, the video is repaired using an appropriate video repair method to obtain a repaired video. It can be seen that the object "ship" does not exist in the repaired video.

[0273] When the user triggers a play instruction for the repaired video, the repaired video can be played.

[0274] It should be noted that Figure 11 , Figure 12 and Figure 13The interface elements, interface layout, and interface text shown are for illustrative purposes only and should not be construed as limitations on this application.

[0275] Secondly, the embodiments of this application provide a method for intelligent video restoration. Through this method, users can use the video restoration application to select and restore one or more objects in a video, achieving the purpose of intelligent restoration. This not only improves the practicality of the solution, but also improves the efficiency of video restoration.

[0276] It can be seen that this application can accurately and efficiently judge the quality of optical flow in video clips, that is, before calling the video restoration method, the optical flow method or the model method is selected, so that the restoration effect is better than the effect of using either method alone. The following examples will introduce the effects of video frame restoration based on the optical flow method and the model method. Figure 14 , Figure 14 This is a schematic diagram comparing the effects of video frame restoration based on the optical flow method and the model method in an embodiment of the present application. As shown in the figure, in one example, Figure 14 The figure (a) shows the effect of filling based on the optical flow method. Figure 14 Figure (b) shows the effect of filling based on the model method. The original mask area is located in the lower left corner of the video frame (i.e., the area circled by the rectangular box). In this case, the lens movement is smooth and the optical flow estimation is good. Therefore, this application chooses to use the optical flow method for filling. In another example, Figure 14 The figure (c) shows the effect of filling based on the optical flow method. Figure 14 Figure (d) shows the effect of filling based on the model method. The original mask area is located in the lower left corner of the video frame (i.e., the area circled by the rectangular box). In this case, since the optical flow is affected by the person's watch, this application chooses to use the model method for filling.

[0277] The video repair device in this application is described in detail below. Figure 15 , Figure 15 This is a schematic diagram of an embodiment of a video restoration device in an embodiment of the present application. The video restoration device 20 includes:

[0278] An acquisition module 210 is configured to acquire a video sample sequence for a video to be repaired, wherein the video sample sequence includes K video frame pairs, each video frame pair includes two adjacent video frames, and K is an integer greater than or equal to 1;

[0279] The acquisition module 210 is further configured to acquire a target mask sample sequence based on the video sample sequence, wherein the target mask sample sequence includes K target mask frames, each target mask frame includes a target mask region obtained by expanding an original mask region, and each target mask frame corresponds to each video frame pair;

[0280] The acquisition module 210 is further configured to acquire an optical flow data sequence according to the video sample sequence, wherein the optical flow data sequence includes K optical flow data, and each optical flow data has a corresponding relationship with each video frame pair;

[0281] The processing module 220 is configured to perform clustering processing on the pixel points included in the target mask area in each target mask frame based on each optical flow data in the optical flow data sequence to obtain an optical flow clustering result for each target mask frame;

[0282] a determination module 230 for determining an optical flow quality score based on the optical flow clustering result of each target mask frame;

[0283] The restoration module 240 is configured to perform restoration processing on the video to be restored by adopting a video restoration method that matches the optical flow quality score.

[0284] In an embodiment of the present application, a video restoration device is provided. Using the above-mentioned device, the optical flow clustering results of the masked area are used to predict the optical flow quality. When the optical flow quality is good, the optical flow method can be used as a video restoration method to obtain filling content with high clarity and credibility. When the optical flow quality is poor, a generative model can be used as a video restoration method to obtain a filling effect with high stability. It can be seen that the present application uses optical flow quality as the basis for selecting a video restoration method, so as to achieve the purpose of learning from the strengths and weaknesses of different video restoration methods, thereby facilitating the acquisition of video images with better restoration effects.

[0285] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0286] The acquisition module 210 is specifically configured to acquire a video sequence from the video to be repaired, wherein the video sequence includes T original video frames, each original video frame displays a target object, and T is an integer greater than 1;

[0287] Extracting K to-be-processed video frame pairs from a video sequence, wherein each to-be-processed video frame pair includes two adjacent original video frames;

[0288] The sizes of the original video frames in each to-be-processed video frame pair are normalized to obtain K video frame pairs, and the K video frame pairs are used as video sample sequences.

[0289] In an embodiment of the present application, a video restoration device is provided. Using this device, several pairs of unprocessed video frames are extracted from a video sequence for subsequent processing, thereby reducing the amount of data to be processed and conserving data processing resources. Furthermore, size normalization is performed on the original video frames, which not only aligns the statistics of each video frame but also reduces the size of the video frames, thereby improving processing efficiency.

[0290] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0291] The acquisition module 210 is specifically configured to acquire, for each video frame pair in the video sample sequence, an original mask frame based on the previous video frame or the next video frame in the video frame pair, wherein the original mask frame includes an original mask area obtained after masking the target object;

[0292] For each video frame pair in the video sample sequence, the original mask region in the original mask frame is expanded to obtain a target mask frame;

[0293] The K target mask frames are used as the target mask sample sequence.

[0294] In an embodiment of the present application, a video restoration device is provided. Using the device, considering that the original mask regions of the two frames before and after a video frame pair are not much different, the region expansion process can be performed on only one of the original mask frames, thereby reducing the complexity of the operation.

[0295] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0296] The acquisition module 210 is specifically configured to expand the original mask region in the original mask frame according to the first number of pixels for each video frame pair in the video sample sequence to obtain a first mask region;

[0297] For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to a second number of pixels to obtain a second mask region, wherein the second number of pixels is greater than the first number of pixels;

[0298] For each video frame pair in the video sample sequence, an XOR operation is performed on the first mask area and the second mask area to obtain a target mask frame.

[0299] In an embodiment of the present application, a video restoration device is provided. Using this device, the optical flow within the original masked area is derived from the surrounding optical flow. If the surrounding optical flow is chaotic, the optical flow within the original masked area cannot be properly filled. Considering that pixels adjacent to the original masked area may contain some noise, the target masked area, which deviates from the original masked area, has less noise, thereby improving the determination of optical flow quality.

[0300] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0301] The acquisition module 210 is specifically configured to acquire, for each video frame pair in the video sample sequence, a first original mask frame based on a first video frame in the video frame pair, and acquire a second original mask frame based on a second video frame in the video frame pair, wherein the first original mask frame and the second original mask frame respectively include original mask regions obtained after masking the target object;

[0302] For each video frame pair in the video sample sequence, performing union processing on the first original mask frame and the second original mask frame to obtain an original mask frame;

[0303] For each video frame pair in the video sample sequence, the original mask region in the original mask frame is expanded to obtain a target mask frame;

[0304] The K target mask frames are used as the target mask sample sequence.

[0305] In an embodiment of the present application, a video restoration device is provided. Using this device, considering that the original mask regions of the two preceding and succeeding video frames in a video frame pair may differ, a more accurate original mask region can be obtained by first taking the union of the original mask regions of the two preceding and succeeding frames. This improves the processing performance of the video frames.

[0306] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0307] The acquisition module 210 is specifically configured to expand the original mask region in the original mask frame according to the first number of pixels for each video frame pair in the video sample sequence to obtain a first mask region;

[0308] For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to a second number of pixels to obtain a second mask region, wherein the second number of pixels is greater than the first number of pixels;

[0309] For each video frame pair in the video sample sequence, an XOR operation is performed on the first mask area and the second mask area to obtain a target mask frame.

[0310] In an embodiment of the present application, a video restoration device is provided. Using this device, the optical flow within the original masked area is derived from the surrounding optical flow. If the surrounding optical flow is chaotic, the optical flow within the original masked area cannot be properly filled. Considering that pixels adjacent to the original masked area may contain some noise, the target masked area, which deviates from the original masked area, has less noise, thereby improving the determination of optical flow quality.

[0311] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0312] The acquisition module 210 is specifically configured to determine, for each video frame pair in the video sample sequence, optical flow data based on the horizontal offset and vertical offset of each pixel point in the subsequent video frame relative to each pixel point in the previous video frame;

[0313] Take K optical flow data as the optical flow data sequence;

[0314] or,

[0315] The acquisition module 210 is specifically configured to determine, for each video frame pair in the video sample sequence, optical flow data based on the horizontal offset and vertical offset of each pixel point in the previous video frame relative to each pixel point in the next video frame;

[0316] Take K optical flow data as the optical flow data sequence.

[0317] In an embodiment of the present application, a video restoration device is provided. The device supports generating optical flow data based on forward optical flow or backward optical flow, thereby improving the flexibility of the solution.

[0318] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0319] The processing module 220 is specifically configured to determine, for each target mask frame, the two-dimensional optical flow values ​​of X pixels in the target mask area based on the corresponding optical flow data in the optical flow data sequence, where X is an integer greater than 1;

[0320] For each target mask frame, cluster the X pixels according to their two-dimensional optical flow values ​​to obtain the optical flow clustering result.

[0321] In an embodiment of the present application, a video restoration device is provided. Using this device, the DBSCAN algorithm can be used to cluster pixels. This algorithm enables adaptive clustering without requiring a predefined number of clusters. Furthermore, the DBSCAN algorithm can effectively identify outliers and discover clusters of any shape.

[0322] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0323] A determination module 230 is specifically configured to determine the total number of categories of each target mask frame based on the optical flow clustering result of each target mask frame;

[0324] Count the number of frames whose total number of categories is less than or equal to the category number threshold;

[0325] According to the ratio between the number of frames and the K value, the single category ratio is determined;

[0326] If the single ratio of the category is greater than the ratio threshold, the optical flow quality score is determined to be the first score;

[0327] If the single category ratio is less than or equal to the ratio threshold, the optical flow quality score is determined to be a second score.

[0328] In one embodiment of the present application, a video restoration device is provided. Using this device, a larger proportion of single categories indicates a smaller total number of categories and more stable video optical flow. Therefore, using the single category proportion can filter out videos with disturbed optical flow, which can serve as a basis for determining optical flow quality, thereby improving the feasibility and operability of the solution.

[0329] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0330] A determination module 230 is specifically configured to determine a moving average of each cluster based on the two-dimensional optical flow values ​​of each pixel in each cluster for the optical flow clustering result of each target mask frame, wherein the optical flow clustering result is used to determine one or more clusters;

[0331] Based on the optical flow clustering results of each target mask frame, the moving average of the target mask frame is determined according to the moving average of each cluster cluster;

[0332] Accumulate the moving average of each target mask frame to obtain the total moving distance;

[0333] If the total moving distance is greater than or equal to the distance threshold, determining the optical flow quality score to be the first score;

[0334] If the total moving distance is less than the distance threshold, the optical flow quality score is determined to be a second score.

[0335] In one embodiment of the present application, a video restoration device is provided. Using this device, considering that a larger total movement distance indicates more pronounced frame motion, which facilitates optical flow estimation, the total movement distance is used to filter out relatively still videos, which can then be used as a basis for determining optical flow quality, thereby improving the feasibility and operability of the solution.

[0336] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0337] A determination module 230 is specifically configured to determine the total number of categories of each target mask frame based on the optical flow clustering result of each target mask frame;

[0338] Count the number of frames whose total number of categories is less than or equal to the category number threshold;

[0339] According to the ratio between the number of frames and the K value, the single category ratio is determined;

[0340] For the optical flow clustering results of each target mask frame, a moving average of each cluster is determined according to the two-dimensional optical flow values ​​of each pixel in each cluster, wherein the optical flow clustering results are used to determine one or more clusters;

[0341] Based on the optical flow clustering results of each target mask frame, the moving average of the target mask frame is determined according to the moving average of each cluster cluster;

[0342] Accumulate the moving average of each target mask frame to obtain the total moving distance;

[0343] If the single ratio of the category is greater than the ratio threshold, and the total moving distance is greater than or equal to the distance threshold, then the optical flow quality score is determined to be the first score;

[0344] If the single category ratio is less than or equal to the ratio threshold, and the total movement distance is less than the distance threshold, the optical flow quality score is determined to be the second score.

[0345] In one embodiment of the present application, a video restoration device is provided. This device, while utilizing a single category ratio, can filter out videos with disturbed optical flow. Furthermore, it can utilize the total distance moved to filter out relatively static videos. This combination of factors can more comprehensively and accurately reflect optical flow quality, thereby improving the reliability of the optical flow quality score.

[0346] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application,

[0347] The restoration module 240 is specifically configured to restore the video to be restored by using the optical flow method if the optical flow quality score is the first score.

[0348] If the optical flow quality score is the second score, the neural network is called to perform repair processing on the video to be repaired.

[0349] In one embodiment of the present application, a video restoration device is provided. Using this device, before performing video restoration, if the optical flow quality is determined to be good, a direct optical flow method can be used to obtain clear and reliable infill content. If the optical flow is unreliable, a model method is used to infill the content, thereby avoiding incorrect infill caused by inaccurate optical flow estimation and achieving a more stable infill effect overall.

[0350] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the video restoration device 20 provided in the embodiment of the present application, the video restoration device 20 further includes a display module 250;

[0351] A display module 250 is configured to display the video to be repaired and a list of repair objects, wherein the list of repair objects includes at least one repairable object;

[0352] The acquisition module 210 is further configured to, in response to a selection instruction for a target object, execute a step of acquiring a video sample sequence for a video to be repaired, wherein the target object is at least one repairable object;

[0353] The display module 250 is further configured to use a video restoration method that matches the optical flow quality score to restore the video to be restored, and then respond to a play instruction for the restored video to play the restored video.

[0354] In an embodiment of the present application, a video restoration device is provided. Using this device, a user can use a video restoration application to select and restore one or more objects in a video, achieving intelligent restoration. This not only improves the practicality of the solution but also enhances the efficiency of video restoration.

[0355] The embodiment of the present application also provides a terminal, such as Figure 16 For the sake of convenience, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. In the embodiment of the present application, the terminal is taken as an example for explanation:

[0356] Figure 16The block diagram shows a partial structure of a mobile phone related to the terminal provided in the embodiment of the present application. Figure 16 The mobile phone includes components such as a radio frequency (RF) circuit 310, a memory 320, an input unit 330, a display unit 340, a sensor 350, an audio circuit 360, a wireless fidelity (WiFi) module 370, a processor 380, and a power supply 390. Those skilled in the art will understand that Figure 16 The mobile phone structure shown in the figure does not constitute a limitation to the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0357] The following combination Figure 16 A detailed introduction to the various components of a mobile phone:

[0358] The RF circuit 310 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is sent to the processor 380 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 310 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 310 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0359] The memory 320 can be used to store software programs and modules. The processor 380 executes the various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 320 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0360] The input unit 330 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 330 may include a touch panel 331 and other input devices 332. The touch panel 331, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel 331) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 331 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction and detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 380, and can receive commands sent by the processor 380 and execute them. In addition, the touch panel 331 can be implemented using various types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch panel 331, the input unit 330 may further include other input devices 332. Specifically, the other input devices 332 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a mouse, and an operating stick.

[0361] The display unit 340 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 340 may include a display panel 341. Optionally, the display panel 341 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 331 may cover the display panel 341. When the touch panel 331 detects a touch operation on or near it, it is transmitted to the processor 380 to determine the type of touch event. Subsequently, the processor 380 provides corresponding visual output on the display panel 341 according to the type of touch event. Although in Figure 16 In the embodiment, the touch panel 331 and the display panel 341 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 331 and the display panel 341 can be integrated to realize the input and output functions of the mobile phone.

[0362] The mobile phone may also include at least one sensor 350, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 341 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 341 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0363] Audio circuit 360, speaker 361, and microphone 362 provide an audio interface between the user and the phone. Audio circuit 360 converts received audio data into electrical signals and transmits them to speaker 361, where they are converted into sound signals for output. Microphone 362, on the other hand, converts collected sound signals into electrical signals, which are received by audio circuit 360 and converted into audio data. The audio data is then processed by processor 380 and transmitted to, for example, another phone via RF circuit 310, or stored in memory 320 for further processing.

[0364] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse the web and access streaming media through the WiFi module 370. It provides users with wireless broadband Internet access. Figure 16A WiFi module 370 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0365] Processor 380 is the control center of the phone, connecting all parts of the phone using various interfaces and circuits. It executes software programs and / or modules stored in memory 320 and accesses data stored in memory 320 to perform various phone functions and process data. Optionally, processor 380 may include one or more processing units. Alternatively, processor 380 may integrate an application processor and a modem processor, with the application processor primarily handling the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 380.

[0366] The mobile phone also includes a power supply 390 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 380 through a power management system, thereby managing charging, discharging, and power consumption through the power management system.

[0367] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0368] The steps performed by the terminal in the above embodiment can be based on the Figure 16 The terminal structure shown.

[0369] Figure 17 : This is a schematic diagram of a server structure provided by an embodiment of the present application. The server 400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 422 (for example, one or more processors) and memories 432, and one or more storage media 430 (for example, one or more massive storage devices) for storing application programs 442 or data 444. Among them, the memories 432 and the storage media 430 may be temporary storage or persistent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 422 may be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the server 400.

[0370] The server 400 may also include one or more power supplies 426, one or more wired or wireless network interfaces 450, one or more input and output interfaces 458, and / or one or more operating systems 441, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM etc.

[0371] The steps performed by the server in the above embodiment can be based on the Figure 17 The server structure shown.

[0372] A computer device is also provided in an embodiment of the present application, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the methods described in the above embodiments are implemented.

[0373] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods described in the above embodiments are implemented.

[0374] A computer program product is also provided in an embodiment of the present application, including a computer program. When the computer program is executed by a processor, the steps of the methods described in the above embodiments are implemented.

[0375] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0376] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0377] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0378] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0379] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0380] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a server or terminal, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0381] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video restoration method, characterized in that: include: Obtain a video sample sequence for a video to be repaired, wherein the video sample sequence includes K video frame pairs, each video frame pair includes two adjacent video frames, and K is an integer greater than or equal to 1; Acquire a target mask sample sequence according to the video sample sequence, wherein the target mask sample sequence includes K target mask frames, each target mask frame includes a target mask area obtained by expanding an original mask area, and each target mask frame has a corresponding relationship with each video frame pair; Acquire an optical flow data sequence according to the video sample sequence, wherein the optical flow data sequence includes K optical flow data, and each optical flow data has a corresponding relationship with each video frame pair; performing clustering processing on the pixel points included in the target mask area in each target mask frame based on each optical flow data in the optical flow data sequence to obtain an optical flow clustering result of each target mask frame; Determining an optical flow quality score according to the optical flow clustering result of each target mask frame; The video to be repaired is repaired using a video repair method that matches the optical flow quality score.

2. The method according to claim 1, characterized in that The step of obtaining a video sample sequence for the video to be repaired includes: Acquire a video sequence from the video to be repaired, wherein the video sequence includes T original video frames, each original video frame displays a target object, and T is an integer greater than 1; Extracting K to-be-processed video frame pairs from the video sequence, wherein each to-be-processed video frame pair includes two adjacent original video frames; Normalizing the size of each original video frame in each to-be-processed video frame pair to obtain the K video frame pairs, and using the K video frame pairs as the video sample sequence.

3. The method according to claim 1, characterized in that The acquiring a target mask sample sequence according to the video sample sequence includes: For each video frame pair in the video sample sequence, obtaining an original mask frame according to a previous video frame or a next video frame in the video frame pair, wherein the original mask frame includes an original mask area obtained after masking the target object; For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame to obtain a target mask frame; The K target mask frames are used as the target mask sample sequence.

4. The method according to claim 3, characterized in that The step of expanding the original mask region in the original mask frame for each video frame pair in the video sample sequence to obtain a target mask frame includes: For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to a first number of pixels to obtain a first mask region; For each video frame pair in the video sample sequence, dilating an original mask region in the original mask frame according to a second number of pixels to obtain a second mask region, wherein the second number of pixels is greater than the first number of pixels; For each video frame pair in the video sample sequence, an exclusive OR operation is performed on the first mask area and the second mask area to obtain the target mask frame.

5. The method according to claim 1, wherein The acquiring a target mask sample sequence according to the video sample sequence includes: For each video frame pair in the video sample sequence, obtaining a first original mask frame based on a previous video frame in the video frame pair, and obtaining a second original mask frame based on a subsequent video frame in the video frame pair, wherein the first original mask frame and the second original mask frame respectively include original mask regions obtained after masking the target object; For each video frame pair in the video sample sequence, performing union processing on the first original mask frame and the second original mask frame to obtain an original mask frame; For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame to obtain a target mask frame; The K target mask frames are used as the target mask sample sequence.

6. The method according to claim 5, characterized in that The step of expanding the original mask region in the original mask frame for each video frame pair in the video sample sequence to obtain a target mask frame includes: For each video frame pair in the video sample sequence, expanding the original mask region in the original mask frame according to a first number of pixels to obtain a first mask region; For each video frame pair in the video sample sequence, dilating an original mask region in the original mask frame according to a second number of pixels to obtain a second mask region, wherein the second number of pixels is greater than the first number of pixels; For each video frame pair in the video sample sequence, an exclusive OR operation is performed on the first mask area and the second mask area to obtain the target mask frame.

7. The method according to claim 1, characterized in that The obtaining of an optical flow data sequence according to the video sample sequence includes: For each video frame pair in the video sample sequence, determining optical flow data according to a horizontal offset and a vertical offset of each pixel point in a subsequent video frame relative to each pixel point in a previous video frame; Taking the K optical flow data as the optical flow data sequence; or, The obtaining of an optical flow data sequence according to the video sample sequence includes: For each video frame pair in the video sample sequence, determining optical flow data according to a horizontal offset and a vertical offset of each pixel point in a previous video frame relative to each pixel point in a subsequent video frame; The K optical flow data are used as the optical flow data sequence.

8. The method according to claim 1, characterized in that The clustering process is performed on the pixel points included in the target mask area in each target mask frame based on each optical flow data in the optical flow data sequence to obtain the optical flow clustering result of each target mask frame, including: For each target mask frame, determining the two-dimensional optical flow values ​​of X pixels in the target mask area according to the corresponding optical flow data in the optical flow data sequence, where X is an integer greater than 1; For each target mask frame, clustering processing is performed on the X pixels according to the two-dimensional optical flow values ​​of the X pixels to obtain the optical flow clustering result.

9. The method according to any one of claims 1 to 8, characterized in that Determining an optical flow quality score according to the optical flow clustering result of each target mask frame includes: Determining the total number of categories of each target mask frame according to the optical flow clustering result of each target mask frame; Counting the number of frames whose total number of categories is less than or equal to the category number threshold; Determining a single category ratio according to a ratio between the number of frames and the K value; If the single category ratio is greater than the ratio threshold, determining the optical flow quality score to be a first score; If the single category ratio is less than or equal to the ratio threshold, the optical flow quality score is determined to be a second score.

10. The method according to any one of claims 1 to 8, characterized in that Determining an optical flow quality score according to the optical flow clustering result of each target mask frame includes: For the optical flow clustering result of each target mask frame, determining a moving average of each cluster according to the two-dimensional optical flow value of each pixel in each cluster, wherein the optical flow clustering result is used to determine one or more clusters; For the optical flow clustering result of each target mask frame, determining a moving average value of the target mask frame according to the moving average value of each cluster cluster; Accumulating the moving average of each target mask frame to obtain a total moving distance; If the total moving distance is greater than or equal to the distance threshold, determining the optical flow quality score to be a first score; If the total moving distance is less than the distance threshold, the optical flow quality score is determined to be a second score.

11. The method according to any one of claims 1 to 8, characterized in that Determining an optical flow quality score according to the optical flow clustering result of each target mask frame includes: Determining the total number of categories of each target mask frame according to the optical flow clustering result of each target mask frame; Counting the number of frames whose total number of categories is less than or equal to the category number threshold; Determining a single category ratio according to a ratio between the number of frames and the K value; For the optical flow clustering result of each target mask frame, determining a moving average of each cluster according to the two-dimensional optical flow value of each pixel in each cluster, wherein the optical flow clustering result is used to determine one or more clusters; For the optical flow clustering result of each target mask frame, determining a moving average value of the target mask frame according to the moving average value of each cluster cluster; Accumulating the moving average of each target mask frame to obtain a total moving distance; If the single category ratio is greater than a ratio threshold, and the total movement distance is greater than or equal to a distance threshold, determining the optical flow quality score to be a first score; If the single category ratio is less than or equal to the ratio threshold, and the total movement distance is less than the distance threshold, the optical flow quality score is determined to be a second score.

12. The method according to claim 1, characterized in that The repairing process of the video to be repaired using a video repair method that matches the optical flow quality score includes: If the optical flow quality score is the first score, repairing the video to be repaired using the optical flow method; If the optical flow quality score is the second score, a neural network is called to perform restoration processing on the video to be restored.

13. The method according to claim 1, wherein The method further comprises: Displaying the video to be repaired and a list of repair objects, wherein the list of repair objects includes at least one repairable object; In response to a selection instruction for a target object, executing the step of obtaining a video sample sequence for the video to be repaired, wherein the target object belongs to the at least one repairable object; After repairing the video to be repaired using a video repair method that matches the optical flow quality score, the method further includes: In response to a play instruction for the repaired video, the repaired video is played.

14. A video restoration device, characterized in that: include: An acquisition module, configured to acquire a video sample sequence for a video to be repaired, wherein the video sample sequence includes K video frame pairs, each video frame pair includes two adjacent video frames, and K is an integer greater than or equal to 1; The acquisition module is further configured to acquire a target mask sample sequence based on the video sample sequence, wherein the target mask sample sequence includes K target mask frames, each target mask frame includes a target mask area obtained by expanding an original mask area, and each target mask frame has a corresponding relationship with each video frame pair; The acquisition module is further configured to acquire an optical flow data sequence according to the video sample sequence, wherein the optical flow data sequence includes K optical flow data, and each optical flow data has a corresponding relationship with each video frame pair; a processing module, configured to perform clustering processing on the pixel points included in the target mask area in each target mask frame based on each optical flow data in the optical flow data sequence, to obtain an optical flow clustering result of each target mask frame; a determination module, configured to determine an optical flow quality score according to the optical flow clustering result of each target mask frame; The restoration module is configured to perform restoration processing on the video to be restored by adopting a video restoration method that matches the optical flow quality score.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images