Video processing method, device, equipment and storage medium

Abstract

The embodiments of the present disclosure provide a video processing method, apparatus, device and storage medium, which relate to the field of computer technology. The method includes: obtaining a first target frame sequence and a first target mask sequence of a video to be processed; performing frame extraction processing to obtain a first sampling frame sequence, a first remaining frame sequence, a first sampling mask sequence, and a first remaining mask sequence; performing optical flow propagation and image restoration processing to obtain a first restored sampling frame sequence and a first restored remaining frame sequence; merging the first restored sampling frame sequence and the first restored remaining frame sequence to obtain a restored video of the video to be processed. The present disclosure can process the sampling frame sequence and the first sampling mask sequence through frame extraction processing to obtain a first restored sampling frame sequence. Then, the first remaining frame sequence is processed through the first restored sampling frame sequence to obtain a restored video. The present disclosure can improve the efficiency of video processing.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, equipment and storage medium. Background Art

[0002] With the development of computer technology, the demand for video processing is also gradually increasing. For example, people may need to remove watermarks, subtitles or moving objects in a video. However, in the video, watermarks, subtitles or moving objects will block the background of the video. Therefore, after the removal operation is completed, unknown areas will be generated in the video due to the removal operation. Therefore, it is also necessary to fill and repair the unknown areas in the video, but the solutions for video repair in related technologies have the problem of low efficiency and are difficult to be deployed on a large scale commercially. Summary of the invention

[0003] The embodiments of the present disclosure provide a video processing method, apparatus, device and storage medium, which can be used to improve the efficiency of video processing.

[0004] The present disclosure provides a video processing method, which includes: obtaining a first target frame sequence and a first target mask sequence of a video to be processed; performing frame sampling processing on the first target frame sequence and the first target mask sequence respectively to obtain a first sampling frame sequence and a first remaining frame sequence thereof, and a first sampling mask sequence and a first remaining mask sequence thereof; performing optical flow propagation and image restoration processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first restored sampling frame sequence; performing optical flow propagation and image restoration processing on the first restored sampling frame sequence, the first remaining frame sequence, and the first remaining mask sequence to obtain a first restored remaining frame sequence; merging the first restored sampling frame sequence and the first restored remaining frame sequence to obtain a restored video of the video to be processed.

[0005] The disclosed embodiment provides a video processing device, which includes: an acquisition module, used to acquire a first target frame sequence and a first target mask sequence of a video to be processed; a frame extraction module, used to perform frame extraction processing on the first target frame sequence and the first target mask sequence respectively to obtain a first sampling frame sequence and a first remaining frame sequence thereof, and a first sampling mask sequence and a first remaining mask sequence thereof; the acquisition module is also used to perform optical flow propagation and image restoration processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first restored sampling frame sequence; the acquisition module is also used to perform optical flow propagation and image restoration processing on the first restored sampling frame sequence, the first remaining frame sequence and the first remaining mask sequence to obtain a first restored remaining frame sequence; the acquisition module is also used to perform merging processing on the first restored sampling frame sequence and the first restored remaining frame sequence to obtain a restored video of the video to be processed.

[0006] In some exemplary embodiments of the present disclosure, an acquisition module is used to perform gradient pixel conversion processing on the first residual frame sequence to obtain a first residual gradient frame sequence; obtain a first residual optical flow sequence based on the first residual frame sequence and the first sampling frame sequence; perform optical flow filling processing on the mask area in the first residual optical flow sequence to obtain a first filled residual optical flow sequence; perform optical flow propagation and image restoration processing on the first residual gradient frame sequence, the first filled residual optical flow sequence, the first repaired sampling frame sequence and the first residual mask sequence to obtain the first repaired residual frame sequence.

[0007] In some exemplary embodiments of the present disclosure, an acquisition module is used to determine a target remaining frame in the first remaining frame sequence; determine a target forward frame and a target backward frame of the target remaining frame from the first sampling frame sequence; obtain a forward optical flow map between the target forward frame and the target remaining frame, and a backward optical flow map between the target backward frame and the target remaining frame; and determine the first remaining optical flow sequence based on the forward optical flow map and the backward optical flow map.

[0008] In some exemplary embodiments of the present disclosure, an acquisition module is used to perform optical flow propagation processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence; perform Poisson reconstruction processing on the first updated sampling gradient frame sequence and the first updated sampling mask sequence to obtain a first updated sampling frame sequence; and perform frame-by-frame image restoration processing on the first updated sampling frame sequence to obtain the first restored sampling frame sequence.

[0009] In some exemplary embodiments of the present disclosure, an acquisition module is used to perform optical flow estimation processing on the first sampling frame sequence to obtain a first sampling optical flow sequence; perform gradient pixel conversion processing on the first sampling frame sequence to obtain a first sampling gradient frame sequence; perform optical flow filling processing on the mask area in the first sampling optical flow sequence to obtain a first filled sampling optical flow sequence; perform optical flow propagation processing on the first sampling gradient frame sequence and the first sampling mask sequence according to the first filled sampling optical flow sequence to obtain the first updated sampling gradient frame sequence and the first updated sampling mask sequence.

[0010] In some exemplary embodiments of the present disclosure, an acquisition module is used to acquire a first frame sequence of the video to be processed and its corresponding first mask sequence; if the proportion of the mask area in the first mask sequence is less than a proportion threshold, a target mask is determined according to the first mask sequence, and the mask area of ​​the target mask covers the mask area of ​​the first mask sequence; the first frame sequence and the first mask sequence are respectively cropped according to the target mask to obtain the first target frame sequence and the first target mask sequence.

[0011] In some exemplary embodiments of the present disclosure, an acquisition module is used to obtain the number of outward extended pixels of the mask area of ​​the target mask according to the length and width of the mask area of ​​the target mask; obtain a cropping boundary of the mask area of ​​the target mask according to the length and width of the mask area of ​​the target mask and the number of outward extended pixels; obtain a cropping area including the mask area of ​​the target mask according to the cropping boundary of the mask area of ​​the target mask and the length and width of the mask area of ​​the target mask; and perform cropping processing on the first frame sequence and the first mask sequence respectively according to the cropping area to obtain the first target frame sequence and the first target mask sequence.

[0012] In some exemplary embodiments of the present disclosure, an acquisition module is used to acquire a first frame sequence of the video to be processed and its corresponding first mask sequence; if the proportion of the mask area in the first mask sequence is greater than or equal to a proportion threshold, the first frame sequence and the first mask sequence are scaled respectively to obtain the first target frame sequence and the first target mask sequence.

[0013] In some exemplary embodiments of the present disclosure, an acquisition module is used to acquire a complete frame sequence and a corresponding complete mask sequence of the video to be processed; perform lens segmentation processing on the complete frame sequence and the complete mask sequence respectively to obtain a single-lens frame sequence and a corresponding single-lens mask sequence; if the single-lens frame sequence and the corresponding single-lens mask sequence exceed a duration threshold, segment the single-lens frame sequence and the corresponding single-lens mask sequence to obtain a segmented frame sequence and a corresponding segmented mask sequence, the segmented frame sequence includes a first frame sequence, and the segmented mask sequence includes a first mask sequence; obtain the first target frame sequence and the first target mask sequence according to the first frame sequence and its corresponding first mask sequence.

[0014] In some exemplary embodiments of the present disclosure, the segmented frame sequence also includes a second frame sequence, and the segmented mask sequence also includes a second mask sequence; an acquisition module is used to insert the tail frames of the first repaired sampling frame sequence and the first repaired remaining frame sequence into the second frame sequence and the second mask sequence, respectively, as the first frames of the second frame sequence and the second mask sequence; the second frame sequence and the second mask sequence are processed to obtain a second repaired sampling frame sequence and a second repaired remaining frame sequence; the first repaired sampling frame sequence and the first repaired remaining frame sequence, the second repaired sampling frame sequence and the second repaired remaining frame sequence are processed to obtain the repaired video.

[0015] An embodiment of the present disclosure provides a computer device, including a processor, a memory, and an input / output interface; the processor is connected to the memory and the input / output interface, respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device including the processor executes the video processing method in any embodiment of the present disclosure.

[0016] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the video processing method in any embodiment of the present disclosure.

[0017] The embodiments of the present disclosure provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the video processing method provided in various optional ways in any embodiment of the present disclosure.

[0018] The technical solution provided by the embodiment of the present disclosure can firstly process the first sampling frame sequence and the first sampling mask sequence through the frame extraction method to obtain the first repaired sampling frame sequence. Then, the first remaining frame sequence is processed through the first repaired sampling frame sequence and the first remaining mask sequence to obtain the repaired video of the video to be processed. Therefore, the present disclosure can process the remaining frame sequence through the first repaired sampling frame sequence through the frame extraction process, avoid multiple rounds of processing for each frame image, reduce the amount of calculation, and thus improve the efficiency of video processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present disclosure.

[0020] Figure 2 It is a flowchart of a video processing method provided by an embodiment of the present disclosure.

[0021] Figure 3 It is a schematic diagram of a clipping area provided in an embodiment of the present disclosure.

[0022] Figure 4 It is a schematic diagram of performing optical flow propagation processing on a frame of image provided by an embodiment of the present disclosure.

[0023] Figure 5 It is a framework diagram of an optical flow estimation module provided in an embodiment of the present disclosure.

[0024] Figure 6 It is a schematic diagram of the process of optical flow estimation and optical flow filling provided by an embodiment of the present disclosure.

[0025] Figure 7 It is a schematic diagram of a process for obtaining a first repaired remaining frame sequence provided by an embodiment of the present disclosure.

[0026] Figure 8 It is a schematic diagram of a video processing process provided by an embodiment of the present disclosure.

[0027] Fig. 9 It is a flowchart of another video processing method provided by an embodiment of the present disclosure.

[0028] Fig.10 It is a schematic diagram of a video processing effect for removing a watermark provided by an embodiment of the present disclosure.

[0029] Fig.11 It is a schematic diagram of a video processing effect of removing subtitles provided by an embodiment of the present disclosure.

[0030] Fig.12 It is a schematic diagram of a video processing effect for removing moving objects provided by an embodiment of the present disclosure.

[0031] Fig.13 is a schematic diagram of a video processing device provided by an embodiment of the present disclosure.

[0032] Fig.14 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0035] The present disclosure provides a video processing method. Figure 1 As shown, it shows a schematic diagram of an implementation environment provided by an embodiment of the present disclosure. The implementation environment may include: a terminal 11 and a server 12.

[0036] Among them, the terminal 11 can send the video to be processed to the server 12. The server 12 can receive the video to be processed, and can process the video to be processed by the method provided in the embodiment of the present disclosure to obtain a repaired video of the video to be processed, and can send the repaired video obtained after processing to the terminal 11. The terminal 11 can receive the repaired video. Alternatively, the terminal 11 can process the video to be processed by the method provided in the embodiment of the present disclosure to obtain a repaired video. In addition, the terminal 11 can store the repaired video.

[0037] Exemplarily, the terminal 11 and the server 12 can be connected through a network, which can be a wireless network or a wired network. The above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network). In some embodiments, the data exchanged through the network is represented by technology and / or format including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology.

[0038] Exemplarily, the terminal 11 can be various electronic devices, including but not limited to mobile phones, computers, laptops, desktop computers, wearable devices, intelligent voice interaction devices, vehicle-mounted terminals, smart home appliances, aircraft, augmented reality devices, virtual reality devices, etc.

[0039] Exemplarily, the server 12 may be a server that provides various services, such as a background management server that provides support for devices operated by the user using the terminal 11. The background management server may analyze and process the received data, and feed back the processing results to the terminal.

[0040] Optionally, the server 12 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0041] Those skilled in the art will know that Figure 1 The numbers of the terminals 11 and the servers 12 are all schematic, and any number of the terminals 11 and the servers 12 may be provided according to actual needs. The embodiments of the present disclosure are not limited to this.

[0042] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0043] First, a video processing method is provided in an embodiment of the present disclosure. The method can be executed by any computer device with computing and processing capabilities. The following takes the application of the method to a terminal as an example.

[0044] Figure 2 A flow chart of a video processing method in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the video processing method provided in the embodiment of the present disclosure may include the following S201 to S205.

[0045] S201, obtaining a first target frame sequence and a first target mask sequence of a video to be processed.

[0046] The disclosed embodiments do not limit the type and acquisition source of the video to be processed. For example, the video to be processed may be a user's photographic work or a video on a certain video platform.

[0047] In some embodiments, the first target frame sequence of the video to be processed may include a continuous multi-frame image in the video to be processed and / or a frame sequence obtained after processing (e.g., cropping and / or scaling) the continuous multi-frame image in the video to be processed. The first target frame sequence contains at least one frame of images that need to be processed (e.g., repair, or remove subtitles, watermarks, or certain specific objects or people, etc.), and the at least one frame of images that need to be processed can be determined according to the actual application scenario. For example, for a video repair scenario, the image that needs to be processed refers to an image with a missing area in the video to be processed, and the missing area is repaired by the method provided in the embodiment of the present disclosure. For another example, for a scenario of removing subtitles from a video, the image that needs to be processed refers to an image with subtitles in the video to be processed, and the subtitle area is removed by the method provided in the embodiment of the present disclosure, and the area after the subtitles are removed is filled and repaired.

[0048] In some embodiments, the first target mask sequence of the video to be processed may correspond to the first target frame sequence, and the first target mask sequence is used to indicate the area to be processed in the image to be processed in the first target frame sequence, so the first target mask sequence can be represented as a coordinate matrix set, and the coordinate matrix set includes multiple coordinate matrices, and each target mask can be represented by a corresponding coordinate matrix. For example, for a scene where subtitles are removed, the area to be processed refers to the area where the subtitles to be removed are located. For another example, for a scene where watermarks need to be removed, the area to be processed refers to the area where the watermarks to be removed are located. The above-mentioned area to be processed can also be referred to as a mask area. The disclosed embodiment does not limit the method for obtaining the first target mask sequence. Exemplarily, the first target mask sequence can be obtained based on an algorithm. For example, for a scene where watermarks and / or subtitles are removed, the watermarks can be detected based on an algorithm to obtain the first target mask sequence. Alternatively, the first target mask sequence can also be obtained based on a manual annotation method.

[0049] For example, the first target frame sequence can be expressed as x={x t}, (t = 0, 1, 2, ..., T), where T can be an integer greater than or equal to zero, x t is the t+1th frame image in the first target frame sequence x. Corresponding to the first target frame sequence, the first target mask sequence can be expressed as m={m t}, (t=0,1,2,…,T). Among them, m in the first target mask sequence m t is the t+1th mask, the m t It can be expressed as a binary matrix. For example, for the area to be processed in the first target frame sequence, m tThe value of can be 1. For the area in the first target frame sequence that does not need to be processed, m t The value of can be 0, but the present disclosure is not limited thereto. Fig.12 The scenario shown in Fig.12 If the person riding a bicycle is removed, the m of the area corresponding to the person riding a bicycle can be set t The value of m in other areas is 1. t The value of can be 0.

[0050] In some embodiments, any multi-frame image containing at least one frame to be processed in the video to be processed can be directly used as the first target frame sequence, and the mask sequence corresponding to the multi-frame image can be used as the first target mask sequence.

[0051] In other embodiments, in order to reduce the workload of processing the video to be processed and to further improve the video processing speed, the first frame sequence and the first mask sequence mentioned above may be processed first to obtain the first target frame sequence and the first target mask sequence of the video to be processed.

[0052] Exemplarily, processing the first frame sequence and the first mask sequence to obtain a first target frame sequence and a first target mask sequence of the video to be processed may include the following two methods.

[0053] Method 1, obtaining a first target frame sequence and a first target mask sequence of a video to be processed, may include: obtaining a first frame sequence of the video to be processed and a first mask sequence corresponding thereto; if a proportion of a mask area in the first mask sequence is less than a proportion threshold, determining a target mask according to the first mask sequence, and a mask area of ​​the target mask covers a mask area of ​​the first mask sequence; and cropping the first frame sequence and the first mask sequence respectively according to the target mask to obtain a first target frame sequence and a first target mask sequence.

[0054] Exemplarily, all masks in the first mask sequence can be obtained. When the proportion of any mask area in all masks to the corresponding frame image in the first frame sequence is less than the proportion threshold, since the area to be filled (i.e., the corresponding mask area) is only related to its surrounding information, if the coverage area of ​​the mask area in the frame image in the first frame sequence is small, there is no need to process the complete first frame sequence and the first mask sequence. Therefore, the first frame sequence and the first mask sequence can be cropped. The embodiment of the present disclosure does not limit the size of the proportion threshold, and the size of the proportion threshold can be limited based on experience or application scenarios.

[0055] In another exemplary embodiment, the target mask M may be determined first. When the target mask M covers all masks in the first mask sequence, and the area ratio of the mask region in the target mask M in the frame image in the first frame sequence is less than the ratio threshold, it is determined that the ratio of the mask region in the first mask sequence is less than the ratio threshold. That is, when it is determined that the mask region in the target mask is small, it can be determined that the mask regions of all frame images in the first mask sequence are relatively small. Therefore, the first frame sequence and the first mask sequence may be cropped to obtain the first target frame sequence and the first target mask sequence.

[0056] In some embodiments, the target mask may be a mask covering all masks in the first mask sequence. In particular, when each mask in the first mask sequence is the same, the target mask may be any mask in the first mask sequence.

[0057] For example, in the application scenario of removing watermarks and / or subtitles, since the areas corresponding to the watermarks and subtitles in each frame of the image are the same, when the first mask sequence is m={m t}, (t=0, 1, 2, ..., T), for different t1 and t2 (t1 and t2 are positive integers greater than or equal to zero and less than or equal to T), the corresponding m t1 With m t2 can be equal, then the target mask M can be equal to any m in the first mask sequence t Afterwards, the first frame sequence and the first mask sequence may be cropped based on the size of the mask region of the target mask to obtain a first target frame sequence and a first target mask sequence.

[0058] In an exemplary embodiment, the first frame sequence and the first mask sequence are respectively cropped according to the target mask to obtain the first target frame sequence and the first target mask sequence, which may include: obtaining the number of outwardly extended pixels of the mask area of ​​the target mask according to the length and width of the mask area of ​​the target mask; obtaining the cropping boundary of the mask area of ​​the target mask according to the length, width and the number of outwardly extended pixels of the mask area of ​​the target mask; obtaining a cropping area including the mask area of ​​the target mask according to the cropping boundary of the mask area of ​​the target mask and the length and width of the mask area of ​​the target mask; and according to the cropping area, cropping the first frame sequence and the first mask sequence respectively to obtain the first target frame sequence and the first target mask sequence.

[0059] In some embodiments, after determining the target mask M, the length and width of the mask area of ​​the target mask M can be determined according to the coverage of the mask area in the target mask M. Exemplarily, the coverage of the mask area in M ​​can be expressed as [x1, y1, x2, y2], where (x1, y1) can be the coordinate value of the upper left corner of the coverage of the mask area in M, and (x2, y2) can be the coordinate value of the lower right corner of the coverage of the mask area. Therefore, the length of the mask area of ​​the target mask can be calculated by the following formula (1) (the length is represented by h in formula (1), and h is a real number greater than 0), and the width of the mask area of ​​the target mask can be calculated by the following formula (2) (the width is represented by w in formula (2), and w is a real number greater than 0).

[0060] h=x2-x1 (1)

[0061] w=y2-y1 (2)

[0062] After obtaining the length h and width w of the target mask's mask area, the number of outward extended pixels minBound of the target mask's mask area can be calculated using the following formula (3): pix Exemplarily, the outward expansion pixel number is the value of the number of pixels that the mask area of ​​the target mask needs to expand outward, for example, the outward expansion pixel number may be 1152 or 2048.

[0063]

[0064] Among them, minBound pix is the number of outward expansion pixels of the mask area of ​​the target mask. a is the minimum outward expansion pixel number set based on the resolution of the video to be processed, b is the slope of the outward expansion pixel number for the resolution of the video to be processed, and c can be the length or width pixel number of the video to be processed with a larger data volume. Exemplarily, a, b and c are all real numbers greater than 0. The embodiment of the present disclosure does not limit the values ​​of a, b and c in formula (3). The values ​​of a, b and c can be set according to experience or application scenarios. For example, a can be 128, b can be 64, and c can be 1024, or a can be 196, b can be 48, and c can be 960. edge is used to indicate the maximum value of the length and width of the mask area of ​​the target mask. The value of edge can be calculated by the following formula (4).

[0065] edge=max(h,w) (4)

[0066] Afterwards, the outward expansion range of the mask area of ​​the target mask in the x-axis direction and the y-axis direction can be obtained by the outward expansion pixel number of the mask area of ​​the target mask. For example, the value of the outward expansion range can be estimated according to the amplitude of video motion. Two aspects can be considered in the estimation process. One is that the outward expansion area needs to cover a certain range to include the pixels and context information required in the optical flow propagation process. At the same time, while ensuring that the optical flow propagation is not affected, the outward expansion range can be as small as possible to reduce processing time. The outward expansion range of the mask area of ​​the target mask in the x-axis direction and the y-axis direction is used as the clipping boundary of the mask area of ​​the target mask. For example, the outward expansion range of the mask area of ​​the target mask in the x-axis direction can be calculated by the following formula (5), and the outward expansion range of the mask area of ​​the target mask in the y-axis direction can be calculated by the following formula (6).

[0067]

[0068]

[0069] Among them, minBound h The outer expansion range of the mask area of ​​the target mask in the x-axis direction. minBound w is the outer expansion range of the target mask area in the y-axis direction. r is the ratio of the side length of the target mask area to the corresponding side length of the cropping area. h with minBound w That is, the clipping boundaries of the mask area of ​​the target mask in the x-axis direction and the y-axis direction.

[0070] Exemplarily, the crop area may be calculated using the following formulas (7)-(10).

[0071] x crop1 =clip(x1-minBound w ,0,w) (7)

[0072] y crop1 =clip(y1-minBound h ,0,h) (8)

[0073] x crop2 =clip(x2+minBound w ,0,w) (9)

[0074] y crop2 =clip(y2+minBound h , 0, h) (10)

[0075] Among them, (xcrop1 ,y crop1 ) is the coordinate value of the upper left corner of the cropping area. (x crop2 ,y crop2 ) is the coordinate value of the lower right corner of the clipping area. The clip function is used to limit the value input to the corresponding interval. crop1 For example, the input value in the clip function is x1-minBound w , so (x1-minBound w ) should be between 0 and w, so when (x1-minBound w ) is between 0 and w, then (x1-minBound w ) as the value of x crop1 When (x1-minBound w ) is greater than w, then the value of w is taken as x crop1 When (x1-minBound w ) is less than 0, then let x crop1 =0.

[0076] For example, Figure 3 A schematic diagram of a cropping area is shown. Figure 3 As shown in FIG. , the target mask corresponds to the person riding a bicycle. Therefore, the mask area of ​​the target mask M can be as follows Figure 3 As shown by the dotted line in , (x1, y1) is the coordinate value of the upper left corner of the coverage range of the mask area, and (x2, y2) is the coordinate value of the lower right corner of the coverage range of the mask area. Therefore, the length h of the mask area of ​​the target mask M and the width w of the mask area of ​​the target mask M can be calculated. Then, the outer expansion range minBound of the mask area of ​​the target mask M in the x-axis direction can be calculated h , and the outer expansion range minBound in the y-axis direction w The resulting clipping boundary can be expressed as Figure 3 Then, the clipping area can be calculated by the above formulas (7)-(10). Figure 3 In the case shown in crop1 ,y crop1 ) and (x crop2 ,y crop2 ) are the coordinate values ​​of the upper left corner and the lower right corner of the area framed by the cropping boundary.

[0077] The purpose of calculating the outward expansion range of the mask area of ​​the target mask in the x-axis direction and the y-axis direction is to retain a sufficient number of pixels in the non-mask area after cropping. The pixels in the non-mask area can be used for optical flow filling and pixel propagation processing inside the mask area, and also affect the image restoration process.

[0078] Through the above calculation process, a cropping region can be obtained. Afterwards, the first frame sequence and the first mask sequence can be cropped according to the cropping region, and only the portion within the cropping region is retained for the first frame sequence and the first mask sequence to obtain a first target frame sequence and a first target mask sequence.

[0079] The above-mentioned cropping process of the first frame sequence and the first mask sequence can effectively reduce the number of processed pixels for the scene where the proportion of the mask area in the first mask sequence is less than the proportion threshold by performing a crop operation, thereby reducing the workload of processing the video to be processed without affecting the effect of video processing, thereby further improving the video processing speed.

[0080] In the scenario where the proportion of the mask area in the first mask sequence is greater than or equal to the proportion threshold, the cropping area calculated according to the above formulas (7)-(10) is larger, and the effect of the cropping process will be correspondingly reduced. In the case where the determined target mask M is too large, the effect of the cropping process is too small, which is equivalent to not cropping the first frame sequence and the first mask sequence. At this time, the first frame sequence and its corresponding first mask sequence can be scaled using the following method 2 to reduce the workload of processing the video to be processed, while not affecting the effect of video processing, thereby further improving the effect of video processing speed.

[0081] Method 2, obtaining the first target frame sequence and the first target mask sequence of the video to be processed, may include: obtaining the first frame sequence of the video to be processed and its corresponding first mask sequence; if the proportion of the mask area in the first mask sequence is greater than or equal to the proportion threshold, scaling the first frame sequence and the first mask sequence respectively to obtain the first target frame sequence and the first target mask sequence.

[0082] Exemplarily, all masks in the first mask sequence can be obtained, and when the proportion of any mask area in all masks to the corresponding frame image in the first frame sequence is greater than or equal to the proportion threshold, it is determined that the proportion of the mask area in the first mask sequence is greater than or equal to the proportion threshold. Alternatively, the target mask M can be determined first, and the target mask M is made to cover all masks in the first mask sequence. When the proportion of the area of ​​the mask area in the target mask M in the frame image in the first frame sequence is greater than or equal to the proportion threshold, it is determined that the proportion of the mask area in the first mask sequence is greater than or equal to the proportion threshold.

[0083] Exemplarily, the embodiments of the present disclosure do not limit the scaling ratio of the scaling processing of the first frame sequence and the first mask sequence. The scaling ratio can be limited based on experience or application scenarios. For example, the scaling ratio can be one-half or one-third of the side length of the original frame image and the mask. The scaling ratio will not cause a significant impact on the filling effect, and at the same time greatly reduces the number of processed pixels. In addition, for the scenario where the proportion of the mask area in the first mask sequence is less than the proportion threshold, but the distribution of the mask area is relatively dispersed, the method provided in the second method can also be used to obtain the first target frame sequence and the first target mask sequence. For example, when the mask is scattered in different areas of the image, such as the mask is located in the upper right corner and the lower left corner of the image, in this case, the coverage area of ​​the target mask M is too large, and the effect of cropping the first frame sequence and the first mask sequence is small.

[0084] Exemplarily, for a video to be processed with a small amount of data, the complete frame sequence and the complete mask sequence of the video to be processed can be directly used as the above-mentioned first frame sequence and its corresponding first mask sequence. However, for a video to be processed with a large amount of data, the memory may not be loaded, so that the video to be processed cannot be processed. In this case, obtaining the first target frame sequence and the first target mask sequence of the video to be processed may include: obtaining the complete frame sequence and the corresponding complete mask sequence of the video to be processed; performing lens segmentation processing on the complete frame sequence and the complete mask sequence respectively to obtain a single-lens frame sequence and a corresponding single-lens mask sequence; if the single-lens frame sequence and the corresponding single-lens mask sequence exceed the duration threshold, then segmenting the single-lens frame sequence and the corresponding single-lens mask sequence to obtain a segmented frame sequence and a corresponding segmented mask sequence, the segmented frame sequence includes a first frame sequence, and the segmented mask sequence includes a first mask sequence; according to the first frame sequence and the corresponding first mask sequence, the first target frame sequence and the first target mask sequence are obtained.

[0085] Exemplarily, if a complete frame sequence and a complete mask sequence are both composed of three lenses. Then, the complete frame sequence and the complete mask sequence are subjected to lens segmentation processing, and three single-lens frame sequences and three corresponding single-lens mask sequences can be obtained. If any single-lens frame sequence and the corresponding single-lens mask sequence among the three single-lens frame sequences and the three corresponding single-lens mask sequences exceed the duration threshold, the single-lens frame sequence and the corresponding single-lens mask sequence exceeding the duration threshold can be subjected to segmentation processing. The embodiment of the present disclosure does not limit the duration threshold, for example, the duration threshold may be 1 second or 2 seconds. The embodiment of the present disclosure also does not limit the number of segments for segmentation processing. Exemplarily, every 5 adjacent frames may be divided into one segment, or every 8 adjacent frames may be divided into one segment. After segmentation processing, multiple segmented frame sequences and multiple corresponding segmented mask sequences may be obtained.

[0086] The method for performing shot cutting and segmenting processing on a video to be processed provided by the embodiment of the present disclosure can avoid the problem that a large video to be processed cannot be loaded into the memory, thereby making it impossible to process the video to be processed.

[0087] S202, performing frame extraction processing on the first target frame sequence and the first target mask sequence respectively to obtain a first sampling frame sequence and a first remaining frame sequence thereof, and a first sampling mask sequence and a first remaining mask sequence thereof.

[0088] The embodiment of the present disclosure does not limit the frame extraction method. For example, one frame may be extracted from the first target frame sequence at intervals of 3 or 4 frames to obtain a first sampling frame sequence, and the unextracted frames constitute the first remaining frame sequence. Similarly, one frame may be extracted from the first target mask sequence at the same interval as the first target frame sequence to obtain a first sampling mask sequence, and the unextracted frames constitute the first remaining mask sequence.

[0089] S203, performing optical flow propagation and image restoration processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first restored sampling frame sequence.

[0090] In some embodiments, the first sampling mask sequence is used to indicate an area in the first sampling frame sequence that needs to be processed. In the area that needs to be processed, due to the removal of watermarks, moving objects, etc., an unknown area will be generated in the image of the first sampling frame sequence. Therefore, the unknown area in the first sampling frame sequence needs to be filled, so as to repair the first sampling frame sequence.

[0091] In an exemplary embodiment, optical flow propagation and image restoration processing are performed on the first sampling frame sequence and the first sampling mask sequence to obtain a first restored sampling frame sequence, including: performing optical flow propagation processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence; performing Poisson reconstruction processing on the first updated sampling gradient frame sequence and the first updated sampling mask sequence to obtain a first updated sampling frame sequence; performing frame-by-frame image restoration processing on the first updated sampling frame sequence to obtain a first restored sampling frame sequence.

[0092] In a possible implementation, the first updated sampling gradient frame sequence is a frame sequence obtained by filling the unknown region in the first sampling frame sequence. The first updated sampling mask sequence is a mask sequence corresponding to the first updated sampling gradient frame sequence. Exemplarily, the optical flow propagation processing of a frame image can be performed by using the previous frame image and the next frame image of the frame image. Figure 4 As shown, the previous frame of a certain frame can be used, that is, Figure 4 (a) Perform forward propagation to fill the unknown area in the frame image and obtain Figure 4 (b1). And through the next frame image of this frame image, that is Figure 4 (c) Perform back propagation to fill the unknown area in the frame image and obtain Figure 4 (b2).

[0093] For the area in the first updated sampling gradient frame sequence that has been filled compared to the first sampling frame sequence, the first sampling mask sequence needs to be updated accordingly, so as to obtain the first updated sampling mask sequence. For example, the value of the first sampling mask sequence corresponding to the above-mentioned filled area can be changed from 1 to 0, so as to obtain the first updated sampling mask sequence.

[0094] In an exemplary embodiment, after obtaining the first updated sampling gradient frame sequence and the first updated sampling mask sequence, the first updated sampling gradient frame sequence and the first updated sampling mask sequence may be subjected to Poisson reconstruction processing to obtain the first updated sampling frame sequence. The Poisson reconstruction processing is used to reconstruct the first updated sampling gradient frame sequence and the first updated sampling mask sequence of the gradient pixel pattern into the first updated sampling frame sequence of the RGB (Red Green Blue, three primary colors) pixel pattern. Exemplarily, each frame of the image is represented by a two-dimensional discrete function, and the two-dimensional discrete function is derived, so that the derivative value can be used to describe the pixel point at the corresponding position. This pixel description method can be called a gradient pixel pattern. For example, the gradient pixel G of the point (i, j) can be described according to the following formula (11).

[0095] G = dx(i, j) + dy(i, j) (11)

[0096] Among them, dx represents the operation of differentiating the independent variable, and dy represents the operation of differentiating the function value.

[0097] In addition, pixels can also be described using RGB values, which is called an RGB pixel mode. For example, the RGB value corresponding to a pixel may be (50, 80, 10), and the red value, green value, and blue value of the pixel are 50, 80, and 10, respectively.

[0098] Exemplarily, an octree may be constructed according to the first updated sampling gradient frame sequence and the first updated sampling mask sequence, and a spatial function may be set for each node of the octree, and the linear sum of all spatial functions may represent a vector field. Thereafter, the Poisson equation may be solved for the vector field, and the solution of the Poisson equation may be obtained by Laplace matrix iteration. The first updated sampling frame sequence may be obtained according to the solution of the Poisson equation.

[0099] In some embodiments, there may be unknown pixels in the first update sampling frame sequence that cannot be filled by the optical flow propagation process. The area composed of these unknown pixels that cannot be filled can be called an isolated area, which is included in the above-mentioned unknown area.

[0100] Exemplarily, the isolated region in the first updated sampling frame sequence may be subjected to frame-by-frame image restoration processing to obtain a first restored sampling frame sequence. Exemplarily, the isolated region may be subjected to image restoration using a pre-trained neural network model. The disclosed embodiments do not limit the type of the neural network model, for example, DeepFillv2 (a neural network model based on gated convolution) or CR-Fill (ContexutalReconstruction-Fill) may be used.

[0101] In an exemplary embodiment, performing optical flow propagation processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence may include: performing optical flow estimation processing on the first sampling frame sequence to obtain a first sampling optical flow sequence; performing gradient pixel conversion processing on the first sampling frame sequence to obtain a first sampling gradient frame sequence; performing optical flow filling processing on the mask area in the first sampling optical flow sequence to obtain a first filled sampling optical flow sequence; performing optical flow propagation processing on the first sampling gradient frame sequence and the first sampling mask sequence according to the first filled sampling optical flow sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence.

[0102] Exemplarily, in the method of optical flow estimation processing, any two adjacent images in the first sampling frame sequence can be input, and the pixels in the two adjacent images can be matched through optical flow estimation processing. Then the first sampling optical flow sequence can be output, and the first sampling optical flow sequence contains a forward optical flow map and a backward optical flow map corresponding to any frame image. The forward optical flow map is the optical flow map between the frame image and the previous frame image of the frame image, and the backward optical flow map is the optical flow map between the frame image and the next frame image of the frame image. Both the forward optical flow map and the backward optical flow map can indicate the offset of the image in the horizontal and vertical directions.

[0103] In an exemplary embodiment, the first sampling frame sequence may be subjected to a gradient pixel conversion process, thereby converting the first sampling frame sequence in an RGB pixel mode into a first sampling gradient frame sequence in a corresponding gradient pixel mode. Then, the first sampling optical flow sequence and the first sampling mask sequence are subjected to an optical flow filling process to obtain a first filled sampling optical flow sequence.

[0104] Exemplarily, the optical flow filling process is used to fill the mask area in the optical flow map. During the optical flow filling process, the Laplace equation can be solved for the corresponding mask area in the optical flow map to make the filled optical flow smoothly connect with the edge optical flow.

[0105] In an exemplary embodiment, during the optical flow filling process, the optical flow filling process can be performed on a frame image according to a forward optical flow map between a frame image and a previous frame image, and a backward optical flow map between the frame image and a next frame image, respectively, to obtain a first filled sampled optical flow sequence. Then, the optical flow propagation process can be performed on the first sampled gradient frame sequence and the first sampled mask sequence according to the matching relationship in the first filled sampled optical flow sequence, to obtain a first updated sampled gradient frame sequence and a first updated sampled mask sequence.

[0106] The disclosed embodiments do not limit the method used for the optical flow estimation process. In a possible implementation, a CNN (Convolutional Neural Network) can be used to perform the optical flow estimation process. For example, a CNN-based optical flow estimation module diagram can be as follows: Figure 5 As shown, the CNN-based optical flow estimation module is a lightweight optical flow estimation network module.

[0107] exist Figure 5In the method, two adjacent frames of images must first pass through the convolution layer to perform two convolution operations, and then perform two pooling operations to extract pyramid features. After that, the first cost volume corresponding to the pyramid feature can be obtained through the cost volume construction module, and then the convolution of the pyramid feature extracted from the previous frame of image and the first cost volume are input into the regressing optical flow module to obtain the predicted optical flow map corresponding to the previous frame of image. Exemplarily, the above-mentioned cost volume can be used to describe the degree of matching between corresponding pixels in two frames of images. The regressing optical flow module can be used to estimate the optical flow of corresponding feature points in two frames of images through a regression model, thereby obtaining the corresponding predicted optical flow map.

[0108] After that, the predicted optical flow map corresponding to the previous frame image can be upsampled to obtain the upsampled result. The upsampled structure and the result of the next frame image after a pooling are input into the warp processing module to obtain the warping result. Exemplarily, the warping process can be used to transform the changes such as the lens position change or the object change in the image, so that the corresponding pixels in the two frames of images can be aligned. The warping process can narrow the processing range corresponding to the optical flow estimation process to reduce the amount of calculation of the optical flow estimation module. Then, the warping result and the result of the previous frame image after a pooling can be input into the cost volume construction module to obtain the second cost volume. And the convolution of the result of the next frame image after a pooling and the second cost volume are input into the regression optical flow module to obtain the predicted optical flow map corresponding to the next frame image. Correspondingly, upsampling and other operations can be performed based on the predicted optical flow map corresponding to the next frame image to obtain the predicted optical flow map corresponding to the next frame image.

[0109] The embodiments of the present disclosure can be Figure 5 The optical flow estimation module in the optical flow estimation module performs optical flow estimation processing, and the speed of optical flow estimation can reach 91FPS (Frames Per Second), which is 27 times that of the RAFT network. Therefore, the embodiment of the present disclosure can improve the speed of optical flow estimation processing and obtain a good optical flow estimation effect. Thereby, the efficiency of the video processing method for processing videos can be improved and the quality of video processing can be ensured.

[0110] In an exemplary embodiment, the process diagram of optical flow estimation and optical flow filling can be as follows: Figure 6 As shown. Figure 6 middle, Figure 6 (a) is a schematic diagram of an image in the first sampling frame sequence, as shown in FIG. Figure 6As shown in , the image contains the mask area. By performing optical flow estimation on the image, we can get Figure 6 (b) shows the predicted optical flow map, which is included in the first sampled optical flow sequence. After that, the predicted optical flow map is processed by the optical flow filling machine, and the following can be obtained: Figure 6 (c) is a filled optical flow map shown in FIG. 1 , which is contained in the first filled sampled optical flow sequence.

[0111] S204, performing optical flow propagation and image restoration processing on the first restored sampling frame sequence, the first remaining frame sequence, and the first remaining mask sequence to obtain a first restored remaining frame sequence.

[0112] In an exemplary embodiment, performing optical flow propagation and image restoration processing on the first repaired sampling frame sequence, the first remaining frame sequence, and the first remaining mask sequence to obtain the first repaired remaining frame sequence may include: performing gradient pixel conversion processing on the first remaining frame sequence to obtain a first remaining gradient frame sequence; obtaining a first remaining optical flow sequence based on the first remaining frame sequence and the first sampling frame sequence; performing optical flow filling processing on the mask area in the first remaining optical flow sequence to obtain a first filled remaining optical flow sequence; performing optical flow propagation and image restoration on the first remaining gradient frame sequence, the first filled remaining optical flow sequence, the first repaired sampling frame sequence, and the first remaining mask sequence to obtain a first repaired remaining frame sequence.

[0113] In some embodiments, the method of performing gradient pixel conversion processing on the first residual frame sequence to obtain the first residual gradient frame sequence may be the same as the method of performing gradient pixel conversion processing on the first sampling frame sequence described above, which will not be described in detail herein.

[0114] In an exemplary embodiment, obtaining a first residual optical flow sequence based on a first residual frame sequence and a first sampling frame sequence may include: determining a target residual frame in the first residual frame sequence; determining a target forward frame and a target backward frame of the target residual frame from the first sampling frame sequence; obtaining a forward optical flow map between the target forward frame and the target residual frame, and a backward optical flow map between the target backward frame and the target residual frame; and determining the first residual optical flow sequence based on the forward optical flow map and the backward optical flow map.

[0115] The target residual frame is not limited in the embodiment of the present disclosure, and the target residual frame can be any frame in the first residual frame sequence. The target forward frame of the target residual frame is a frame that is in front of the target residual frame in the first sampling frame sequence and is closest to the target residual frame; the target backward frame of the target residual frame is a frame that is behind the target residual frame in the first sampling frame sequence and is closest to the target residual frame.

[0116] Exemplarily, assuming that the first sampling frame sequence can be {X_0, X_4, X_8, ...}, and the first remaining frame sequence can be {X_1, X_2, X_3, X_5, X_6, X_7, X_9, ...}, when the target remaining frame is X_1, the target forward frame of the target remaining frame is X_0, and the target backward frame is X_4. After determining the target forward frame and the target backward frame of the target remaining frame, the forward optical flow map between the target forward frame and the target remaining frame, and the backward optical flow map between the target backward frame and the target remaining frame can be obtained by the optical flow estimation processing method. Afterwards, the first remaining optical flow sequence can be determined based on the forward optical flow maps and the backward optical flow maps of multiple target remaining frames. Exemplarily, the optical flow estimation processing method can be the same as the above-mentioned method for performing optical flow estimation processing on the first sampling frame sequence, which will not be repeated here.

[0117] In some embodiments, the method of performing optical flow filling processing on the mask area in the first residual optical flow sequence may be the same as the method of performing optical flow filling processing on the first sampled optical flow sequence described above, which will not be described in detail here. Then, the first residual gradient frame sequence may be subjected to optical flow propagation and image restoration processing through the first filled residual optical flow sequence and the first restored sampled frame sequence to obtain a first restored residual frame sequence. Exemplarily, the method of performing optical flow propagation and image restoration processing on the first residual gradient frame sequence may be the same as the method of performing optical flow propagation and image restoration processing on the first sampled frame sequence and the first sampled mask sequence described above, which will not be described in detail here.

[0118] Optionally, while performing optical flow propagation processing on the first residual gradient frame sequence, the first residual mask sequence may be correspondingly updated to obtain a first updated residual mask sequence. The corresponding updating of the first residual mask sequence may be the same as the method for correspondingly updating the first sampling mask sequence, which will not be described in detail herein.

[0119] In an exemplary embodiment, a schematic diagram of a process of obtaining a first repaired residual frame sequence can be as follows: Figure 7 As shown. The first target frame sequence can be processed by extracting frames. Figure 7In the example, it is assumed that the first target frame sequence includes 13 frames from frame X_0 to frame X_12, and the first target frame sequence is subjected to frame extraction processing to obtain a first sampling frame sequence including frame X_0, frame X_4, frame X_8, and frame X_12. Then, the first sampling frame sequence can be subjected to optical flow propagation processing and image restoration processing to obtain a first restored sampling frame sequence. Then, since the target forward frame of frame X_1 is frame X_0 and the target backward frame is frame X_4, the filled optical flow map corresponding to frame X_1 can be obtained through frame X_0 and frame X_4. Similarly, the filled optical flow maps corresponding to frames X_2 and X_3 can be obtained through frames X_0 and X_4. According to the above steps, the filled optical flow maps corresponding to frames X_5, X_6, and X_7 can be obtained in sequence through frames X_4 and X_8; and the filled optical flow maps corresponding to frames X_9, X_10, and X_11 can be obtained in sequence through frames X_8 and X_12, which will not be repeated here.

[0120] S205: merge the first restored sampling frame sequence and the first restored remaining frame sequence to obtain a restored video of the video to be processed.

[0121] In some embodiments, each frame image in the first restored sample frame sequence can be inserted into the first restored remaining frame sequence in the order of frames, so as to obtain a complete restored frame sequence, which can correspond to the restored video of the video to be processed.

[0122] In an exemplary embodiment, before the first sampling frame sequence and the first remaining frame sequence are processed to obtain the first repaired sampling frame sequence and the first repaired remaining frame sequence, the video to be processed may be subjected to shot cutting processing and / or segmentation processing. At this time, the segmented frame sequence may also include a second frame sequence, and the segmented mask sequence may also include a second mask sequence. In this case, merging the first repaired sampling frame sequence and the first repaired remaining frame sequence to obtain the repaired video of the video to be processed may include: inserting the tail frames of the first repaired sampling frame sequence and the first repaired remaining frame sequence into the second frame sequence and the second mask sequence, respectively, as the first frames of the second frame sequence and the second mask sequence; processing the second frame sequence and the second mask sequence to obtain the second repaired sampling frame sequence and the second repaired remaining frame sequence; merging the first repaired sampling frame sequence and the first repaired remaining frame sequence, and the second repaired sampling frame sequence and the second repaired remaining frame sequence to obtain the repaired video.

[0123] For example, a schematic diagram of a video processing process can be as follows: Figure 8 As shown. Figure 8 In the example, each 4 consecutive frames can be divided into a segment, so a 12-frame video to be processed can be divided into the following segments: Figure 8 The three segments shown are frame x_0 to frame x_3, frame x_4 to frame x_7, and frame x_8 to frame x_11. Afterwards, the segmented frame sequence and segmented mask sequence corresponding to each video segment can be repaired according to the order of the frames. Therefore, the segmented frame sequence and segmented mask sequence corresponding to frame x_0 to frame x_3 can be processed first to obtain the repaired video corresponding to frame x_0 to frame x_3, that is, Figure 8 The repaired {X_0, X_1, X_2, X_3} shown in . Afterwards, the last frame in the repaired video, i.e., frame x_3, can be inserted into the beginning of the segmented frame sequence and segmented mask sequence corresponding to frames x_4 to x_7. Afterwards, the segmented frame sequence and segmented mask sequence corresponding to frames x_3 to x_7 can be repaired to obtain the repaired video corresponding to frames x_4 to x_7, i.e. Figure 8 The repaired {X_3, X_4, X_5, X_6, X_7} shown in . The method for processing the segmented frame sequence and the segmented mask sequence corresponding to frames x_8 to x_11 is similar. Through the method of inserting the tail frame, when processing the segmented frame sequence and the segmented mask sequence, the information in the previous repaired video can be obtained. In the embodiment of the present disclosure, after the video to be processed is shot-sliced ​​and / or segmented, each segmented frame sequence and its corresponding segmented mask sequence are sequentially repaired. At the same time, the last frame of each repaired segmented frame sequence is inserted as the first frame into the next segmented frame sequence for repairing the next segmented frame sequence. This method can prevent the loss of information between segments caused by simple segmentation, and therefore, a better video processing effect can be obtained.

[0124] It should be noted that the above description of dividing each continuous 4 frames into one segment and dividing a 12-frame video into 3 segments is only an example and is not used to limit the above-mentioned tail frame insertion method.

[0125] The method provided by the embodiment of the present disclosure, on the one hand, can firstly process the first sampling frame sequence and the first sampling mask sequence by the frame extraction method to obtain the first repaired sampling frame sequence. Then, the first remaining frame sequence is processed by the first repaired sampling frame sequence and the first remaining mask sequence to obtain the repaired video of the video to be processed. Therefore, the present disclosure can process the remaining frame sequence by the first repaired sampling frame sequence by the frame extraction process, avoid multiple rounds of processing for each frame image, reduce the amount of calculation, and thus improve the efficiency of video processing. On the other hand, it can also be processed by shot segmentation and / or video segmentation, so as to effectively avoid the problem of insufficient memory caused by large video. In addition, by inserting the last frame of each repaired segmented frame sequence as the first frame into the next segmented frame sequence for repairing the next segmented frame sequence, the method can prevent the loss of information between segments caused by simple segmentation, so as to obtain a better video processing effect.

[0126] like Fig. 9 As shown, Fig. 9 FIG. 4 is a flowchart of a video processing method provided by an embodiment of the present disclosure. For example, the first target frame sequence can be represented as X={X t}, (t=0, 1, 2, ..., T), the first target mask sequence can be expressed as M={M t The purpose of this scheme is to obtain the processed video sequence y = {y t}, (t = 0, 1, 2, ..., T), the processed video sequence y differs from X only in the mask region, making y natural and consistent in time and space. Since natural and consistent are difficult to define in a formula, this solution makes the processed video sequence y and y gt Close to, where y gt Used to represent the true value of the video sequence without the mask.

[0127] First, a complete frame sequence and a complete mask sequence of the video to be restored can be obtained, and the complete frame sequence and the complete mask sequence can be shot-cut and / or segmented to obtain a first frame sequence X_1 and a first mask sequence M_1. Afterwards, the first frame sequence X_1 and the first mask sequence M_1 can be cropped or scaled by a DPM (Data Preprocessing Module) to obtain a first target frame sequence X and a first target mask sequence M.

[0128] Then, the first target frame sequence X can be processed by frame extraction to obtain the first sampling frame sequence X_sampled and the first remaining frame sequence X_remain. Exemplarily, if 1 frame is extracted from every 4 frames of the first target frame sequence, then X_sampled = {X_0, X_4, X_8, ...}. If the tail frame of the first target frame sequence X is not selected, the tail frame is added to the first sampling frame sequence X_sampled. And the first target mask sequence M is processed by frame extraction to obtain the first sampling mask sequence M_sampled and the first remaining mask sequence X_remain. X_sampled and M_sampled can then be input into FPM (Flow Propagation Module) and IIM (Image Inpainting Module) to output the restored subset. Different from the multi-round propagation scheme, here, we only use the FPM module for one round of filling, and for the unfilled isolated areas, we use IIM to fill frame by frame.

[0129] exist Fig. 9 In the method, the first sampling frame sequence X_sampled can be subjected to gradient pixel conversion processing to obtain a first sampling gradient frame sequence X_sgrad, and the first sampling frame sequence X_sampled can be input into a lightweight FEM (Flow Estimation Module, optical flow estimation module) for optical flow estimation processing to obtain a first sampling optical flow sequence F_sampled. The first sampling optical flow sequence F_sampled is input into a FCM (Flow Completion Module optical flow filling module) for optical flow filling processing to obtain a first filled sampling optical flow sequence F_scomp.

[0130] Then, the first filled sampled optical flow sequence F_scomp, the first sampled gradient frame sequence X_sgrad and the first sampled mask sequence M_sampled can be input into the FPM module for optical flow propagation processing to obtain the first updated sampled gradient frame sequence X_sgrad_fill and the first updated sampled mask sequence M_sfill. Exemplarily, the FPM and frame extraction processing here can be included in the SPM (Sampled Propagation Module).

[0131] The first updated sampling gradient frame sequence X_sgrad_fill and the first updated sampling mask sequence M_sfill are then input into a Poisson Reconstruction Module (PRM) for Poisson reconstruction processing to obtain a first updated sampling frame sequence X_sfill_fpm. The first updated sampling frame sequence is subjected to frame-by-frame image restoration processing through an IIM to obtain a first restored sampling frame sequence X_sfill.

[0132] Similarly, the first remaining frame sequence X_remain can be subjected to gradient pixel conversion processing to obtain a first remaining gradient frame sequence X_rgrad. The first remaining frame sequence X_remain and the first sampled frame sequence X_sampled are input into the lightweight FEM for optical flow filling processing to obtain a first remaining optical flow sequence. The mask area in the first remaining optical flow sequence is subjected to optical flow filling processing to obtain a first filled remaining optical flow sequence F_rcomp.

[0133] Then, the first residual gradient frame sequence X_rgrad, the first filled residual optical flow sequence F_rcomp, the first repaired sampled frame sequence X_sfill and the first residual mask sequence X_remain can be input into FPM and PRM for optical flow filling and Poisson reconstruction to obtain the first updated residual frame sequence X_rgrad_fill. The first updated residual frame sequence X_rgrad_fill is subjected to frame-by-frame image restoration through IIM to obtain the first repaired residual frame sequence X_rfill. Optionally, for each unselected frame X_remain, find the closest forward frame X_prev and backward frame X_next in X_sampled. Exemplarily, for the above-mentioned case of extracting one frame every four frames of the first target frame sequence, assuming that X_remain is frame X_1, then the closest forward frame X_prev is frame X_0, and the closest backward frame X_next is frame X_4.

[0134] The optical flow propagation process is performed between X_remain and the forward frame X_prev and the backward frame X_next respectively, and the pixels of the corresponding area are filled in using the optical flow. If the same position has pixels filled, the pixels are averaged. The unfilled area is filled using the IIM module alone. Finally, the first repaired remaining frame sequence is merged with the first repaired sampling frame sequence to obtain the repaired video of the video to be processed.

[0135] In the disclosed embodiment, the amount of calculation performed by the FPM module is relatively large. In order to reduce the time consumption of the optical flow propagation module, this solution first extracts frames from the frame sequence before optical flow propagation. After the extracted frames are filled, we use the optical flow to propagate the contents of the forward frame and the backward frame to the remaining frames at one time. Therefore, this solution only performs one round of optical flow propagation processing, which reduces the amount of calculation of the FPM module and reduces the time consumption. Afterwards, the IIM module can be used to perform frame-by-frame image restoration processing on the isolated areas so that each frame is filled completely. Avoiding the use of multiple rounds of optical flow filling to process the isolated areas here can reduce the amount of calculation of the FPM module. Therefore, this solution can accelerate the video processing algorithm to achieve commercially available processing efficiency.

[0136] For this Fig. 9 The provided video processing method can avoid multiple rounds of optical flow propagation to speed up the deployment of video processing algorithms. As shown in Table 1, for a 188-frame lens, the process of preprocessing the video through DPM in this scheme, that is, the process of cropping or scaling the video, takes about 267.8 seconds, and the multi-round optical flow propagation scheme takes about 267.8 seconds, which is about 67.4% less time. The process of performing optical flow estimation on the video through FEM in this scheme takes about 217.4 seconds, which is about 73.5% less time than the multi-round optical flow propagation scheme. The process of performing frame extraction and optical flow propagation on the video through SPM in this scheme takes about 88.2 seconds. Since frame extraction is performed before optical flow propagation, the time can be reduced by about 89.3% compared with the multi-round optical flow propagation scheme.

[0137] Table 1

[0138]

[0139]

[0140] For example, a schematic diagram of video processing effect for removing watermarks can be as follows: Fig.10 As shown. Fig.10 In (a), there is an elliptical watermark in the lower left corner of the video to be processed. The method provided by the embodiment of the present disclosure can remove the elliptical watermark. The effect after removal can be as follows: Fig.10 (b) As shown. For example, a schematic diagram of a video processing effect for removing subtitles can be as follows: Fig.11 As shown. Fig.11 In (a), there is a line of subtitles below the video to be processed. The method provided by the embodiment of the present disclosure can remove the line of subtitles. The effect after removal can be as follows: Fig.11 (b) As shown. For example, a schematic diagram of a video processing effect for removing moving objects can be as follows: Fig.12 As shown. Fig.12 In (a), there is a person riding a bicycle in the video to be processed. The method provided by the embodiment of the present disclosure can remove the person riding a bicycle. The effect after removal can be as follows: Fig.12 (b) as shown.

[0141] Fig.13 Schematic diagram of a video processing device provided by an embodiment of the present disclosure. Fig.13 As shown, the video processing device provided by the embodiment of the present disclosure may include an acquisition module 1301 and a frame extraction module 1302 .

[0142] An acquisition module 1301 is used to acquire a first target frame sequence and a first target mask sequence of a video to be processed;

[0143] The frame extraction module 1302 is used to perform frame extraction processing on the first target frame sequence and the first target mask sequence respectively to obtain a first sampling frame sequence and a first remaining frame sequence thereof, and a first sampling mask sequence and a first remaining mask sequence thereof;

[0144] The acquisition module 1301 is further used to perform optical flow propagation and image restoration processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first restored sampling frame sequence;

[0145] The acquisition module 1301 is further used to perform optical flow propagation and image restoration processing on the first restored sampling frame sequence, the first remaining frame sequence and the first remaining mask sequence to obtain a first restored remaining frame sequence;

[0146] The acquisition module 1301 is further used to merge the first restored sampling frame sequence and the first restored remaining frame sequence to obtain a restored video of the video to be processed.

[0147] In an exemplary embodiment, the acquisition module 1301 is used to perform gradient pixel conversion processing on the first residual frame sequence to obtain a first residual gradient frame sequence; obtain a first residual optical flow sequence based on the first residual frame sequence and the first sampling frame sequence; perform optical flow filling processing on the mask area in the first residual optical flow sequence to obtain a first filled residual optical flow sequence; perform optical flow propagation and image restoration processing on the first residual gradient frame sequence, the first filled residual optical flow sequence, the first repaired sampling frame sequence and the first residual mask sequence to obtain a first repaired residual frame sequence.

[0148] In an exemplary embodiment, an acquisition module 1301 is used to determine a target remaining frame in a first remaining frame sequence; determine a target forward frame and a target backward frame of the target remaining frame from a first sampling frame sequence; obtain a forward optical flow map between the target forward frame and the target remaining frame, and a backward optical flow map between the target backward frame and the target remaining frame; and determine a first remaining optical flow sequence based on the forward optical flow map and the backward optical flow map.

[0149] In an exemplary embodiment, the acquisition module 1301 is used to perform optical flow propagation processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence; perform Poisson reconstruction processing on the first updated sampling gradient frame sequence and the first updated sampling mask sequence to obtain a first updated sampling frame sequence; perform frame-by-frame image restoration processing on the first updated sampling frame sequence to obtain a first restored sampling frame sequence.

[0150] In an exemplary embodiment, the acquisition module 1301 is used to perform optical flow estimation processing on the first sampling frame sequence to obtain a first sampling optical flow sequence; perform gradient pixel conversion processing on the first sampling frame sequence to obtain a first sampling gradient frame sequence; perform optical flow filling processing on the mask area in the first sampling optical flow sequence to obtain a first filled sampling optical flow sequence; perform optical flow propagation processing on the first sampling gradient frame sequence and the first sampling mask sequence according to the first filled sampling optical flow sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence.

[0151] In an exemplary embodiment, the acquisition module 1301 is used to obtain a first frame sequence of a video to be processed and its corresponding first mask sequence; if the proportion of the mask area in the first mask sequence is less than a proportion threshold, the target mask is determined according to the first mask sequence, and the mask area of ​​the target mask covers the mask area of ​​the first mask sequence; the first frame sequence and the first mask sequence are respectively cropped according to the target mask to obtain a first target frame sequence and a first target mask sequence.

[0152] In an exemplary embodiment, the acquisition module 1301 is used to obtain the number of outwardly extended pixels of the mask area of ​​the target mask according to the length and width of the mask area of ​​the target mask; obtain the cropping boundary of the mask area of ​​the target mask according to the length, width and the number of outwardly extended pixels of the mask area of ​​the target mask; obtain a cropping area including the mask area of ​​the target mask according to the cropping boundary of the mask area of ​​the target mask and the length and width of the mask area of ​​the target mask; and perform cropping processing on the first frame sequence and the first mask sequence respectively according to the cropping area to obtain a first target frame sequence and a first target mask sequence.

[0153] In an exemplary embodiment, the acquisition module 1301 is used to obtain a first frame sequence of a video to be processed and its corresponding first mask sequence; if the proportion of the mask area in the first mask sequence is greater than or equal to a proportion threshold, the first frame sequence and the first mask sequence are scaled respectively to obtain a first target frame sequence and a first target mask sequence.

[0154] In an exemplary embodiment, the acquisition module 1301 is used to obtain a complete frame sequence and a corresponding complete mask sequence of a video to be processed; perform lens segmentation processing on the complete frame sequence and the complete mask sequence respectively to obtain a single-lens frame sequence and a corresponding single-lens mask sequence; if the single-lens frame sequence and the corresponding single-lens mask sequence exceed a duration threshold, segment the single-lens frame sequence and the corresponding single-lens mask sequence to obtain a segmented frame sequence and a corresponding segmented mask sequence, the segmented frame sequence includes a first frame sequence, and the segmented mask sequence includes a first mask sequence; obtain a first target frame sequence and a first target mask sequence according to the first frame sequence and its corresponding first mask sequence.

[0155] In an exemplary embodiment, the segmented frame sequence also includes a second frame sequence, and the segmented mask sequence also includes a second mask sequence; the acquisition module 1301 is used to insert the tail frames of the first repaired sampling frame sequence and the first repaired remaining frame sequence into the second frame sequence and the second mask sequence, respectively, as the first frames of the second frame sequence and the second mask sequence; the second frame sequence and the second mask sequence are processed to obtain a second repaired sampling frame sequence and a second repaired remaining frame sequence; and the first repaired sampling frame sequence and the first repaired remaining frame sequence, and the second repaired sampling frame sequence and the second repaired remaining frame sequence are used to obtain a repaired video.

[0156] The device provided by the embodiment of the present disclosure can first process the first sampling frame sequence and the first sampling mask sequence through the frame extraction processing method to obtain the first repaired sampling frame sequence. Then, the first remaining frame sequence is processed through the first repaired sampling frame sequence and the first remaining mask sequence to obtain the repaired video of the video to be processed. Therefore, the present disclosure can process the remaining frame sequence through the first repaired sampling frame sequence through the frame extraction processing, avoid multiple rounds of processing for each frame image, reduce the amount of calculation, and thus improve the efficiency of video processing.

[0157] See also Fig.14 , Fig.14 Schematic diagram of a computer device provided by an embodiment of the present disclosure. Fig.14As shown, the computer device in the embodiment of the present disclosure may include: one or more processors 1401, a memory 1402, and an input-output interface 1403. The processor 1401, the memory 1402, and the input-output interface 1403 are connected via a bus 1404. The memory 1402 is used to store a computer program, which includes program instructions. The input-output interface 1403 is used to receive data and output data, such as for data interaction between a host machine and a computer device, or for data interaction between various virtual machines in a host machine; the processor 1401 is used to execute the program instructions stored in the memory 1402.

[0158] Among them, the processor 1401 can perform the following operations: obtain the first target frame sequence and the first target mask sequence of the video to be processed; perform frame extraction processing on the first target frame sequence and the first target mask sequence respectively to obtain a first sampling frame sequence and its first residual frame sequence, and a first sampling mask sequence and its first residual mask sequence; obtain a first repaired sampling frame sequence according to the first sampling frame sequence and the first sampling mask sequence; obtain a first repaired residual frame sequence according to the first repaired sampling frame sequence, the first residual frame sequence and the first residual mask sequence; obtain a repaired video of the video to be processed according to the first repaired sampling frame sequence and the first repaired residual frame sequence.

[0159] In some feasible implementations, the processor 1401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0160] The memory 1402 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1401 and the input / output interface 1403. A portion of the memory 1402 may also include a non-volatile random access memory. For example, the memory 1402 may also store information about the device type.

[0161] In a specific implementation, the computer device can execute the video processing implementation method provided in each step of any of the above method embodiments through its built-in functional modules. For details, please refer to the implementation method provided in each step in the figure shown in the above method embodiment, which will not be repeated here.

[0162] The embodiments of the present disclosure provide a computer device, including a processor, an input / output interface, and a memory. The processor obtains a computer program in the memory to execute each step of the method shown in any of the above embodiments.

[0163] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by the processor and executing the training method of the acoustic model provided in each step of any of the above embodiments. For details, please refer to the implementation method provided in each step of any of the above embodiments, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present disclosure, please refer to the description of the method embodiment of the present disclosure. As an example, the computer program can be deployed to be executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network.

[0164] The computer-readable storage medium may be the video processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0165] The present disclosure also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in any of the above-mentioned optional modes.

[0166] The terms "first", "second", etc. in the specification, claims, and drawings of the embodiments of the present disclosure are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other step units inherent to these processes, methods, devices, products, or equipment.

[0167] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0168] The method and related apparatus provided by the embodiments of the present disclosure are described with reference to the method flow chart and / or structural diagram provided by the embodiments of the present disclosure. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable application display device to generate a machine, so that the instructions executed by the processor of the computer or other programmable application display device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable application display device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable application display device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0169] The above disclosure is only a preferred embodiment of the present disclosure, which certainly cannot be used to limit the scope of rights of the present disclosure. Therefore, equivalent changes made according to the claims of the present disclosure are still within the scope of the present disclosure.

Claims

1. A video processing method, characterized in that: include: Acquire a first target frame sequence and a first target mask sequence of a video to be processed; Performing frame extraction processing on the first target frame sequence and the first target mask sequence respectively to obtain a first sampling frame sequence and a first remaining frame sequence thereof, and a first sampling mask sequence and a first remaining mask sequence thereof; Performing optical flow propagation and image restoration processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first restored sampling frame sequence; The first restored sampling frame sequence, the first remaining frame sequence and the first remaining mask sequence are subjected to optical flow propagation and image restoration processing to obtain a first restored remaining frame sequence, which includes: performing gradient pixel conversion processing on the first remaining frame sequence to obtain a first remaining gradient frame sequence; obtaining a first remaining optical flow sequence according to the first remaining frame sequence and the first sampling frame sequence; performing optical flow filling processing on the mask area in the first remaining optical flow sequence to obtain a first filled remaining optical flow sequence; performing optical flow propagation and image restoration processing on the first remaining gradient frame sequence, the first filled remaining optical flow sequence, the first restored sampling frame sequence and the first remaining mask sequence to obtain the first restored remaining frame sequence; The first repaired sampling frame sequence and the first repaired remaining frame sequence are merged to obtain a repaired video of the video to be processed.

2. The method according to claim 1, characterized in that Obtaining a first residual optical flow sequence according to the first residual frame sequence and the first sampling frame sequence, comprising: determining a target remaining frame in the first remaining frame sequence; Determining a target forward frame and a target backward frame of the target residual frame from the first sampling frame sequence; Obtaining a forward optical flow map between the target forward frame and the target residual frame, and a backward optical flow map between the target backward frame and the target residual frame; The first residual optical flow sequence is determined according to the forward optical flow map and the backward optical flow map.

3. The method according to any one of claims 1 to 2, characterized in that: Performing optical flow propagation and image restoration processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first restored sampling frame sequence, including: Performing optical flow propagation processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence; Performing Poisson reconstruction processing on the first updated sampling gradient frame sequence and the first updated sampling mask sequence to obtain a first updated sampling frame sequence; Performing frame-by-frame image restoration processing on the first updated sampling frame sequence to obtain the first restored sampling frame sequence.

4. The method according to claim 3, characterized in that Performing optical flow propagation processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence, including: Performing optical flow estimation processing on the first sampling frame sequence to obtain a first sampling optical flow sequence; Performing gradient pixel conversion processing on the first sampling frame sequence to obtain a first sampling gradient frame sequence; Performing optical flow filling processing on the mask area in the first sampled optical flow sequence to obtain a first filled sampled optical flow sequence; Optical flow propagation processing is performed on the first sampled gradient frame sequence and the first sampled mask sequence according to the first filled sampled optical flow sequence to obtain the first updated sampled gradient frame sequence and the first updated sampled mask sequence.

5. The method according to any one of claims 1 to 2, characterized in that: Obtaining a first target frame sequence and a first target mask sequence of a video to be processed, including: Acquire a first frame sequence of the video to be processed and a first mask sequence corresponding thereto; If the proportion of the mask area in the first mask sequence is less than the proportion threshold, determining a target mask according to the first mask sequence, the mask area of ​​the target mask covering the mask area of ​​the first mask sequence; The first frame sequence and the first mask sequence are respectively cropped according to the target mask to obtain the first target frame sequence and the first target mask sequence.

6. The method according to claim 5, characterized in that The first frame sequence and the first mask sequence are respectively cropped according to the target mask to obtain the first target frame sequence and the first target mask sequence, including: According to the length and width of the mask area of ​​the target mask, obtaining the number of outwardly extended pixels of the mask area of ​​the target mask; Obtaining a clipping boundary of the mask area of ​​the target mask according to the length and width of the mask area of ​​the target mask and the number of outwardly extended pixels; Obtaining a cropping region including the mask region of the target mask according to a cropping boundary of the mask region of the target mask and a length and a width of the mask region of the target mask; According to the cropping area, the first frame sequence and the first mask sequence are cropped respectively to obtain the first target frame sequence and the first target mask sequence.

7. The method according to any one of claims 1 to 2, characterized in that: Obtaining a first target frame sequence and a first target mask sequence of a video to be processed, including: Acquire a first frame sequence of the video to be processed and a first mask sequence corresponding thereto; If the proportion of the mask area in the first mask sequence is greater than or equal to a proportion threshold, scaling processing is performed on the first frame sequence and the first mask sequence respectively to obtain the first target frame sequence and the first target mask sequence.

8. The method according to any one of claims 1 to 2, characterized in that: Obtaining a first target frame sequence and a first target mask sequence of a video to be processed, including: Obtaining a complete frame sequence of the video to be processed and a corresponding complete mask sequence; Performing shot segmentation processing on the complete frame sequence and the complete mask sequence respectively to obtain a single-shot frame sequence and a corresponding single-shot mask sequence; If the single-shot frame sequence and the corresponding single-shot mask sequence exceed the duration threshold, segmenting the single-shot frame sequence and the corresponding single-shot mask sequence to obtain a segmented frame sequence and a corresponding segmented mask sequence, wherein the segmented frame sequence includes a first frame sequence, and the segmented mask sequence includes a first mask sequence; The first target frame sequence and the first target mask sequence are obtained according to the first frame sequence and the first mask sequence corresponding thereto.

9. The method according to claim 8, characterized in that The segmented frame sequence further includes a second frame sequence, and the segmented mask sequence further includes a second mask sequence; The step of merging the first repaired sampling frame sequence and the first repaired remaining frame sequence to obtain a repaired video of the video to be processed includes: Inserting the tail frames of the first repaired sampling frame sequence and the first repaired remaining frame sequence into the second frame sequence and the second mask sequence respectively as the first frames of the second frame sequence and the second mask sequence; Processing the second frame sequence and the second mask sequence to obtain a second repaired sampling frame sequence and a second repaired remaining frame sequence; The first repaired sampling frame sequence and the first repaired remaining frame sequence, the second repaired sampling frame sequence and the second repaired remaining frame sequence are merged to obtain the repaired video.

10. A video processing device, characterized in that: include: An acquisition module, used to acquire a first target frame sequence and a first target mask sequence of a video to be processed; A frame extraction module, used for performing frame extraction processing on the first target frame sequence and the first target mask sequence respectively, to obtain a first sampling frame sequence and a first remaining frame sequence thereof, and a first sampling mask sequence and a first remaining mask sequence thereof; The acquisition module is further used to perform optical flow propagation and image restoration processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first restored sampling frame sequence; The acquisition module is also used to perform optical flow propagation and image restoration processing on the first restored sampling frame sequence, the first remaining frame sequence and the first remaining mask sequence to obtain a first restored remaining frame sequence, which includes: performing gradient pixel conversion processing on the first remaining frame sequence to obtain a first remaining gradient frame sequence; According to the first remaining frame sequence and the first sampling frame sequence, a first remaining optical flow sequence is obtained; an optical flow filling process is performed on the mask area in the first remaining optical flow sequence to obtain a first filled remaining optical flow sequence; and optical flow propagation and image restoration processes are performed on the first remaining gradient frame sequence, the first filled remaining optical flow sequence, the first restored sampling frame sequence, and the first remaining mask sequence to obtain the first restored remaining frame sequence; The acquisition module is further used to merge the first repaired sampling frame sequence and the first repaired remaining frame sequence to obtain the repaired video of the video to be processed.

11. The device according to claim 10, characterized in that The acquisition module is used to: determine a target remaining frame in the first remaining frame sequence; determine a target forward frame and a target backward frame of the target remaining frame from the first sampling frame sequence; Obtain a forward optical flow map between the target forward frame and the target residual frame, and a backward optical flow map between the target backward frame and the target residual frame; and determine the first residual optical flow sequence according to the forward optical flow map and the backward optical flow map.

12. The device according to any one of claims 10 to 11, characterized in that The acquisition module is used to: perform optical flow propagation processing on the first sampling frame sequence and the first sampling mask sequence to obtain a first updated sampling gradient frame sequence and a first updated sampling mask sequence; perform Poisson reconstruction processing on the first updated sampling gradient frame sequence and the first updated sampling mask sequence to obtain a first updated sampling frame sequence; Performing frame-by-frame image restoration processing on the first updated sampling frame sequence to obtain the first restored sampling frame sequence.

13. The device according to claim 12, characterized in that The acquisition module is used to: perform optical flow estimation processing on the first sampling frame sequence to obtain a first sampling optical flow sequence; perform gradient pixel conversion processing on the first sampling frame sequence to obtain a first sampling gradient frame sequence; perform optical flow filling processing on the mask area in the first sampling optical flow sequence to obtain a first filled sampling optical flow sequence; perform optical flow propagation processing on the first sampling gradient frame sequence and the first sampling mask sequence according to the first filled sampling optical flow sequence to obtain the first updated sampling gradient frame sequence and the first updated sampling mask sequence.

14. The device according to any one of claims 10 to 11, characterized in that The acquisition module is used to: acquire a first frame sequence of the video to be processed and a first mask sequence corresponding thereto; if the proportion of the mask area in the first mask sequence is less than a proportion threshold, determine a target mask according to the first mask sequence, and the mask area of ​​the target mask covers the mask area of ​​the first mask sequence; The first frame sequence and the first mask sequence are respectively cropped according to the target mask to obtain the first target frame sequence and the first target mask sequence.

15. The device according to claim 14, characterized in that The acquisition module is used to: obtain the number of outwardly extended pixels of the mask area of ​​the target mask according to the length and width of the mask area of ​​the target mask; obtain the clipping boundary of the mask area of ​​the target mask according to the length, width and the outwardly extended pixels of the mask area of ​​the target mask; Obtaining a cropping region including the mask region of the target mask according to a cropping boundary of the mask region of the target mask and a length and a width of the mask region of the target mask; According to the cropping area, the first frame sequence and the first mask sequence are cropped respectively to obtain the first target frame sequence and the first target mask sequence.

16. The device according to any one of claims 10 to 11, characterized in that The acquisition module is used to: acquire a first frame sequence of the video to be processed and its corresponding first mask sequence; if the proportion of the mask area in the first mask sequence is greater than or equal to a proportion threshold, then scale the first frame sequence and the first mask sequence respectively to obtain the first target frame sequence and the first target mask sequence.

17. The device according to any one of claims 10 to 11, characterized in that The acquisition module is used to: acquire a complete frame sequence and a corresponding complete mask sequence of the video to be processed; perform lens segmentation processing on the complete frame sequence and the complete mask sequence respectively to obtain a single-lens frame sequence and a corresponding single-lens mask sequence; if the single-lens frame sequence and the corresponding single-lens mask sequence exceed a duration threshold, perform segmentation processing on the single-lens frame sequence and the corresponding single-lens mask sequence to obtain a segmented frame sequence and a corresponding segmented mask sequence, wherein the segmented frame sequence includes a first frame sequence, and the segmented mask sequence includes a first mask sequence; The first target frame sequence and the first target mask sequence are obtained according to the first frame sequence and the first mask sequence corresponding thereto.

18. The device according to claim 17, characterized in that The segmented frame sequence further includes a second frame sequence, and the segmented mask sequence further includes a second mask sequence; Among them, the acquisition module is used to: insert the tail frames of the first repaired sampling frame sequence and the first repaired remaining frame sequence into the second frame sequence and the second mask sequence respectively as the first frames of the second frame sequence and the second mask sequence; process the second frame sequence and the second mask sequence to obtain the second repaired sampling frame sequence and the second repaired remaining frame sequence; merge the first repaired sampling frame sequence and the first repaired remaining frame sequence, and the second repaired sampling frame sequence and the second repaired remaining frame sequence to obtain the repaired video.

19. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the video processing method described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the video processing method according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video processing method according to any one of claims 1 to 9 is implemented.

Images