Method and device for removing foreground target in video stream

Abstract

The embodiment of the invention provides a method and device for removing a foreground target in a video stream, and the method and device are applied to the technical field of computers, and the method comprises the steps: detecting a video frame with the foreground target from an original video stream, and generating a mask corresponding to the foreground target for the video frame with the foreground target; removing and repairing any video frame with the foreground target to obtain a repaired video frame; obtaining a repaired video stream according to each repaired video frame; the removing and repairing operation comprises the following steps: removing the foreground target from the video frame with the foreground target according to the mask to obtain a to-be-repaired video frame; calculating a forward optical flow and a backward optical flow of the video frame with the foreground target, and performing pixel filling on the video frame to be repaired through the forward optical flow and the backward optical flow; and carrying out pixel alignment on the to-be-repaired video frame after pixel filling through the forward optical flow and the backward optical flow to obtain a repaired video frame. According to the invention, the accuracy and visual consistency of the video in a dynamic scene or a multi-foreground target scene are ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method and device for removing a foreground object in a video stream. Background Art

[0002] Foreground object removal is a challenging task in the field of video processing.

[0003] Existing technical solutions are mainly based on the propagation method guided by optical flow, which propagates pixel information from adjacent frames to the current frame by calculating the optical flow between video frames, thereby filling the area blocked by the foreground object. However, since the calculation of optical flow is usually performed at a low resolution, this leads to a decrease in the accuracy of spatial alignment, which in turn affects the propagation effect.

[0004] Therefore, the area after removing the foreground object in the video using the optical flow-guided propagation method may appear blurred, have artifacts, or lack background information. In addition, when dealing with complex scenes such as dynamic scenes or multi-target scenes, the optical flow method is difficult to ensure that the generated background is sufficiently accurate and cannot guarantee the visual consistency of the video. Summary of the invention

[0005] The embodiments of the present application provide a method and device for removing foreground objects in a video stream, which are used to accurately remove subtitle areas in a video and efficiently repair the subtitle background to ensure the visual consistency of the video.

[0006] In a first aspect, an embodiment of the present application provides a method for removing a foreground object in a video stream, comprising:

[0007] Detecting a video frame with a foreground object from an original video stream, and generating a mask corresponding to the foreground object for the video frame with the foreground object;

[0008] Perform a removal and repair operation on any video frame with a foreground object to obtain a repaired video frame:

[0009] A repaired video stream is obtained according to each repaired video frame; wherein the removal repair operation includes:

[0010] The foreground object is removed from the video frame with the foreground object according to the mask to obtain the video frame to be repaired;

[0011] Calculating a forward optical flow and a backward optical flow of the video frame having the foreground object, and performing pixel filling on the video frame to be repaired by using the forward optical flow and the backward optical flow;

[0012] The pixel-filled video frame to be repaired is aligned with the forward optical flow and the backward optical flow to obtain a repaired video frame.

[0013] The present application performs a removal and restoration operation on a video frame with a foreground target through a mask, and uses forward optical flow and backward optical flow to fill pixels of the video frame to be restored, thereby ensuring the accuracy and visual consistency of the video in dynamic scenes or scenes with multiple foreground targets.

[0014] Optionally, the performing pixel alignment on the forward optical flow and the backward optical flow to obtain a repaired video frame includes:

[0015] Performing pixel alignment through the forward optical flow and the backward optical flow to obtain a pixel-aligned video frame to be repaired;

[0016] For the pixel-aligned video frame to be repaired, determining a query window for a self-attention operation by using the mask;

[0017] According to the query window, a self-attention operation is performed on the pixel-aligned video frame to be repaired to obtain a repaired video frame.

[0018] Optionally, performing a self-attention operation on the pixel-aligned to-be-repaired video frame according to the query window to obtain a repaired video frame includes:

[0019] Determining the key value space of the video frame to be repaired after the pixel alignment;

[0020] Determine the different key-value spaces of each attention module when performing self-attention operations;

[0021] Each attention module performs a self-attention operation according to the query window and the respective key value space to obtain a repaired video frame.

[0022] This application introduces a mask-guided sparse video Transformer, which reduces unnecessary calculations in the image processing process and improves processing efficiency by screening key windows in the query space and key / value space.

[0023] Optionally, before performing pixel filling on the to-be-repaired video frame by using the forward optical flow and the backward optical flow, the method further includes:

[0024] Determine the pixel points in the video frame to be repaired that meet the preset conditions as the pixel points for pixel filling; wherein the preset conditions include:

[0025] The optical flow consistency error of the pixel point is less than a preset error threshold; and

[0026] The pixel point is located in the area corresponding to the mask; and

[0027] The pixel point is located outside the area corresponding to the mask in the adjacent frame.

[0028] Optionally, the video frame with the foreground object is the tth video frame in the original video stream;

[0029] The calculating of the forward optical flow and the backward optical flow of the video frame having the foreground object comprises:

[0030] Determine the aligned forward optical flow features according to the t+1th video frame and the video frame to be repaired,

[0031] Determine the aligned backward optical flow features according to the t-1th video frame and the video frame to be repaired,

[0032] The forward optical flow feature and the backward optical flow feature are decoded and reconstructed to obtain the forward optical flow and the backward optical flow.

[0033] In order to complete the optical flow field between video frames and better support the subsequent dual-domain propagation process, the forward optical flow and backward optical flow are estimated through a recurrent neural network, and the discontinuous areas between frames are filled to ensure the integrity and continuity of the optical flow field of the video stream.

[0034] Optionally, the performing pixel alignment by using the forward optical flow and the backward optical flow includes:

[0035] Predicting a first offset and a first modulation mask of the forward optical flow;

[0036] Predicting a second offset and a second modulation mask of the backward optical flow;

[0037] Pixel alignment is performed according to the first offset, the first modulation mask, the second offset, and the second modulation mask.

[0038] Optionally, the pixel alignment by the forward optical flow is specifically performed by the following formula:

[0039]

[0040] The pixel alignment by the backward optical flow is specifically performed by the following formula:

[0041]

[0042] Among them, R represents the fused convolution operation, D represents the deformed convolution operation, Characterize the characteristics of time t, A first offset residual value representing the forward optical flow, Characterizing the aligned forward optical flow features, Characterize the first offset, m t→t+1 Characterize the first modulation mask, f t Characterize optical flow features; a second offset residual value representing the backward optical flow, Characterizing the aligned backward optical flow features, Characterize the second offset, m t-1→t Characterize the second modulation mask.

[0043] This application proposes a dual-domain propagation mechanism, namely, a dual-domain propagation mechanism of global propagation in the image domain and local propagation in the feature domain. The global propagation in the image domain is mainly responsible for filling the large background area after the foreground object is removed, while the local propagation in the feature domain further repairs the details to ensure the integrity and consistency of the background.

[0044] Optionally, before detecting the video frame with the foreground object from the original video stream, the method further includes:

[0045] Determine video segments belonging to different scenes in the original video stream;

[0046] The step of detecting a video frame having a foreground target from an original video stream comprises:

[0047] For each video clip of a scene, video frames with foreground objects are detected from the video clip.

[0048] Optionally, removing the foreground object from the video frame having the foreground object according to the mask to obtain the video frame to be repaired includes:

[0049] Determine a cropping area of ​​a video frame having a foreground object according to the mask; the cropping area is larger than an area corresponding to the mask;

[0050] Removing the foreground object in the cropped area according to the mask to obtain a video frame to be repaired;

[0051] The step of obtaining a repaired video stream according to each repaired video frame comprises:

[0052] Each repaired video frame is spliced ​​with a reserved area of ​​a corresponding video frame to obtain a repaired video stream; the reserved area is an area in the video frame excluding the cropped area.

[0053] When processing video streams, the amount of calculation is reduced by cropping the video frames, and the integrity of the video is restored through precise splicing technology after processing. This technology greatly reduces the computational complexity, improves the processing speed, and ensures the integrity and consistency of the video after splicing. It is suitable for efficient processing of large-scale video data in an environment with limited resources.

[0054] In a second aspect, an embodiment of the present application provides a device for removing a foreground object in a video stream, comprising:

[0055] A generating module, used for detecting a video frame with a foreground target from an original video stream, and generating a mask corresponding to the foreground target for the video frame with the foreground target;

[0056] A repair module is used to perform a removal and repair operation on any video frame with a foreground object to obtain a repaired video frame;

[0057] The repair module is further used to obtain a repaired video stream according to each repaired video frame; wherein the removal repair operation includes:

[0058] The repair module is further used to remove the foreground object from the video frame having the foreground object according to the mask to obtain the video frame to be repaired;

[0059] A filling module, used for calculating the forward optical flow and the backward optical flow of the video frame with the foreground object, and performing pixel filling on the video frame to be repaired by using the forward optical flow and the backward optical flow;

[0060] The alignment module is used to perform pixel alignment on the to-be-repaired video frame after pixel filling through the forward optical flow and the backward optical flow to obtain a repaired video frame.

[0061] The present application performs a removal and restoration operation on a video frame with a foreground target through a mask, and uses forward optical flow and backward optical flow to fill pixels of the video frame to be restored, thereby ensuring the accuracy and visual consistency of the video in dynamic scenes or scenes with multiple foreground targets.

[0062] Optionally, the alignment module is specifically used to:

[0063] Performing pixel alignment through the forward optical flow and the backward optical flow to obtain a pixel-aligned video frame to be repaired;

[0064] For the pixel-aligned video frame to be repaired, determining a query window for a self-attention operation by using the mask;

[0065] According to the query window, a self-attention operation is performed on the pixel-aligned video frame to be repaired to obtain a repaired video frame.

[0066] Optionally, the alignment module is specifically used for:

[0067] Determining the key value space of the video frame to be repaired after the pixel alignment;

[0068] Determine the different key-value spaces of each attention module when performing self-attention operations;

[0069] Each attention module performs a self-attention operation according to the query window and the respective key value space to obtain a repaired video frame.

[0070] Optionally, the filling module is specifically used for:

[0071] Determine the pixel points in the video frame to be repaired that meet the preset conditions as the pixel points for pixel filling; wherein the preset conditions include:

[0072] The optical flow consistency error of the pixel point is less than a preset error threshold; and

[0073] The pixel point is located in the area corresponding to the mask; and

[0074] The pixel point is located outside the area corresponding to the mask in the adjacent frame.

[0075] Optionally, the alignment module is specifically used to:

[0076] The video frame with the foreground object is the tth video frame in the original video stream;

[0077] The calculating of the forward optical flow and the backward optical flow of the video frame having the foreground object comprises:

[0078] Determine the aligned forward optical flow features according to the t+1th video frame and the video frame to be repaired,

[0079] Determine the aligned backward optical flow features according to the t-1th video frame and the video frame to be repaired,

[0080] The forward optical flow feature and the backward optical flow feature are decoded and reconstructed to obtain the forward optical flow and the backward optical flow.

[0081] Optionally, the alignment module is specifically used to:

[0082] Predicting a first offset and a first modulation mask of the forward optical flow;

[0083] Predicting a second offset and a second modulation mask of the backward optical flow;

[0084] Pixel alignment is performed according to the first offset, the first modulation mask, the second offset, and the second modulation mask.

[0085] Optionally, the alignment module is specifically used to:

[0086] The pixel alignment by the forward optical flow is specifically performed by the following formula:

[0087]

[0088] The pixel alignment by the backward optical flow is specifically performed by the following formula:

[0089]

[0090] Among them, R represents the fused convolution operation, D represents the deformed convolution operation, Characterize the characteristics of time t, Characterize the characteristics of time t+1, Characterizing the aligned forward optical flow features, Characterize the first offset residual obtained based on the learning condition, m t→t+1 Characterize the first modulation mask, f t Characterize optical flow features; Characterize the eigenvalue at time t-1, Characterizing the aligned backward optical flow features, Characterize the second offset residual obtained based on the learning condition of the second offset, m t-1→t Characterize the second modulation mask.

[0091] Optionally, the generating module is specifically used for:

[0092] Determine video segments belonging to different scenes in the original video stream;

[0093] The step of detecting a video frame having a foreground target from an original video stream comprises:

[0094] For each video clip of a scene, video frames with foreground objects are detected from the video clip.

[0095] Optionally, the generating module is specifically used for:

[0096] Determine a cropping area of ​​a video frame having a foreground object according to the mask; the cropping area is larger than an area corresponding to the mask;

[0097] Removing the foreground object in the cropped area according to the mask to obtain a video frame to be repaired;

[0098] The step of obtaining a repaired video stream according to each repaired video frame comprises:

[0099] Each repaired video frame is spliced ​​with a reserved area of ​​a corresponding video frame to obtain a repaired video stream; the reserved area is an area in the video frame excluding the cropped area.

[0100] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any method for removing foreground objects in a video stream as described in the first aspect above.

[0101] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, the computer device executes any method for removing foreground objects in a video stream as described in the first aspect above.

[0102] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, it implements the method for removing a foreground target in a video stream as described in any of the first aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0104] Figure 1 A schematic diagram of a flow chart of a method for removing a foreground object in a video stream provided in an embodiment of the present application;

[0105] Figure 2 A schematic diagram of a video frame provided in an embodiment of the present application;

[0106] Figure 3 A schematic diagram of a video frame provided in an embodiment of the present application;

[0107] Figure 4 A schematic diagram of a video frame provided in an embodiment of the present application;

[0108] Figure 5 A schematic diagram of a video frame provided in an embodiment of the present application;

[0109] Figure 6 A schematic diagram of a flow chart of a method for removing a foreground object in a video stream provided in an embodiment of the present application;

[0110] Figure 7 A schematic diagram of a device for removing a foreground object in a video stream provided in an embodiment of the present application;

[0111] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0112] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0113] The following are explanations of the terms used in this application:

[0114] PaddleOCR: full name Paddle Optical Character Recognition, is a powerful open source OCR (Optical Character Recognition) tool developed based on the deep learning framework PaddlePaddle.

[0115] OCR-SAM: is an open source OCR system that uses deep learning models such as Tesseract and EAST for text detection and recognition.

[0116] like Figure 1 FIG. 1 is a flow chart of a method for removing a foreground object in a video stream provided by an embodiment of the present application, which specifically includes the following steps:

[0117] Step S101 , detecting video frames with foreground objects from an original video stream, and generating masks corresponding to the foreground objects for the video frames with foreground objects.

[0118] Specifically, the video frames in the original video stream are detected frame by frame through the target detection algorithm to detect the video frames with foreground targets. Among them, the foreground target refers to an object or person that is in the front position, more prominent and usually the focus of attention in the video frame. And a mask of corresponding size is generated according to the foreground target, so that the mask can cover the foreground target in the video frame. Among them, the target detection algorithm can be PaddelOCR (Paddle Optical Character Recognition) or OCRSAM algorithm based on deep learning.

[0119] If the subtitles appearing in the video stream are taken as foreground targets, the PaddelOCR target detection algorithm is used to detect the video frames in the original video stream frame by frame to detect the video frames with subtitles, such as Figure 2 The video frame shown in FIG. 1 is a video frame in which the foreground object is a subtitle, and a mask of a corresponding size is generated according to the subtitle area of ​​each video frame, so that the mask can completely cover the subtitle appearing on the video frame, such as Figure 3The video frame shown. Advanced target detection algorithms (such as PaddelOCR or OCRSAM) are used to detect foreground targets in the video frame, and combined with edge optimization processing to generate high-precision masks. Accurate foreground target detection and mask optimization ensure the accuracy of subsequent processing and reduce the risk of false detection or missed detection.

[0120] Step S102, performing a removal and repair operation on any video frame with a foreground object to obtain a repaired video frame;

[0121] Step S103, obtaining a repaired video stream according to each repaired video frame;

[0122] Step S104, wherein the removal and repair operation includes: removing the foreground object from the video frame having the foreground object according to the mask to obtain the video frame to be repaired;

[0123] Step S105, calculating the forward optical flow and the backward optical flow of the video frame with the foreground object, and performing pixel filling on the video frame to be repaired by using the forward optical flow and the backward optical flow;

[0124] Step S106, performing pixel alignment on the to-be-repaired video frame after pixel filling through forward optical flow and backward optical flow to obtain a repaired video frame.

[0125] The present application performs a removal and restoration operation on a video frame with a foreground target through a mask, and uses forward optical flow and backward optical flow to fill pixels of the video frame to be restored, thereby ensuring the accuracy and visual consistency of the video in dynamic scenes or scenes with multiple foreground targets.

[0126] In some embodiments, in order to improve the efficiency and accuracy of video stream processing, before performing step S101 to detect video frames with foreground targets from the original video stream, video segments in the original video stream belonging to different scenes are determined; for the video segments of each scene, video frames with foreground targets are detected from the video segments.

[0127] Specifically, the SceneDetect algorithm can be used to perform scene detection and division on the video first. SceneDetect is a content-based scene detection technology that can identify natural scene changes in a video and divide the video stream into multiple continuous scene segments. Specifically, the content of the video stream is analyzed by the SceneDetect algorithm, different scene change points in the video stream are identified, and the video stream is divided into multiple sub-video segments according to the scene changes. This step can decompose the video stream so that each sub-video segment is processed under a relatively consistent background, thereby improving the accuracy of the processing. According to the detection results, the original video stream is divided into several sub-video streams, each of which corresponds to a continuous scene. These sub-video streams will be used as the original video stream in the above-mentioned step S101, and the subsequent steps S102 to S106 will be performed.

[0128] By introducing scene detection, the present invention can perform targeted processing on different scene segments, further improving processing efficiency and repair effect, and avoiding frame skipping or incoherence that may occur at scene switching.

[0129] In some embodiments, in order to further improve the processing speed of video frames with foreground objects and reduce the amount of calculation, before performing the removal and repair operation on any video frame with foreground objects, the area to be cropped can be determined according to the mask information generated by the object detection module, and the video frame and the corresponding mask can be cropped. The size of the cropped video frame is smaller, which reduces the computational complexity of subsequent processing. When processing a video stream, by cropping a video frame, such as Figure 4 and Figure 5 The video frames shown in the figure reduce the amount of calculation and restore the integrity of the video through precise splicing technology after processing. This technology greatly reduces the computational complexity and improves the processing speed, while ensuring the integrity and consistency of the video after splicing. It is suitable for efficient processing of large-scale video data in an environment with limited resources.

[0130] In some embodiments, the generated mask may be subjected to optimization processing such as edge smoothing to ensure that the boundary of the mask is smooth and accurate in subsequent processing to avoid artifacts or other adverse effects.

[0131] In order to complete the optical flow field between video frames and better support the subsequent dual-domain propagation process, the forward optical flow and backward optical flow are estimated through a recurrent neural network, and the discontinuous areas between frames are filled to ensure the integrity and continuity of the optical flow field of the video stream.

[0132] Optionally, the video frame with the foreground target is the tth video frame in the original video stream; the forward optical flow and the backward optical flow of the video frame with the foreground target are calculated in the following manner: the aligned forward optical flow features are determined according to the t+1th video frame and the video frame to be repaired, the aligned backward optical flow features are determined according to the t-1th video frame and the video frame to be repaired, the forward optical flow features and the backward optical flow features are decoded and reconstructed to obtain the forward optical flow and the backward optical flow.

[0133] Specifically, the optical flow between adjacent frames is calculated through the optical flow estimation network to obtain the forward optical flow and the backward optical flow. If the video frame with the foreground target is the tth video frame X in the original video stream, t For example, the forward optical flow refers to the current frame X t To the next frame X t+1 The pixel motion of the current frame X t Return to previous frame X t-1 The pixel movement. Then the solution formula for the forward optical flow is as follows:

[0134] F t→t+1 =RAFT(X t ,X t+1 )

[0135] ...Formula 1

[0136] Among them, X t and X t+1 represents the frames at time t and t+1, F t→t+1 It is the forward optical flow obtained from a pre-trained model RAFT.

[0137] The solution formula for backward optical flow is shown in Formula 2 below:

[0138] F t→t-1 =RAFT(X t ,X t-1 )

[0139] ...Formula 2 In which, X t and X t-1 represents the frames at time t and t-1, F t→t-1 is the backward optical flow obtained from a pre-trained model RAFT. It should be noted that Formula 2 can be transformed into the following formula:

[0140] F t+1→t =RAFT(X t+1 ,X t )

[0141] In order to improve the computational efficiency, the forward optical flow and the backward optical flow are downsampled, the forward optical flow and the backward optical flow are reduced by 8 times respectively, and the current frame is encoded to obtain the encoding feature f t , in this t It represents both the downsampled forward optical flow encoding features and the downsampled backward optical flow encoding features.

[0142] In order to propagate optical flow information on the time axis and repair the missing parts, an alignment strategy based on deformable convolution (DCN) is adopted. The convolution operation is dynamically adjusted by predicting the offset and modulation mask of the convolution, so that the motion information between frames can be accurately aligned. Pixel alignment is performed through forward optical flow and backward optical flow, including: predicting the first offset and the first modulation mask of the forward optical flow; predicting the second offset and the second modulation mask of the backward optical flow; and performing pixel alignment according to the first offset, the first modulation mask, the second offset and the second modulation mask.

[0143] Specifically, the method of aligning the forward optical flow is to encode the feature f of the tth frame t and the encoding feature f of the t+1th frame t+1 Splice to generate joint feature c1(f t , f t+1 ), and then input the joint features into a lightweight convolutional network to predict the first offset o t→t+1 and the first modulation mask m t→t+1 ; The same method is used to align the backward optical flow by encoding the feature f of the tth frame t and the encoding feature f of the t+1th frame t+1 Splice to generate joint feature c2(f t , f t-1 ), and then input the joint features into a lightweight convolutional network to predict the second offset o t→t-1 and the second modulation mask m t→t-1 .

[0144] The aligned forward optical flow feature is calculated by the following formula 3

[0145]

[0146] Specifically, through the deformed convolutional network D, and using the first offset o t→t+1 and the first modulation mask m t→t+1 The encoding feature f of the next frame t+1 Align so that the encoding feature f of the next frame t+1 Then, the fusion convolution network R is used to fuse the aligned features and the current frame features f t, get the aligned forward optical flow features

[0147] The aligned backward optical flow feature is calculated by the following formula 4

[0148]

[0149] Specifically, through the deformed convolutional network D, and using the second offset o t→t-1 and the second modulation mask m t→t-1 The encoding feature f of the previous frame t-1 Align so that the encoding feature f of the previous frame t-1 Then, the fusion convolution network R is used to fuse the aligned features and the current frame features f t , get the aligned backward optical flow features

[0150] After obtaining the aligned forward optical flow features And the aligned backward optical flow features Afterwards, the decoder is used to reconstruct the two features separately to obtain the complete forward optical flow and the complete backward optical flow

[0151]

[0152] Optionally, before performing pixel filling on the video frame to be repaired by using the forward optical flow and the backward optical flow, it is also necessary to determine the area in the video frame to be repaired where pixel filling is required, that is, it is necessary to determine the pixel points in the video frame to be repaired that meet the preset conditions as the pixel points p for pixel filling; wherein the preset conditions include the following three conditions C1, C2 and C3:

[0153] C1 condition: The optical flow consistency error of the pixel point is less than the preset error threshold.

[0154] Specifically, only when the consistency error of the forward and backward optical flow of the pixel is less than the preset error threshold ε, the motion of the pixel is considered reliable, which can avoid motion propagation error estimation. For each pixel p, the consistency error of the forward optical flow is defined as shown in the following formula 5:

[0155]

[0156] in, is the forward optical flow of the pixel, is the backward optical flow of the pixel point, and only the consistency error e of the forward and backward optical flow of the pixel point p t→t+1 When (p) is less than the preset error threshold ε, the pixel point p meets the C1 condition.

[0157] C2 condition: The pixel is located in the area corresponding to the mask.

[0158] Specifically, only when the pixel point is a pixel point in the foreground target mask area in the current frame t, it will be considered for processing. It is specifically expressed as the following formula 6:

[0159] M t (p) = 1...Formula 6

[0160] C3 condition: The pixel is outside the area corresponding to the mask in the adjacent frame.

[0161] Specifically, only when the corresponding position in the adjacent frame t+1 is not a subtitle mask area, that is, an area that is not marked to be filled, is it used as the pixel point p for pixel filling.

[0162] Only when the three conditions C1, C2 and C3 are met at the same time, it is considered a reliable propagation area A. r , as follows formula seven:

[0163]

[0164] After determining the reliable propagation area A r After that, the aligned forward optical flow is used to reliably propagate area A r Filling is performed as follows:

[0165]

[0166] Among them, A r is the reliable propagation area, X t is the image frame at time t, X t-1 is the image frame at time t-1, W is a computational operation that converts the image frame X at time t-1 into t-1 Through the aligned forward optical flow Mapping back to the image frame X at time t t ; After calculating the pixels at the mask of the current frame, the masked pixel state is promptly converted to the unmasked state, thus obtaining a partially filled video sequence. Similarly, the aligned backward optical flow is used to identify the reliable propagation area A. r Fill the image frame X at time t+1 t+1 Through the aligned backward optical flow Mapping back to the image frame X at time t t , as shown in Formula 9:

[0167]

[0168] The above is a global propagation of pixel information between adjacent frames in the image domain through optical flow guidance, which can quickly fill the video area after the foreground object is removed and restore the overall structure of the background.

[0169] However, after propagation in the image domain, some areas in the video are still not completely filled, and it is difficult for image domain propagation to handle the problem of detail misalignment. Therefore, it is also necessary to use the refined optical flow to propagate the video frame locally in the feature domain, that is, to propagate the video frame globally in the feature domain. This is similar to the alignment operation of recursive flow filling, but additionally introduces masks and validity checks.

[0170] Therefore, in the global propagation of the feature domain of the video frame, the features of the previous and next frames can be aligned through the deformable convolution (DCN). Specifically, the pixel features at time t are aligned through the forward optical flow. The alignment is specifically performed through the following formula:

[0171]

[0172] Through the backward optical flow, the pixel features at time t The alignment is specifically performed by the following formula 11:

[0173]

[0174] Among them, R represents the fused convolution operation, D represents the deformed convolution operation, Characterize the characteristics of time t, Characterize the characteristics of time t+1, Characterize the aligned forward optical flow features, Characterize the first offset residual obtained based on the learning condition, m t→t+1 Characterize the first modulation mask, f t Characterize optical flow features; Characterize the eigenvalue at time t-1, Characterize the aligned backward optical flow features, Characterize the second offset residual obtained based on the learning condition, m t-1→t Characterizes the second modulation mask. and The offset residuals are learned based on the conditions (mask, C1, C2, C3) and are predicted by deformable convolution (DCN). Decode and get the restored picture.

[0175] This application proposes a dual-domain propagation mechanism, namely, a dual-domain propagation mechanism of global propagation in the image domain and local propagation in the feature domain. The global propagation in the image domain is mainly responsible for filling the large background area after the foreground object is removed, while the local propagation in the feature domain further repairs the details to ensure the integrity and consistency of the background.

[0176] In the above step S106, pixel alignment is performed through the forward optical flow and the backward optical flow to obtain the pixel-aligned video frame to be repaired; for the pixel-aligned video frame to be repaired, the query window of the self-attention operation is determined by the mask; according to the query window, the self-attention operation is performed on the pixel-aligned video frame to be repaired to obtain the repaired video frame.

[0177] Specifically, whether filling pixels in the image domain or the feature domain, the process is to perform convolution operations on the image frame. The deep learning architecture transformer used in this process processes the entire image frame, which introduces a lot of unnecessary calculations in the process of aligning all pixels in the entire image frame. Therefore, the query window of the self-attention operation is determined by the mask; according to the query window, the self-attention operation is performed on the video frame to be repaired after pixel alignment, and the query window of the processed image data is sparse. The self-attention operation is performed only on the query window containing the mask area, skipping the non-subtitle area, thereby reducing the amount of calculation.

[0178] Therefore, according to the query window, a self-attention operation is performed on the video frame to be repaired after pixel alignment to obtain a repaired video frame, including: determining the key value space of the video frame to be repaired after pixel alignment; determining the different key value spaces of each attention module when performing the self-attention operation; each attention module performs a self-attention operation according to the query window and its own key value space to obtain a repaired video frame.

[0179] First, the query space is sparse, that is, by downsampling and summarizing the mask generated by the subtitle recognition network, the area that needs to be paid attention to in the spatiotemporal domain is screened out to reduce unnecessary calculations. Assuming that the mask is M, M is downsampled to obtain M′, and the query window containing the subtitle pixels is determined by summing in the time dimension to obtain the area that needs to be paid attention to, that is, the sparse mask and the sparse mask is recorded as SQ. Specifically, the sparse query space operation is performed through the following formula 12, where Clip is the cropping operation and t is the t time in the video stream:

[0180]

[0181] As long as a query window contains a masked area in any frame, the window will be marked as needing to participate in the attention calculation, otherwise it will be ignored.

[0182] Since there is usually a strong similarity between video frames, especially in temporally adjacent video frames, a lot of content is repeated, so only specific frames are selected within a longer time interval to construct the key / value space. In the key / value space, the time step method is used to reduce the amount of key / value calculations. For example, there are multiple self-attention modules (Transformer blocks), each of which has a number, and a time step can be set for each self-attention module (Transformer block) in the form of alternating odd and even frames. For example, a sparse strategy with a time step of 2 can be set, that is, in each odd-numbered Transformer block, odd frames are selected as keys / values, and in even-numbered Transformer blocks, only even frames are selected, so that the amount of video frame processing can be reduced.

[0183] Based on the sparse query and sparse key / value space designed above, the attention calculation is performed. The input features are divided into m×n non-overlapping windows with a size of h×w by the sparse strategy. Through the linear layer, the divided window features are projected into query Q, key K and value V respectively. The sparse attention calculation is performed using the generated sparse mask SQ and the key / value space controlled by the time step. Finally, the repaired features are obtained, and the decoder is used to reconstruct the repaired features to obtain the repaired video frame.

[0184] This application introduces a mask-guided sparse video Transformer, which reduces unnecessary calculations in the image processing process and improves processing efficiency by screening key windows in the query space and key / value space.

[0185] Optionally, a cropped area of ​​a video frame having a foreground target is determined according to a mask; the cropped area is larger than an area corresponding to the mask; the foreground target in the cropped area is removed according to the mask to obtain a video frame to be repaired; each repaired video frame is spliced ​​with a reserved area of ​​a corresponding video frame to obtain a repaired video stream; the reserved area is an area in the video frame excluding the cropped area.

[0186] Specifically, after the video frames are filled and repaired as described above, each repaired video frame is obtained, and each repaired video frame needs to be re-stitched into the original video to generate a complete output video. For cropped frames, each repaired video frame is re-covered back to the position of the cropped area of ​​the original video frame, that is, each repaired video frame is spliced ​​with the reserved area of ​​the corresponding video frame to ensure the continuity and consistency of the video. Video output: All processed frames and scene segmented fragments are spliced ​​in chronological order to generate a complete video output. The above operations ensure the consistency of the video stream in the time sequence, and there is no frame skipping or content incoherence.

[0187] When processing video streams, the amount of calculation is reduced by cropping the video frames, and the integrity of the video is restored through precise splicing technology after processing. This technology greatly reduces the computational complexity, improves the processing speed, and ensures the integrity and consistency of the video after splicing. It is suitable for efficient processing of large-scale video data in an environment with limited resources.

[0188] like Figure 6 A method for removing a foreground object in a video stream is shown, and the specific process is as follows:

[0189] Detect and crop the video scene of the acquired video stream; you can first use the SceneDetect algorithm to detect and divide the video scene. SceneDetect is a content-based scene detection technology that can identify natural scene changes in the video and divide the video stream into multiple continuous scene segments. Specifically, the SceneDetect algorithm analyzes the content of the video stream, identifies different scene change points in the video stream, and divides the video stream into multiple sub-video segments according to the scene changes. This step can decompose the video stream so that each sub-video segment is processed under a relatively consistent background, thereby improving the accuracy of the processing. In order to continue to improve the processing speed of video frames with foreground targets and reduce the amount of calculation, before performing the removal and repair operation on any video frame with a foreground target, the area to be cropped can be determined based on the mask information generated by the target detection module, and the video frame and the corresponding mask can be cropped.

[0190] The foreground target detection module generates a mask and a background area. Specifically, the target detection algorithm can be used to detect the video frames in the original video stream frame by frame, detect the video frames with foreground targets, and generate a mask of corresponding size according to the subtitle area of ​​each video frame. The mask area and the background area of ​​a video frame are distinguished according to the mask.

[0191] Recursive flow filling is performed on the video frame to be repaired obtained by removing the masked background area of ​​the video frame, and the aligned forward optical flow features are determined according to the t+1th video frame and the video frame to be repaired. The aligned backward optical flow features are determined according to the t-1th video frame and the video frame to be repaired. The forward optical flow features and the backward optical flow features are decoded and reconstructed to obtain the forward optical flow and the backward optical flow, and the pixel of the video frame to be repaired is filled by the forward optical flow and the backward optical flow.

[0192] First, the forward optical flow and backward optical flow are used to perform global propagation in the image domain on the repaired video frame; then the forward optical flow and backward optical flow are used to perform local propagation in the feature domain on the repaired video frame to obtain the restored video frame.

[0193] A self-attention operation is performed on the video frame to be repaired after pixel alignment, that is, mask-guided sparseness is performed on the video frame to be repaired after pixel alignment. The specific steps are: determining the key value space of the video frame to be repaired after pixel alignment; determining the different key value spaces of each attention module when performing the self-attention operation; each attention module performs a self-attention operation according to the query window and its own key value space to obtain a repaired video frame.

[0194] The video frames after filling and repair processing are stitched together as well as single-frame image stitching. After the video frames are filled and repaired as described above, each repaired video frame is obtained. Each repaired video frame needs to be re-stitched into the original video to generate a complete output video. For the cropped frames, each repaired video frame is re-covered back to the position of the cropped area of ​​the original video frame, that is, each repaired video frame is spliced ​​with the reserved area of ​​the corresponding video frame to ensure the continuity and consistency of the video. Video output: All processed frames and scene segmented fragments are spliced ​​in chronological order to generate a complete video output.

[0195] like Figure 7 As shown, an embodiment of the present application provides a device 700 for removing a foreground object in a video stream, comprising:

[0196] A generating module 701 is used to detect a video frame with a foreground object from an original video stream, and generate a mask corresponding to the foreground object for the video frame with the foreground object;

[0197] A repair module 702 is used to perform a removal and repair operation on any video frame with a foreground object to obtain a repaired video frame;

[0198] The repair module 702 is further configured to obtain a repaired video stream according to each repaired video frame; wherein the removal repair operation includes:

[0199] The repair module 702 is further used to remove the foreground object from the video frame having the foreground object according to the mask to obtain the video frame to be repaired;

[0200] A filling module 703 is used to calculate the forward optical flow and the backward optical flow of the video frame with the foreground object, and fill the pixels of the video frame to be repaired by using the forward optical flow and the backward optical flow;

[0201] The alignment module 704 is used to perform pixel alignment on the to-be-repaired video frame after pixel filling through the forward optical flow and the backward optical flow to obtain a repaired video frame.

[0202] The present application performs a removal and restoration operation on a video frame with a foreground target through a mask, and uses forward optical flow and backward optical flow to fill pixels of the video frame to be restored, thereby ensuring the accuracy and visual consistency of the video in dynamic scenes or scenes with multiple foreground targets.

[0203] Optionally, the alignment module 704 is specifically configured to:

[0204] Performing pixel alignment through the forward optical flow and the backward optical flow to obtain a pixel-aligned video frame to be repaired;

[0205] For the pixel-aligned video frame to be repaired, determining a query window for a self-attention operation by using the mask;

[0206] According to the query window, a self-attention operation is performed on the pixel-aligned video frame to be repaired to obtain a repaired video frame.

[0207] Optionally, the alignment module 704 is specifically configured to:

[0208] Determining the key value space of the video frame to be repaired after the pixel alignment;

[0209] Determine the different key-value spaces of each attention module when performing self-attention operations;

[0210] Each attention module performs a self-attention operation according to the query window and the respective key value space to obtain a repaired video frame.

[0211] Optionally, the filling module 703 is specifically used for:

[0212] Determine the pixel points in the video frame to be repaired that meet the preset conditions as the pixel points for pixel filling; wherein the preset conditions include:

[0213] The optical flow consistency error of the pixel point is less than a preset error threshold; and

[0214] The pixel point is located in the area corresponding to the mask; and

[0215] The pixel point is located outside the area corresponding to the mask in the adjacent frame.

[0216] Optionally, the alignment module 703 is specifically used to:

[0217] The video frame with the foreground object is the tth video frame in the original video stream;

[0218] The calculating of the forward optical flow and the backward optical flow of the video frame having the foreground object comprises:

[0219] Determine the aligned forward optical flow features according to the t+1th video frame and the video frame to be repaired,

[0220] Determine the aligned backward optical flow features according to the t-1th video frame and the video frame to be repaired,

[0221] The forward optical flow feature and the backward optical flow feature are decoded and reconstructed to obtain the forward optical flow and the backward optical flow.

[0222] Optionally, the alignment module 703 is specifically used to:

[0223] Predicting a first offset and a first modulation mask of the forward optical flow;

[0224] Predicting a second offset and a second modulation mask of the backward optical flow;

[0225] Pixel alignment is performed according to the first offset, the first modulation mask, the second offset, and the second modulation mask.

[0226] Optionally, the alignment module 703 is specifically used to:

[0227] The pixel alignment by the forward optical flow is specifically performed by the following formula:

[0228]

[0229] The pixel alignment by the backward optical flow is specifically performed by the following formula:

[0230]

[0231] Among them, R represents the fused convolution operation, D represents the deformed convolution operation, Characterize the characteristics of time t, Characterize the characteristics of time t+1, Characterizing the aligned forward optical flow features, Characterize the first offset residual obtained based on the learning condition, m t→t+1 Characterize the first modulation mask, f t Characterize optical flow features; Characterize the eigenvalue at time t-1, Characterizing the aligned backward optical flow features, Characterize the second offset residual obtained based on the learning condition of the second offset, m t-1→t Characterize the second modulation mask.

[0232] Optionally, the generating module 701 is specifically used for:

[0233] Determine video segments belonging to different scenes in the original video stream;

[0234] The step of detecting a video frame having a foreground target from an original video stream comprises:

[0235] For each video clip of a scene, video frames with foreground objects are detected from the video clip.

[0236] Optionally, the generating module 701 is specifically used for:

[0237] Determine a cropping area of ​​a video frame having a foreground object according to the mask; the cropping area is larger than an area corresponding to the mask;

[0238] Removing the foreground object in the cropped area according to the mask to obtain a video frame to be repaired;

[0239] The step of obtaining a repaired video stream according to each repaired video frame comprises:

[0240] Each repaired video frame is spliced ​​with a reserved area of ​​a corresponding video frame to obtain a repaired video stream; the reserved area is an area in the video frame excluding the cropped area.

[0241] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 8 The terminal device 801 and / or the server 802 shown in FIG. Figure 8 As shown, it includes at least one processor 801 and a memory 802 connected to the at least one processor. The specific connection medium between the processor 801 and the memory 802 is not limited in the embodiment of the present application. Figure 8 For example, the processor 801 and the memory 802 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0242] In the embodiment of the present application, the memory 802 stores instructions that can be executed by at least one processor 801. The at least one processor 801 can perform the steps of the method for removing the foreground object in the video stream by executing the instructions stored in the memory 802.

[0243] Among them, the processor 801 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and by running or executing instructions stored in the memory 802 and calling data stored in the memory 802, it is used to accurately remove the subtitle area in the video and efficiently repair the subtitle background to ensure the visual consistency of the video. Optionally, the processor 801 may include one or more processing units, and the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801. In some embodiments, the processor 801 and the memory 802 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0244] Processor 801 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0245] The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 802 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 802 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0246] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the method for removing foreground objects in the video stream.

[0247] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0248] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0249] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0250] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0251] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for removing a foreground object in a video stream, characterized in that: include: Detecting a video frame with a foreground object from an original video stream, and generating a mask corresponding to the foreground object for the video frame with the foreground object; Perform a removal and repair operation on any video frame with a foreground object to obtain a repaired video frame; A repaired video stream is obtained according to each repaired video frame; wherein the removal repair operation includes: The foreground object is removed from the video frame with the foreground object according to the mask to obtain the video frame to be repaired; Calculating a forward optical flow and a backward optical flow of the video frame having the foreground object, and performing pixel filling on the video frame to be repaired by using the forward optical flow and the backward optical flow; The pixel-filled video frame to be repaired is aligned with the forward optical flow and the backward optical flow to obtain a repaired video frame.

2. The method according to claim 1, characterized in that The step of performing pixel alignment by the forward optical flow and the backward optical flow to obtain a repaired video frame includes: Performing pixel alignment through the forward optical flow and the backward optical flow to obtain a pixel-aligned video frame to be repaired; For the pixel-aligned video frame to be repaired, determining a query window for a self-attention operation by using the mask; According to the query window, a self-attention operation is performed on the pixel-aligned video frame to be repaired to obtain a repaired video frame.

3. The method according to claim 2, characterized in that The step of performing a self-attention operation on the pixel-aligned video frame to be repaired according to the query window to obtain a repaired video frame includes: Determining the key value space of the video frame to be repaired after the pixel alignment; Determine the different key-value spaces of each attention module when performing self-attention operations; Each attention module performs a self-attention operation according to the query window and the respective key value space to obtain a repaired video frame.

4. The method according to any one of claims 1 to 3, characterized in that Before performing pixel filling on the to-be-repaired video frame by using the forward optical flow and the backward optical flow, the method further includes: Determine the pixel points in the video frame to be repaired that meet the preset conditions as the pixel points for pixel filling; wherein the preset conditions include: The optical flow consistency error of the pixel point is less than a preset error threshold; and The pixel point is located in the area corresponding to the mask; and The pixel point is located outside the area corresponding to the mask in the adjacent frame.

5. The method according to claim 4, characterized in that The video frame with the foreground object is the tth video frame in the original video stream; The calculating of the forward optical flow and the backward optical flow of the video frame having the foreground object comprises: Determine the aligned forward optical flow features according to the t+1th video frame and the video frame to be repaired, Determine the aligned backward optical flow features according to the t-1th video frame and the video frame to be repaired, The forward optical flow feature and the backward optical flow feature are decoded and reconstructed to obtain the forward optical flow and the backward optical flow.

6. The method according to claim 4, characterized in that The performing pixel alignment by using the forward optical flow and the backward optical flow comprises: Predicting a first offset and a first modulation mask of the forward optical flow; Predicting a second offset and a second modulation mask of the backward optical flow; Pixel alignment is performed according to the first offset, the first modulation mask, the second offset, and the second modulation mask.

7. The method according to claim 6, characterized in that The pixel alignment by the forward optical flow is specifically performed by the following formula: The pixel alignment by the backward optical flow is specifically performed by the following formula: Among them, R represents the fused convolution operation, D represents the deformed convolution operation, Characterize the characteristics of time t, Characterize the characteristics of time t+1, Characterizing the aligned forward optical flow features, Characterize the first offset residual obtained based on the learning condition, m t→t+1 Characterize the first modulation mask, f t Characterize optical flow features; Characterize the eigenvalue at time t-1, Characterizing the aligned backward optical flow features, Characterize the second offset residual obtained based on the learning condition of the second offset, m t-1→t Characterize the second modulation mask.

8. The method according to any one of claims 1 to 3, characterized in that Before detecting the video frame with the foreground target from the original video stream, the method further includes: Determine video segments belonging to different scenes in the original video stream; The step of detecting a video frame having a foreground target from an original video stream comprises: For each video clip of a scene, video frames with foreground objects are detected from the video clip.

9. The method according to claim 8, characterized in that The step of removing the foreground object from the video frame having the foreground object according to the mask to obtain the video frame to be repaired includes: Determine a cropping area of ​​a video frame having a foreground object according to the mask; the cropping area is larger than an area corresponding to the mask; Removing the foreground object in the cropped area according to the mask to obtain a video frame to be repaired; The step of obtaining a repaired video stream according to each repaired video frame comprises: Each repaired video frame is spliced ​​with a reserved area of ​​a corresponding video frame to obtain a repaired video stream; the reserved area is an area in the video frame excluding the cropped area.

10. A device for removing a foreground object in a video stream, characterized in that: include: A generating module, used for detecting a video frame with a foreground target from an original video stream, and generating a mask corresponding to the foreground target for the video frame with the foreground target; A repair module is used to remove and repair any video frame with a foreground object to obtain a repaired video frame; The repair module is further used to obtain a repaired video stream according to each repaired video frame; wherein the removal repair operation includes: The repair module is further used to remove the foreground object from the video frame having the foreground object according to the mask to obtain the video frame to be repaired; A filling module, used for calculating the forward optical flow and the backward optical flow of the video frame with the foreground object, and performing pixel filling on the video frame to be repaired by using the forward optical flow and the backward optical flow; The alignment module is used to perform pixel alignment on the to-be-repaired video frame after pixel filling through the forward optical flow and the backward optical flow to obtain a repaired video frame.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of any one of the methods of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of any method described in claims 1 to 9 are implemented.

Images