Video processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a video processing method and device, a storage medium and a program product. The method comprises the steps of obtaining optical flow information of a to-be-processed video; obtaining a first intermediate video frame with a target object removed and an initial mask image from any initial video frame of the to-be-processed video; determining a pixel value of at least one target pixel in a to-be-repaired area of the first intermediate video frame according to the optical flow information and an adjacent video frame of the initial video frame, and filling the target pixel of the first intermediate video frame and a corresponding pixel of the initial mask image to obtain a second intermediate video frame and an intermediate mask image; and according to the optical flow information, the second intermediate video frame and the intermediate mask image, calling a preset video restoration model to restore the second intermediate video frame. The optical flow information and the adjacent video frames are combined for preliminary restoration, richer original textures are reserved, the area of the area needing restoration is reduced, then the video restoration model is adopted for restoration, and the restoration efficiency and the erasing quality of the target object are improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular, to a video processing method, device, storage medium, and program product. Background Art

[0002] During the video editing and processing process, users may need to erase subtitles, advertisements, characters, icons, and other content in the video.

[0003] However, the erasing quality of some video processing methods in the prior art is not high, especially for some complex scenes, which are prone to omissions, scratches, blurring, etc., and require manual frame-by-frame repair, which is costly. Or directly projecting without manual repair will affect the user's viewing experience. Summary of the invention

[0004] The embodiments of the present disclosure provide a video processing method, device, storage medium and program product to improve the erasing quality of target objects in a video.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0006] Obtain the optical flow information of the video to be processed;

[0007] Determine the position of the target object in any initial video frame of the video to be processed, and segment the target object on the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired in the first intermediate video frame;

[0008] Determine, according to the optical flow information and adjacent video frames of the initial video frame, a pixel value of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame, and fill the target pixel of the first intermediate video frame and corresponding pixels of the initial mask image according to the pixel value of the target pixel, to obtain a second intermediate video frame and an intermediate mask image;

[0009] According to the optical flow information, the second intermediate video frame and the intermediate mask image, a preset video restoration model is called to restore the second intermediate video frame to obtain a target video frame.

[0010] In a second aspect, an embodiment of the present disclosure provides a video processing device, including:

[0011] An optical flow acquisition unit, used to acquire optical flow information of the video to be processed;

[0012] A target object identification unit is used to determine the position of the target object in any initial video frame of the video to be processed, and to segment the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired in the first intermediate video frame;

[0013] A first restoration unit is used to determine a pixel value of at least one target pixel in a to-be-restored area of ​​the first intermediate video frame according to the optical flow information and adjacent video frames of the initial video frame, and fill the target pixel of the first intermediate video frame and corresponding pixels of the initial mask image according to the pixel value of the target pixel, so as to obtain a second intermediate video frame and an intermediate mask image;

[0014] The second restoration unit is used to call a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image to obtain a target video frame.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video processing method as described in the first aspect and various possible designs of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the video processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the video processing method as described in the first aspect and various possible designs of the first aspect.

[0020] The video processing method, device, storage medium and program product provided by the embodiment of the present disclosure obtain the optical flow information of the video to be processed; determine the position of the target object in any initial video frame of the video to be processed, and segment the target object of the initial video frame according to the position of the target object to obtain the first intermediate video frame and the initial mask image from which the target object is removed; wherein the target object is in the non-mask area in the initial mask image, which is used to identify the area to be repaired of the first intermediate video frame; determine the pixel value of at least one target pixel in the area to be repaired of the first intermediate video frame according to the optical flow information and the adjacent video frames of the initial video frame, and fill the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel, to obtain the second intermediate video frame and the intermediate mask image; according to the optical flow information, the second intermediate video frame and the intermediate mask image, call the preset video repair model to repair the second intermediate video frame to obtain the target video frame. In the embodiment of the present disclosure, the optical flow information and the adjacent video frames can be combined for preliminary repair, so as to retain richer original textures and reduce the area of ​​the area to be repaired, and then the video repair model is used for repair, so as to improve the repair efficiency and the erasing quality of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 A schematic diagram of a scene of a video processing method provided by an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of a flow chart of a video processing method provided by another embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a flow chart of a video processing method provided by another embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of a flow chart of a video processing method provided by another embodiment of the present disclosure;

[0027] Figure 6 A structural block diagram of a video processing device provided by an embodiment of the present disclosure;

[0028] Figure 7A schematic diagram of the hardware structure of a video processing device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] During video editing and processing, users have the need to erase subtitles, advertisements, characters, icons, etc. However, some existing video processing methods have low erasure quality, especially for some complex scenes, which are prone to omission, scratches, blurring, etc., and require manual frame-by-frame repair, which is costly. Or, the video is directly shown without manual repair, which affects the user's viewing experience.

[0031] In order to solve the above technical problems, the embodiment of the present disclosure provides a video processing method, which obtains the optical flow information of the video to be processed; determines the position of the target object in any initial video frame of the video to be processed, and performs target object segmentation on the initial video frame according to the position of the target object, so as to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired of the first intermediate video frame; determines the pixel value of at least one target pixel in the area to be repaired of the first intermediate video frame according to the optical flow information and the adjacent video frames of the initial video frame, and fills the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel, so as to obtain a second intermediate video frame and an intermediate mask image; and calls a preset video repair model to repair the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image, so as to obtain a target video frame. In this embodiment, the optical flow information and the adjacent video frames can be combined for preliminary repair, so as to retain richer original textures and reduce the area of ​​the area to be repaired, and then the video repair model is used for repair, so as to improve the repair efficiency and the erasing quality of the target object.

[0032] The application scenarios of the video processing method of the present disclosure embodiment are as follows: Figure 1As shown, the method of this embodiment can be applied in electronic devices such as terminal devices or servers to obtain optical flow information of a video to be processed; determine the position of a target object in any initial video frame of the video to be processed, and segment the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired of the first intermediate video frame; determine the pixel value of at least one target pixel in the area to be repaired of the first intermediate video frame according to the optical flow information and adjacent video frames of the initial video frame, and fill the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain a second intermediate video frame and an intermediate mask image; call a preset video repair model according to the optical flow information, the second intermediate video frame and the intermediate mask image to repair the second intermediate video frame to obtain a target video frame, and finally output a target video from which the target object is erased.

[0033] The video processing method disclosed herein will be described in detail below in conjunction with specific embodiments.

[0034] refer to Figure 2 , Figure 2 The video processing method provided in one embodiment of the present disclosure is a flow chart. The method of this embodiment can be applied in electronic devices such as terminal devices or servers, and the video processing method includes:

[0035] S201, obtaining optical flow information of the video to be processed.

[0036] In this embodiment, in order to perform fine erasing of the target object in the processed video, considering that the partial area blocked by the target object in any video frame may not be blocked in the adjacent video frame, the original texture in the adjacent video frame can be used to complete the partial area blocked by the target object in the current video frame. Since the object in the video may move, the partial area blocked by the target object in the current video frame may not be in the same position in the adjacent video frame, so it is also necessary to determine the corresponding position of the partial area blocked by the target object in the current video frame in the adjacent video frame. Considering that optical flow is a two-dimensional vector field that describes the motion trajectory and speed of pixels over time in continuous frame images, it is used to capture the motion of objects and the displacement information between adjacent frames. In this embodiment, the optical flow information can be used to determine the corresponding position of the partial area blocked by the target object in the current video frame in the adjacent video frame. In this embodiment, optical flow information can be obtained for the processed video, and the method for obtaining the optical flow information can adopt any known method, such as a sparse optical flow algorithm, a dense optical flow algorithm, an optical flow algorithm based on deep learning, etc.

[0037] Optionally, in this embodiment, a preset optical flow model may be used to obtain optical flow information for the video to be processed, wherein the preset optical flow model may be any known optical flow algorithm model.

[0038] Furthermore, considering that the acquired optical flow information may be inaccurate or missing, optical flow completion can be performed on the initial optical flow information based on the acquired initial optical flow information, wherein the optical flow completion can be performed by any known method, such as an optical flow completion method based on forward flow, reverse flow and non-adjacent frames, or an optical flow completion method based on deep learning, etc.

[0039] Optionally, in this embodiment, according to the video to be processed and the initial optical flow information, a preset optical flow completion model can be called to complete the initial optical flow information to obtain the final optical flow information, wherein the preset optical flow completion model can be any known optical flow completion model.

[0040] S202. Determine the position of the target object in any initial video frame of the video to be processed, and segment the target object of the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein in the initial mask image, the target object is in a non-masked area, which is used to identify the area to be repaired in the first intermediate video frame.

[0041] In this embodiment, in order to perform fine-grained erasing of the target object in the video to be processed, it is also necessary to determine the position of the target object in each initial video frame of the video to be processed. Any recognition method can be used to identify the target object from the initial video frame to determine the position of the target object. Optionally, a recognition model can be used to identify the target object from the initial video frame, and the recognition model can be any machine learning model.

[0042] Furthermore, the target object can be segmented from the initial video frame according to the position of the target object, and the target object can be taken out from the initial video frame to obtain a video frame without the target object, which is recorded as the first intermediate video frame. In addition, a mask image (mask) of the target object can be obtained, which is recorded as the initial mask image. The mask image is a binary image with the same size as the original image, in which the selected area (non-masked area or unobstructed area) is marked as 1 (or True), and the remaining areas (masked areas or obstructed areas) are marked as 0 (or False), which is used to indicate a specific area to be processed when applying certain image processing operations. In this embodiment, the area of ​​the target object is in the non-masked area (selected area or unobstructed area) in the initial mask image, indicating that the area is an area to be repaired, that is, the non-target object area in the initial mask image does not need to be repaired but is obstructed, while the target object area needs to be repaired but is not obstructed.

[0043] Optionally, when determining the position of the target object in any initial video frame, the position of the target object to be erased can also be determined according to user needs. The user can pre-set preset screening conditions for the target object, such as the type, color, position, etc. of the target object, that is, only the target objects that meet the preset screening conditions need to be erased. Therefore, when determining the position of the target object in any initial video frame, all target objects in any initial video frame of the video to be processed can be first identified, and then based on the preset screening conditions, the target objects that meet the preset screening conditions can be screened out from all the target objects in the initial video frame, and the position of the target object that meets the preset screening conditions can be determined. Subsequent steps are performed based on the position of the target object, so that only the target objects that meet the preset screening conditions can be erased.

[0044] It should be noted that, in this embodiment, S201 and S202 may be executed in any order and may be executed in parallel.

[0045] S203. Determine the pixel value of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame according to the optical flow information and the adjacent video frames of the initial video frame, and fill the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain a second intermediate video frame and an intermediate mask image.

[0046] In this embodiment, based on the optical flow information and the adjacent video frames of the initial video frame, the corresponding pixel of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame in the adjacent video frame can be determined, and then the pixel value of the target pixel is determined based on the corresponding pixel in the adjacent video frame, and the corresponding pixel of the initial mask image is filled in the second intermediate video frame of the target pixel of an intermediate video frame according to the pixel value of the target pixel (the non-masked area is changed to the masked area, in other words, the corresponding pixel is changed from 1 to 0), indicating that the position of the corresponding pixel of the initial mask image has been repaired and no subsequent repair is required, thereby obtaining an intermediate mask image.

[0047] In a specific implementation, the pixel value of the target pixel can be filled into the target pixel of the first intermediate video frame to obtain the second intermediate video frame; in addition, the corresponding pixel of the target pixel in the corresponding initial mask image is determined, and the corresponding pixel in the initial mask image is masked to change the non-masked area into the masked area, and finally the intermediate mask image is obtained.

[0048] Optionally, in this embodiment, when determining the pixel value of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame, the corresponding pixel of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame in the adjacent video frame can be determined according to the optical flow information, and then the pixel value of the target pixel can be determined according to the corresponding pixel in the adjacent video frame, so that the original texture of the target pixel can be retained as much as possible based on the corresponding pixel in the adjacent video frame.

[0049] For pixels in the area to be repaired for which no corresponding pixels in the adjacent video frames are found, no operation is required in this step.

[0050] Of course, the above method for determining the pixel value of the target pixel may adopt any other feasible method, such as any feasible algorithm or machine learning model.

[0051] S204. Call a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame.

[0052] In this embodiment, since the second intermediate video frame has undergone a preliminary repair, but all pixels in the area to be repaired may not have been repaired, the preset video repair model can be called to further repair the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image, so that the second intermediate video frame is completely repaired, and finally the target video frame is obtained, and then the final target video can be output. The preset video repair model can be any machine learning model, such as a Transformer model, which is pre-trained to have the ability to perform video repair based on optical flow information, video frames and mask images.

[0053] It should be noted that in this embodiment, the target object can be any object that needs to be erased in the video to be processed, such as subtitles, advertising labels, characters, icons, etc. For example, subtitles, text-based advertisements, etc., since they are generally small and have complex strokes, the shape of the occluded area is also relatively complex, so they need to be erased in a refined manner, and the above-mentioned video processing method can be used; for other target objects that do not require refined erasure, although the above-mentioned video processing method can also be used for processing, the cost is relatively high, so other erasing methods can also be used, for example, the above-mentioned preset video repair model or other video repair models can be called for repair based only on optical flow information and the first intermediate video frame.

[0054] Optionally, in this embodiment, the video processing method of this embodiment may be executed on all video frames of the video to be processed, or the video processing method of this embodiment may be executed on a frame-by-frame basis of the video to be processed.

[0055] Furthermore, after the target object is erased, other video editing operations may be performed, such as adding new subtitles (eg, subtitles in different colors or languages), adding new advertisements, icons, and the like.

[0056] The video processing method provided in this embodiment obtains the optical flow information of the video to be processed; determines the position of the target object in any initial video frame of the video to be processed, and performs target object segmentation on the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired of the first intermediate video frame; determines the pixel value of at least one target pixel in the area to be repaired of the first intermediate video frame according to the optical flow information and the adjacent video frames of the initial video frame, and fills the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel, respectively, to obtain a second intermediate video frame and an intermediate mask image; and calls a preset video repair model to repair the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image, to obtain a target video frame. In this embodiment, the optical flow information and the adjacent video frames can be combined for preliminary repair, so as to retain richer original textures and reduce the area of ​​the area to be repaired, and then the video repair model is used for repair, thereby improving the repair efficiency and the erasing quality of the target object.

[0057] In any of the above embodiments, when determining the pixel value of the target pixel according to the corresponding pixel in the adjacent video frame, one of the following methods may be specifically adopted:

[0058] Method 1: Determine the pixel value that appears most frequently among the pixel values ​​of corresponding pixels in the adjacent video frames, and determine the pixel value that appears most frequently as the pixel value of the target pixel.

[0059] In this embodiment, since there may be multiple adjacent video frames, such as several frames before and after, the pixel values ​​of the corresponding pixels of the target pixel in each adjacent video frame may be exactly the same, or there may be certain differences. In this embodiment, the pixel value that appears most times can be determined from the pixel values ​​of the corresponding pixels in the adjacent video frames. The pixel value that appears most times has higher accuracy, so the pixel value that appears most times can be determined as the pixel value of the target pixel.

[0060] Method 2: performing weighted averaging on the pixel values ​​of corresponding pixels in the adjacent video frames, and determining the weighted averaging result as the pixel value of the target pixel.

[0061] In this embodiment, the pixel values ​​of corresponding pixels in adjacent video frames may be weighted averaged, and the pixel values ​​of corresponding pixels in multiple adjacent video frames may be comprehensively considered, and the weighted average result may be determined as the pixel value of the target pixel. The weight of the weighted average may be set according to actual conditions, or may be determined by pre-training.

[0062] Of course, the pixel value of the target pixel may also be determined according to corresponding pixels in adjacent video frames by other methods, which will not be elaborated here.

[0063] Based on any of the above embodiments, since the pixel value of the target pixel is inferred, it may not be accurate. Therefore, before filling the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel, the confidence of the pixel value of the target pixel may also be considered. When the confidence is high, for example, higher than a preset confidence threshold, the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image are filled according to the pixel value of the target pixel. Otherwise, if the confidence is not high, the pixel value of the target pixel is ignored.

[0064] The confidence level of the pixel value of the target pixel can be determined in any feasible manner, such as by a pre-trained machine learning model, in which the video to be processed, the position of the target pixel, and the pixel value can be input into the model to output the confidence level of the pixel value of the target pixel. The specific model is not limited in this embodiment.

[0065] On the basis of any of the above embodiments, S204 calls a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, which may specifically include:

[0066] According to the optical flow information, the second intermediate video frame and the intermediate mask image, the preset video restoration model is called to restore the area to be restored of the second intermediate video frame to obtain a target video frame, wherein the area to be restored of the second intermediate video frame is the area in the second intermediate video frame corresponding to the non-masked area of ​​the intermediate mask image.

[0067] In this embodiment, in order to reduce the computational complexity of the preset video repair model and improve the repair efficiency, the preset video repair model can be allowed to repair only the area to be repaired in the second intermediate video frame. The area to be repaired in the second intermediate video frame is the area in the second intermediate video frame corresponding to the non-mask area of ​​the intermediate mask image, that is, the non-mask area of ​​the initial mask image minus the pixels filled by the initial repair, which greatly reduces the area of ​​the area to be repaired, thereby reducing the computational complexity of the preset video repair model.

[0068] On the basis of any of the above embodiments, when repairing the area to be repaired in the current second intermediate video frame, feature extraction can be performed within a certain time and space range, and local features can be extracted by combining images of the same window area of ​​adjacent second intermediate video frames. Of course, global features can also be extracted by combining the overall image of the current second intermediate video frame to improve the accuracy and completeness of the features, and then the area to be repaired in the current second intermediate video frame can be repaired based on the features.

[0069] In an optional embodiment, the preset video restoration model is a Transformer model. The Transformer model is a neural network model based on a self-attention mechanism. In this embodiment, the images of the target areas of multiple second intermediate video frames can be processed simultaneously through the self-attention mechanism to better perform spatiotemporal prediction learning, capture long-distance dependencies, and repair the area to be repaired in the current second intermediate video frame. Of course, other models can also be used. The preset video restoration model includes an encoder (Encoder) and a decoder (Decoder), wherein the encoder is used to extract features and the decoder is used to generate target video frames. The Transformer model is used for explanation below, and the processing process of other models can be adaptively adjusted according to the specific model. Specifically, if Figure 3 As shown, the method of invoking the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame includes:

[0070] S301, acquiring an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0071] S302, inputting the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map;

[0072] S303: Input the feature map into the decoder of the preset video restoration model to generate the target video frame.

[0073] In this embodiment, in order to improve the processing efficiency of the Transformer model and reduce the amount of data, in this embodiment, only the image of the neighborhood range of the area to be repaired in the second intermediate video frame (the preset range around the area to be repaired), that is, the image of the area corresponding to the preset neighborhood of the non-masked area of ​​the intermediate mask image in the second intermediate video frame, is required, without taking the entire second intermediate video frame as the input of the Transformer model. At the same time, it is necessary to refer to the image of the same area in the adjacent second intermediate video frame to form a set of time-series images, and then input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder (Encoder) of the Transformer model to extract the feature map, wherein the encoder (Encoder) of the Transformer model is used to extract features from the input data and output a feature map (vector), which includes the dependency and context information of the input sequence, etc., and may specifically include a multi-head self-attention mechanism (Multi-Head Self-Attention) layer for capturing the global dependency between image blocks, a feed-forward neural network (Feed-Forward The first layer is a network layer, which is used to perform further nonlinear transformation on the features of each image block and enhance the expressiveness of the model. The second layer is a residual connection and a layer normalization (Add&Norm) layer, which is used to alleviate the gradient vanishing problem through residual connection and make the input distribution of each layer more stable through layer normalization. The specific processing process is not described here.

[0074] More specifically, before being input into the encoder of the Transformer model, the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame can be embedded to generate corresponding vector representations, which are then input into the encoder of the Transformer model.

[0075] Furthermore, the feature map output by the encoder is input into the decoder of the Transformer model for prediction, thereby generating a repaired target video frame, wherein the decoder of the Transformer model is used to predict and generate a sequence according to the encoder output sequence, and may specifically include a masked multi-head self-attention layer for calculating the attention score between each token and other tokens in the target sequence, a multi-head self-attention mechanism layer for aligning the current state of the decoder with the output of the encoder, allowing the decoder to refer to the information of the input sequence when generating the target sequence, a feed-forward neural network for further processing the output of the attention layer through a series of nonlinear transformations, a residual connection and layer normalization (Add&Norm) layer for alleviating the gradient vanishing problem through residual connections, and making the input distribution of each layer more stable through layer normalization, etc. The specific processing process is not repeated here.

[0076] In this embodiment, only the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame are used as inputs, and the global features are not considered. When the global features are considered, it can be divided into the following two cases. One case is to reduce the current second intermediate video frame to the size of the target area and also use it as the input of the encoder to achieve image-level fusion. The other case is to extract features from the current second intermediate video frame alone to obtain a global feature map, and perform feature-level fusion with the feature map output by the encoder. The following is a specific introduction:

[0077] In an alternative embodiment, Figure 4 As shown, according to the optical flow information, the second intermediate video frame and the intermediate mask image, the preset video restoration model is called to restore the to-be-restored area of ​​the second intermediate video frame to obtain a target video frame, including

[0078] S401, acquiring an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0079] S402, reducing the current second intermediate video frame to a size of the target area to obtain a reduced current second intermediate video frame;

[0080] S403, inputting the optical flow information, the reduced current second intermediate video frame, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map;

[0081] S404: Input the feature map into the decoder of the preset video restoration model to generate the target video frame.

[0082] In this embodiment, an image of the target area in the current second intermediate video frame and images of the target areas in a preset number of adjacent second intermediate video frames are obtained, similar to the above embodiment S301. In addition, the current second intermediate video frame is reduced (Resized) to the size of the target area, and then the optical flow information, the reduced current second intermediate video frame, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame are input into the encoder of the Transformer model to extract a feature map. The obtained feature map is a feature map that integrates global features, and then the feature map is input into the decoder of the Transformer model to generate a repaired target video frame, similar to the above embodiment S303.

[0083] In another alternative embodiment, Figure 5 As shown, according to the optical flow information, the second intermediate video frame and the intermediate mask image, the preset video restoration model is called to restore the to-be-restored area of ​​the second intermediate video frame to obtain a target video frame, including

[0084] S501, acquiring an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0085] S502, inputting the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a first feature map;

[0086] S503, extracting features from the current second intermediate video frame to obtain a second feature map;

[0087] S504: Fusing the first feature map with the second feature map to obtain a fused feature map;

[0088] S505: Input the fused feature map into the decoder of the preset video restoration model to generate the target video frame.

[0089] In this embodiment, S501-S502 are the same as the above-mentioned embodiments S301-S302, that is, the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame are input into the encoder of the Transformer model to extract the first feature map, which is a local feature map; and S503 extracts features from the current second intermediate video frame alone to obtain the second feature map, which is a global feature map, wherein a separate feature extraction layer can be used to extract features from the current second intermediate video frame, or the encoder of the Transformer model can also be used. Furthermore, the first feature map and the second feature map can be fused to achieve the fusion of local features and global features to obtain a fused feature map, and then the fused feature map is input into the decoder of the Transformer model to generate a repaired target video frame, which is the same as the above-mentioned embodiment S303.

[0090] Through the video processing methods of the above-mentioned embodiments, it is possible to achieve fine-grained erasing of the target object in the video to be processed. In the preliminary repair process, the optical flow information is combined to retain the rich original texture as much as possible, and the area that needs to be repaired is reduced. The video repair model is then used for repair, which improves the repair efficiency and the erasing quality of the target object.

[0091] It should be noted that the above Figure 3-Figure 5 In the method shown, the preset video restoration model is not limited to the Transformer model for illustration, and the processing principles of other models are similar and can be adaptively adjusted according to the specific model.

[0092] Corresponding to the video processing method of the above embodiment, Figure 6 The structure block diagram of the video processing device provided by the embodiment of the present disclosure. For the convenience of explanation, only the part related to the embodiment of the present disclosure is shown. Figure 6 The video processing device 600 includes: an optical flow acquisition unit 601, a target object recognition unit 602, a first restoration unit 603, and a second restoration unit 604.

[0093] The optical flow acquisition unit 601 is used to acquire the optical flow information of the video to be processed;

[0094] The target object identification unit 602 is used to determine the position of the target object in any initial video frame of the video to be processed, and to segment the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired in the first intermediate video frame;

[0095] A first restoration unit 603 is configured to determine a pixel value of at least one target pixel in a to-be-restored area of ​​the first intermediate video frame according to the optical flow information and adjacent video frames of the initial video frame, and fill the target pixel of the first intermediate video frame and corresponding pixels of the initial mask image according to the pixel value of the target pixel, to obtain a second intermediate video frame and an intermediate mask image;

[0096] The second restoration unit 604 is used to call a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image to obtain a target video frame.

[0097] In one or more embodiments of the present disclosure, when the first repairing unit 603 fills the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain the second intermediate video frame and the intermediate mask image, it is configured to:

[0098] Filling the pixel value of the target pixel into the target pixel of the first intermediate video frame to obtain the second intermediate video frame;

[0099] The corresponding pixel of the target pixel in the initial mask image is determined, and the corresponding pixel in the initial mask image is subjected to mask processing to obtain the intermediate mask image.

[0100] In one or more embodiments of the present disclosure, when determining the pixel value of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame according to the optical flow information and the adjacent video frames of the initial video frame, the first restoration unit 603 is configured to:

[0101] A corresponding pixel of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame in an adjacent video frame is determined according to the optical flow information, and a pixel value of the target pixel is determined according to the corresponding pixel in the adjacent video frame.

[0102] In one or more embodiments of the present disclosure, when determining the pixel value of the target pixel according to the corresponding pixel in the adjacent video frame, the first repairing unit 603 is configured to:

[0103] Determine the pixel value that appears most frequently among the pixel values ​​of corresponding pixels in the adjacent video frames, and determine the pixel value that appears most frequently as the pixel value of the target pixel; or

[0104] The pixel values ​​of corresponding pixels in the adjacent video frames are weighted averaged, and the weighted average result is determined as the pixel value of the target pixel.

[0105] In one or more embodiments of the present disclosure, when the first repairing unit 603 fills the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain the second intermediate video frame and the intermediate mask image, it is configured to:

[0106] Determine the confidence of the pixel value of the target pixel. If the confidence is higher than a preset confidence threshold, fill the target pixel of the first intermediate video frame and the corresponding pixels of the initial mask image according to the pixel value of the target pixel to obtain a second intermediate video frame and an intermediate mask image.

[0107] In one or more embodiments of the present disclosure, when the second restoration unit 604 calls a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, it is configured to:

[0108] According to the optical flow information, the second intermediate video frame and the intermediate mask image, the preset video restoration model is called to restore the area to be restored of the second intermediate video frame to obtain a target video frame, wherein the area to be restored of the second intermediate video frame is the area in the second intermediate video frame corresponding to the non-masked area of ​​the intermediate mask image.

[0109] In one or more embodiments of the present disclosure, when the second restoration unit 604 calls the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, it is configured to:

[0110] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0111] Input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map;

[0112] The feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0113] In one or more embodiments of the present disclosure, when the second restoration unit 604 calls the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, it is configured to:

[0114] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0115] Reducing the current second intermediate video frame to the size of the target area to obtain a reduced current second intermediate video frame;

[0116] Input the optical flow information, the reduced current second intermediate video frame, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map;

[0117] The feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0118] In one or more embodiments of the present disclosure, when the second restoration unit 604 calls the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, it is configured to:

[0119] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0120] Input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a first feature map;

[0121] Extract features from the current second intermediate video frame to obtain a second feature map;

[0122] Fusing the first feature map with the second feature map to obtain a fused feature map;

[0123] The fused feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0124] In one or more embodiments of the present disclosure, when acquiring the optical flow information of the video to be processed, the optical flow acquisition unit 601 is used to:

[0125] According to the video to be processed, calling a preset optical flow model to obtain initial optical flow information;

[0126] According to the video to be processed and the initial optical flow information, a preset optical flow completion model is called to complete the initial optical flow information to obtain the optical flow information of the video to be processed.

[0127] In one or more embodiments of the present disclosure, when determining the position of the target object in any initial video frame of the video to be processed, the target object recognition unit 602 is used to:

[0128] Identify all target objects in any initial video frame of the video to be processed;

[0129] Target objects that meet preset screening conditions are screened out from all target objects in the initial video frame, and positions of the target objects that meet the preset screening conditions are determined.

[0130] In one or more embodiments of the present disclosure, the target object is a subtitle.

[0131] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0132] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0133] refer to Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure, and the electronic device 700 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0134] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0135] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0136] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0137] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0138] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0139] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0140] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0143] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0144] In a first aspect, according to one or more embodiments of the present disclosure, a video processing method is provided, including:

[0145] Obtain the optical flow information of the video to be processed;

[0146] Determine the position of the target object in any initial video frame of the video to be processed, and segment the target object on the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired in the first intermediate video frame;

[0147] Determine, according to the optical flow information and adjacent video frames of the initial video frame, a pixel value of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame, and fill the target pixel of the first intermediate video frame and corresponding pixels of the initial mask image according to the pixel value of the target pixel, to obtain a second intermediate video frame and an intermediate mask image;

[0148] According to the optical flow information, the second intermediate video frame and the intermediate mask image, a preset video restoration model is called to restore the second intermediate video frame to obtain a target video frame.

[0149] According to one or more embodiments of the present disclosure, the step of filling the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain the second intermediate video frame and the intermediate mask image includes:

[0150] Filling the pixel value of the target pixel into the target pixel of the first intermediate video frame to obtain the second intermediate video frame;

[0151] The corresponding pixel of the target pixel in the initial mask image is determined, and the corresponding pixel in the initial mask image is subjected to mask processing to obtain the intermediate mask image.

[0152] According to one or more embodiments of the present disclosure, determining the pixel value of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame according to the optical flow information and adjacent video frames of the initial video frame includes:

[0153] A corresponding pixel of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame in an adjacent video frame is determined according to the optical flow information, and a pixel value of the target pixel is determined according to the corresponding pixel in the adjacent video frame.

[0154] According to one or more embodiments of the present disclosure, determining the pixel value of the target pixel according to the corresponding pixel in the adjacent video frame includes:

[0155] Determine the pixel value that appears most frequently among the pixel values ​​of corresponding pixels in the adjacent video frames, and determine the pixel value that appears most frequently as the pixel value of the target pixel; or

[0156] The pixel values ​​of corresponding pixels in the adjacent video frames are weighted averaged, and the weighted average result is determined as the pixel value of the target pixel.

[0157] According to one or more embodiments of the present disclosure, the step of filling the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain the second intermediate video frame and the intermediate mask image includes:

[0158] Determine the confidence of the pixel value of the target pixel. If the confidence is higher than a preset confidence threshold, fill the target pixel of the first intermediate video frame and the corresponding pixels of the initial mask image according to the pixel value of the target pixel to obtain a second intermediate video frame and an intermediate mask image.

[0159] According to one or more embodiments of the present disclosure, the method of invoking a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame includes:

[0160] According to the optical flow information, the second intermediate video frame and the intermediate mask image, the preset video restoration model is called to restore the area to be restored of the second intermediate video frame to obtain a target video frame, wherein the area to be restored of the second intermediate video frame is the area in the second intermediate video frame corresponding to the non-masked area of ​​the intermediate mask image.

[0161] According to one or more embodiments of the present disclosure, the method of invoking the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame includes:

[0162] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0163] Input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map;

[0164] The feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0165] According to one or more embodiments of the present disclosure, the preset video restoration model is called to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image to obtain a target video frame, including:

[0166] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0167] Reducing the current second intermediate video frame to the size of the target area to obtain a reduced current second intermediate video frame;

[0168] Input the optical flow information, the reduced current second intermediate video frame, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map;

[0169] The feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0170] According to one or more embodiments of the present disclosure, the preset video restoration model is called to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image to obtain a target video frame, including:

[0171] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0172] Input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a first feature map;

[0173] Extract features from the current second intermediate video frame to obtain a second feature map;

[0174] Fusing the first feature map with the second feature map to obtain a fused feature map;

[0175] The fused feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0176] According to one or more embodiments of the present disclosure, the step of obtaining optical flow information of a video to be processed includes:

[0177] According to the video to be processed, calling a preset optical flow model to obtain initial optical flow information;

[0178] According to the video to be processed and the initial optical flow information, a preset optical flow completion model is called to complete the initial optical flow information to obtain the optical flow information of the video to be processed.

[0179] According to one or more embodiments of the present disclosure, determining the position of the target object in any initial video frame of the video to be processed includes:

[0180] Identify all target objects in any initial video frame of the video to be processed;

[0181] Target objects that meet preset screening conditions are screened out from all target objects in the initial video frame, and positions of the target objects that meet the preset screening conditions are determined.

[0182] According to one or more embodiments of the present disclosure, the target object is a subtitle.

[0183] In a second aspect, according to one or more embodiments of the present disclosure, a video processing device is provided, including:

[0184] An optical flow acquisition unit, used to acquire optical flow information of the video to be processed;

[0185] A target object identification unit is used to determine the position of the target object in any initial video frame of the video to be processed, and to segment the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired in the first intermediate video frame;

[0186] A first restoration unit is used to determine a pixel value of at least one target pixel in a to-be-restored area of ​​the first intermediate video frame according to the optical flow information and adjacent video frames of the initial video frame, and fill the target pixel of the first intermediate video frame and corresponding pixels of the initial mask image according to the pixel value of the target pixel, so as to obtain a second intermediate video frame and an intermediate mask image;

[0187] The second restoration unit is used to call a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image to obtain a target video frame.

[0188] According to one or more embodiments of the present disclosure, when the first repairing unit fills the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain the second intermediate video frame and the intermediate mask image, it is configured to:

[0189] Filling the pixel value of the target pixel into the target pixel of the first intermediate video frame to obtain the second intermediate video frame;

[0190] The corresponding pixel of the target pixel in the initial mask image is determined, and the corresponding pixel in the initial mask image is subjected to mask processing to obtain the intermediate mask image.

[0191] According to one or more embodiments of the present disclosure, when the first restoration unit determines the pixel value of at least one target pixel in the to-be-restored area of ​​the first intermediate video frame according to the optical flow information and the adjacent video frames of the initial video frame, it is configured to:

[0192] A corresponding pixel of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame in an adjacent video frame is determined according to the optical flow information, and a pixel value of the target pixel is determined according to the corresponding pixel in the adjacent video frame.

[0193] According to one or more embodiments of the present disclosure, when determining the pixel value of the target pixel according to the corresponding pixel in the adjacent video frame, the first repairing unit is configured to:

[0194] Determine the pixel value that appears most frequently among the pixel values ​​of corresponding pixels in the adjacent video frames, and determine the pixel value that appears most frequently as the pixel value of the target pixel; or

[0195] The pixel values ​​of corresponding pixels in the adjacent video frames are weighted averaged, and the weighted average result is determined as the pixel value of the target pixel.

[0196] According to one or more embodiments of the present disclosure, when the first repairing unit fills the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain the second intermediate video frame and the intermediate mask image, it is configured to:

[0197] Determine the confidence of the pixel value of the target pixel. If the confidence is higher than a preset confidence threshold, fill the target pixel of the first intermediate video frame and the corresponding pixels of the initial mask image according to the pixel value of the target pixel to obtain a second intermediate video frame and an intermediate mask image.

[0198] According to one or more embodiments of the present disclosure, when the second restoration unit calls a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, it is configured to:

[0199] According to the optical flow information, the second intermediate video frame and the intermediate mask image, the preset video restoration model is called to restore the area to be restored of the second intermediate video frame to obtain a target video frame, wherein the area to be restored of the second intermediate video frame is the area in the second intermediate video frame corresponding to the non-masked area of ​​the intermediate mask image.

[0200] According to one or more embodiments of the present disclosure, when the second restoration unit calls the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, it is configured to:

[0201] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0202] Input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map;

[0203] The feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0204] According to one or more embodiments of the present disclosure, when the second restoration unit calls the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, it is configured to:

[0205] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0206] Reducing the current second intermediate video frame to the size of the target area to obtain a reduced current second intermediate video frame;

[0207] Input the optical flow information, the reduced current second intermediate video frame, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map;

[0208] The feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0209] According to one or more embodiments of the present disclosure, when the second restoration unit calls the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame, it is configured to:

[0210] Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image;

[0211] Input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a first feature map;

[0212] Extract features from the current second intermediate video frame to obtain a second feature map;

[0213] Fusing the first feature map with the second feature map to obtain a fused feature map;

[0214] The fused feature map is input into the decoder of the preset video restoration model to generate the target video frame.

[0215] According to one or more embodiments of the present disclosure, when acquiring the optical flow information of the video to be processed, the optical flow acquisition unit is used to:

[0216] According to the video to be processed, calling a preset optical flow model to obtain initial optical flow information;

[0217] According to the video to be processed and the initial optical flow information, a preset optical flow completion model is called to complete the initial optical flow information to obtain the optical flow information of the video to be processed.

[0218] According to one or more embodiments of the present disclosure, when determining the position of the target object in any initial video frame of the video to be processed, the target object recognition unit is used to:

[0219] Identify all target objects in any initial video frame of the video to be processed;

[0220] Target objects that meet preset screening conditions are screened out from all target objects in the initial video frame, and positions of the target objects that meet the preset screening conditions are determined.

[0221] According to one or more embodiments of the present disclosure, the target object is a subtitle.

[0222] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0223] The memory stores computer-executable instructions;

[0224] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video processing method described in the first aspect and various possible designs of the first aspect.

[0225] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the video processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0226] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the video processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0227] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0228] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0229] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that: include: Obtain the optical flow information of the video to be processed; Determine the position of the target object in any initial video frame of the video to be processed, and segment the target object on the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, which is used to identify the area to be repaired in the first intermediate video frame; Determine, according to the optical flow information and adjacent video frames of the initial video frame, a pixel value of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame, and fill the target pixel of the first intermediate video frame and corresponding pixels of the initial mask image according to the pixel value of the target pixel, to obtain a second intermediate video frame and an intermediate mask image; According to the optical flow information, the second intermediate video frame and the intermediate mask image, a preset video restoration model is called to restore the second intermediate video frame to obtain a target video frame.

2. The method according to claim 1, characterized in that: The step of filling the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain a second intermediate video frame and an intermediate mask image includes: Filling the pixel value of the target pixel into the target pixel of the first intermediate video frame to obtain the second intermediate video frame; The corresponding pixel of the target pixel in the initial mask image is determined, and the corresponding pixel in the initial mask image is subjected to mask processing to obtain the intermediate mask image.

3. The method according to claim 1, characterized in that The step of determining, according to the optical flow information and adjacent video frames of the initial video frame, a pixel value of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame comprises: A corresponding pixel of at least one target pixel in the to-be-repaired area of ​​the first intermediate video frame in an adjacent video frame is determined according to the optical flow information, and a pixel value of the target pixel is determined according to the corresponding pixel in the adjacent video frame.

4. The method according to claim 3, characterized in that: The step of determining the pixel value of the target pixel according to the corresponding pixel in the adjacent video frames comprises: Determine the pixel value that appears most frequently among the pixel values ​​of corresponding pixels in the adjacent video frames, and determine the pixel value that appears most frequently as the pixel value of the target pixel; or The pixel values ​​of corresponding pixels in the adjacent video frames are weighted averaged, and the weighted average result is determined as the pixel value of the target pixel.

5. The method according to any one of claims 1 to 4, characterized in that: The step of filling the target pixel of the first intermediate video frame and the corresponding pixel of the initial mask image according to the pixel value of the target pixel to obtain a second intermediate video frame and an intermediate mask image includes: Determine the confidence of the pixel value of the target pixel. If the confidence is higher than a preset confidence threshold, fill the target pixel of the first intermediate video frame and the corresponding pixels of the initial mask image according to the pixel value of the target pixel to obtain a second intermediate video frame and an intermediate mask image.

6. The method according to any one of claims 1 to 4, characterized in that: The method of invoking a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame includes: According to the optical flow information, the second intermediate video frame and the intermediate mask image, the preset video restoration model is called to restore the area to be restored of the second intermediate video frame to obtain a target video frame, wherein the area to be restored of the second intermediate video frame is the area in the second intermediate video frame corresponding to the non-masked area of ​​the intermediate mask image.

7. The method according to claim 6, characterized in that The method of invoking the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame, and the intermediate mask image to obtain a target video frame includes: Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image; Input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map; The feature map is input into the decoder of the preset video restoration model to generate the target video frame.

8. The method according to claim 6, characterized in that The method of invoking the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image to obtain a target video frame includes: Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image; Reducing the current second intermediate video frame to the size of the target area to obtain a reduced current second intermediate video frame; Input the optical flow information, the reduced current second intermediate video frame, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a feature map; The feature map is input into the decoder of the preset video restoration model to generate the target video frame.

9. The method according to claim 6, characterized in that The method of invoking the preset video restoration model to restore the to-be-restored area of ​​the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image to obtain a target video frame includes: Acquire an image of a target area in a current second intermediate video frame and images of the target area in a preset number of adjacent second intermediate video frames, wherein the target area is an area corresponding to a preset neighborhood of a non-masked area of ​​the intermediate mask image; Input the optical flow information, the image of the target area of ​​the current second intermediate video frame, and the image of the target area of ​​the adjacent second intermediate video frame into the encoder of the preset video restoration model to extract a first feature map; Extract features from the current second intermediate video frame to obtain a second feature map; Fusing the first feature map with the second feature map to obtain a fused feature map; The fused feature map is input into the decoder of the preset video restoration model to generate the target video frame.

10. The method according to claim 1, characterized in that The step of obtaining optical flow information of the video to be processed includes: According to the video to be processed, calling a preset optical flow model to obtain initial optical flow information; According to the video to be processed and the initial optical flow information, a preset optical flow completion model is called to complete the initial optical flow information to obtain the optical flow information of the video to be processed.

11. The method according to claim 1, characterized in that The determining the position of the target object in any initial video frame of the video to be processed includes: Identify all target objects in any initial video frame of the video to be processed; Target objects that meet preset screening conditions are screened out from all target objects in the initial video frame, and positions of the target objects that meet the preset screening conditions are determined.

12. The method according to claim 1, characterized in that The target object is a subtitle.

13. A video processing device, characterized in that: include: An optical flow acquisition unit, used to acquire optical flow information of the video to be processed; A target object recognition unit, used to determine the position of the target object in any initial video frame of the video to be processed, and to segment the initial video frame according to the position of the target object to obtain a first intermediate video frame and an initial mask image from which the target object is removed; wherein the target object is in a non-masked area in the initial mask image, and is used to identify the area to be repaired in the first intermediate video frame; A first restoration unit is configured to determine a pixel value of at least one target pixel in a to-be-restored area of ​​the first intermediate video frame according to the optical flow information and adjacent video frames of the initial video frame, and fill the target pixel of the first intermediate video frame and corresponding pixels of the initial mask image according to the pixel value of the target pixel, so as to obtain a second intermediate video frame and an intermediate mask image; The second restoration unit is used to call a preset video restoration model to restore the second intermediate video frame according to the optical flow information, the second intermediate video frame and the intermediate mask image to obtain a target video frame.

14. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 11 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images