Video subtitle removal method and device

Abstract

The present application discloses a method for removing subtitles from a video. The method includes: obtaining a current video frame from which subtitles are to be removed and a previous video frame of the current video frame; inputting the current video frame and the previous video frame into a subtitle removal model to obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion area information corresponding to the optical flow map, and a repaired image, wherein the occlusion area information is used to determine an area where the subtitle pixels in the current video frame cannot be replaced using the optical flow map; performing subtitle removal processing on the current video frame based on the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame, and the previous video frame to obtain an image with subtitles removed. The present application can repair images containing subtitles.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to a method and device for removing video subtitles. Background Art

[0002] With the prevalence of smart devices, people are increasingly communicating online by sharing videos. These videos often come with subtitles or comments, which can sometimes affect the viewing experience. Therefore, we might want to remove these subtitles or comments when watching videos.

[0003] In the prior art, in order to remove text from a video, the position of the text is detected through video completion and repair technology, and then the detected text position is filled with pixels. However, the inventors found that the pixel filling effect in the prior art is poor, which affects the user's viewing experience of the video. Summary of the Invention

[0004] In view of this, a method, apparatus, computer device and computer-readable storage medium for removing video subtitles are provided to solve the problem in the prior art of poor effect when pixel filling is performed on the subtitle position, resulting in a poor user experience.

[0005] This application provides a video subtitle removal method, including:

[0006] Obtaining a current video frame from which subtitles are to be removed and a previous video frame of the current video frame from the video;

[0007] Inputting the current video frame and the previous video frame into a subtitle removal model to obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion area information corresponding to the optical flow map, and a repaired image, wherein the occlusion area information is used to determine an area in which subtitle pixels in the current video frame cannot be replaced using the optical flow map;

[0008] Subtitle removal processing is performed on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame and the previous video frame to obtain an image with subtitles removed.

[0009] Optionally, performing subtitle removal processing on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame, and the previous video frame to obtain the subtitle-removed image includes:

[0010] Determining subtitle pixels contained in the current video frame according to subtitle position information and the current video frame;

[0011] replacing subtitle pixels contained in the current video frame according to the optical flow map and the previous video frame to obtain a replaced image;

[0012] The subtitle-removed image is generated according to the occlusion area information, the repaired image, and the replaced image.

[0013] Optionally, generating the subtitle-removed image according to the occlusion area information, the repaired image, and the replaced image includes:

[0014] Determining a replacement pixel point according to the occluded area information and the repaired image;

[0015] The replacement pixel point replaces the pixel point corresponding to the replacement pixel point in the replaced image to generate the subtitle removal model. Optionally, the subtitle removal model is pre-trained in the following manner:

[0016] Acquire a plurality of training sample images, wherein each training sample image includes a first video frame and a video frame preceding the first video frame;

[0017] generating training labels for the plurality of training sample images according to the subtitle addition position in the first video frame of the plurality of training sample images and the original image corresponding to the first video frame;

[0018] A preset neural network model is trained based on a plurality of training sample images and respective training labels to obtain the subtitle removal model.

[0019] Optionally, each training sample image is obtained as follows:

[0020] Obtaining a second video frame and a previous video frame of the second video frame from a video without subtitles;

[0021] adding subtitles to the second video frame to obtain the first video frame;

[0022] The first video frame and the previous video frame of the second video frame are used as a training sample image.

[0023] Optionally, obtaining a current video frame from which subtitles are to be removed and a previous video frame of the current video frame from the video includes:

[0024] A subtitle detection model is used to detect each video frame of the video, and when subtitles are detected, a current video frame from which subtitles are to be removed and a previous video frame of the current video frame are obtained from the video.

[0025] Optionally, the subtitle removal model includes an encoder and a decoder, and inputting the current video frame and the previous video frame into the subtitle removal model to obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion area information corresponding to the optical flow map, and a repaired image includes:

[0026] The current video frame and the previous video frame are input into a subtitle removal model, the current video frame and the previous video frame are encoded by the encoder to obtain a feature map, and the feature map is decoded by the decoder to obtain the subtitle position information contained in the current video frame, the optical flow map of the current video frame relative to the previous video frame, the occlusion area information corresponding to the optical flow map, and the repaired image, wherein the decoder contains a convGRU neural network structure.

[0027] The present application also provides a video subtitle removal device, comprising:

[0028] An acquisition module is used to acquire a current video frame from which subtitles are to be removed and a previous video frame of the current video frame from which subtitles are to be removed.

[0029] An input module, configured to input the current video frame and the previous video frame into a subtitle removal model, and obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion area information corresponding to the optical flow map, and a repaired image, wherein the occlusion area information is used to determine areas in the current video frame where subtitle pixels cannot be replaced using the optical flow map;

[0030] The subtitle processing module is used to perform subtitle removal processing on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame and the previous video frame to obtain an image with subtitles removed.

[0031] The present application also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0032] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0033] In this embodiment, the image is encoded by an encoder in a pre-trained subtitle removal model to obtain a feature map, and then the feature map is decoded by a decoder to predict the subtitle position information, optical flow map, occlusion area information, and repaired image of the current video frame from which the subtitles are to be removed. Then, the image after subtitle removal can be synthesized based on the subtitle position information, optical flow map, occlusion area information, repaired image, and the current video frame from which the subtitles are to be removed. Since the subtitle removal model trained in an end-to-end manner in this application can accurately predict the subtitle position information, optical flow map, occlusion area information, and repaired image, when performing pixel replacement and filling, the image after subtitle removal synthesized based on the subtitle position information, optical flow map, occlusion area information, repaired image, and the current video frame from which the subtitles are to be removed can be closer to the original video frame, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of an environment of an embodiment of the video subtitle removal method of the present application;

[0035] Figure 2 This is a schematic diagram of an environment of another embodiment of the video subtitle removal method of the present application;

[0036] Figure 3 This is a flowchart of an embodiment of the video subtitle removal method described in this application;

[0037] Figure 4 This is a schematic diagram of a detailed flow chart of the steps of performing target object detection processing on the original frame image of the video using a first preset frequency during video playback, and outputting a display area of ​​the target object in the original frame image when the target object is detected, in one embodiment of the present application;

[0038] Figure 5 This is a schematic diagram of a detailed flow chart of the steps of performing target object detection processing on the original frame image of the video using a first preset frequency and outputting a display area of ​​the target object in the original frame image when the target object is detected in one embodiment of the present application;

[0039] Figure 6 This is a schematic diagram of a detailed flow chart of the steps of performing target object detection processing on an original frame image of a video using a first preset frequency during video playback, and outputting a display area of ​​the target object in the original frame image when the target object is detected, in one embodiment of the present application;

[0040] Figure 7 A program module diagram of an embodiment of the video subtitle removal device described in this application;

[0041] Figure 8 A schematic diagram of the hardware structure of a computer device for executing a video subtitle removal method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The advantages of the present application are further described below with reference to the accompanying drawings and specific embodiments.

[0043] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0044] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0046] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed. They are only used to facilitate the description of this application and to distinguish each step. Therefore, they cannot be understood as limitations on this application.

[0047] Figure 1 The following is a schematic diagram of an environment of a method for removing subtitles from a video according to an embodiment of the present application.

[0048] The terminal device 2 can be configured to remove subtitles from the current video frame based on the current video frame and the previous frame of the current video frame, and generate an image after subtitle removal. The terminal device 2 can include, for example, a smart phone, a tablet device, a laptop computer, a workstation, etc.

[0049] The terminal device 2 may include a client 2A, such as an application for video subtitle removal, and the client 2A may output (eg, display, render, present) an image obtained after subtitle removal to a user.

[0050] The server 4 can connect multiple terminal devices 2 via a network 3. The server 4 can be located in a data center such as a single location, or distributed in different physical locations (e.g., in multiple locations). The server 4 can provide services via one or more networks 3. The network 3 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 3 can include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, etc. The network 3 can include wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0051] Server 4 can be an application server for providing some functional services. Server 4 includes multiple network nodes. Multiple network nodes can handle tasks associated with message services. Multiple network nodes can be implemented as one or more computing devices, one or more processors, one or more virtual computing instances, combinations thereof, and / or the like. Multiple network nodes can be implemented by one or more computer devices. One or more computer devices can include virtualized computing instances. Virtualized computing instances can include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computer device can load a virtual machine based on a virtual image and / or other data that defines specific software for simulation (e.g., an operating system, a dedicated application, a server). As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computer devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computer device.

[0052] See Figure 2 , which is a flow chart of a video subtitle removal method according to an embodiment of the present application. The video subtitle removal method according to the present application can be applied to a video subtitle removal client. It is understood that the flow chart in the embodiment of the present method is not used to limit the order of executing the steps. As can be seen from the figure, the video subtitle removal method provided in this embodiment includes:

[0053] Step S21: Acquire a current video frame from which subtitles are to be removed and a previous video frame of the current video frame from the video.

[0054] Specifically, each video frame in the video can be detected by the subtitle detection model, and when subtitles are detected in the video frame, the video frame with subtitles can be used as the current video frame from which subtitles are to be removed. In this way, after determining the current video frame from which subtitles are to be removed, the current video frame from which subtitles are to be removed and the previous video frame of the current video frame can be obtained from the video, so that these two video frames can be subsequently input into the subtitle removal model for subtitle removal processing.

[0055] In this embodiment, the subtitle detection model can be obtained by training a neural network model using training sample data, and the subtitle detection model can be used to detect whether subtitles exist in a video frame. In one embodiment, the subtitle detection model can also adopt an open source subtitle detection model in the prior art, such as an OCR recognition model.

[0056] It is understandable that after the first current video frame to be subtitled is acquired, the detection process will continue to be performed on other video frames in the video to find all the video frames that need to be processed for subtitle removal.

[0057] It should be noted that the subtitles in this embodiment can be subtitles in the traditional sense, or they can be bullet comments, or even watermarks.

[0058] Step S21: Input the current video frame and the previous video frame into a subtitle removal model to obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion area information corresponding to the optical flow map, and a repaired image. The occlusion area information is used to determine an area in which the subtitle pixels in the current video frame cannot be replaced using the optical flow map.

[0059] Specifically, the subtitle removal model may be trained based on a CNN (Convolutional Neural Network) structure. The subtitle removal model may include an encoder and a decoder.

[0060] In an exemplary embodiment, the subtitle removal model includes an encoder and a decoder, and the inputting the current video frame and the previous video frame into the subtitle removal model to obtain the subtitle position information contained in the current video frame, the optical flow map of the current video frame relative to the previous video frame, the occlusion area information corresponding to the optical flow map, and the repaired image includes: inputting the current video frame and the previous video frame into the subtitle removal model, encoding the current video frame and the previous video frame by the encoder to obtain a feature map, and decoding the feature map by the decoder to obtain the subtitle position information contained in the current video frame, the optical flow map of the current video frame relative to the previous video frame, the occlusion area information corresponding to the optical flow map, and the repaired image, wherein the decoder includes a convGRU neural network structure.

[0061] The CNN encoder can encode the current video frame and the previous video frame to generate a hidden feature code, also known as a feature map. In one embodiment, the pixel values ​​of three channels (a total of six channels) in the current video frame and the previous video frame can be input into the CNN encoder for encoding, resulting in a 512x32x32 feature map.

[0062] Among them, the feature map can be decoded and restored by the CNN decoder to restore the feature map into subtitle position information, the optical flow map of the current video frame relative to the previous video frame, the occlusion area information corresponding to the optical flow map and the repaired image.

[0063] In this embodiment, the subtitle position information is the predicted output information obtained after the subtitle removal model processes the current video frame and the previous video frame. The subtitle position information can use a subtitle mask to represent the pixel-level position of the subtitle. For example, if a pixel is a subtitle pixel, it can be represented by "1". If a pixel is not a subtitle pixel, it can be represented by "0". In this way, when decoding the output subtitle position information, N masks can be output, and the value of N is the total number of pixels contained in the current video frame.

[0064] In this embodiment, the optical flow map is the predicted output information obtained after the subtitle removal model processes the current video frame and the previous video frame. The optical flow map is used to describe the change of each pixel in the current video frame relative to each pixel in the previous video frame of the current video frame, that is, whether each pixel in the current video frame has changed relative to the corresponding pixel in the previous video frame. In one embodiment, the optical flow map can also use a mask to represent the change of each pixel. For example, "1" is used to indicate that the pixel of the current video frame has changed relative to the pixel of the previous video frame, and "0" is used to indicate that the pixel of the current video frame has not changed relative to the pixel of the previous video frame. In this way, when decoding and outputting, an optical flow map composed of N masks can also be output, and the value of N is the total number of pixels contained in the current video frame.

[0065] Optical flow (or optic flow) is a concept used in detecting the motion of objects in the field of view. It describes the motion of an observed object, surface, or edge caused by its motion relative to the observer. Optical flow methods are very useful in pattern recognition, computer vision, and other image processing fields. They can be used for motion detection, object segmentation, calculation of collision time and object expansion, motion-compensated coding, and stereoscopic measurement of object surfaces and edges.

[0066] In this embodiment, the occlusion area information is used to determine the area where the optical flow map cannot be used to replace the subtitle pixel points in the current video frame, that is, the pixel points in the area where the optical flow is occluded (also called changed) cannot be replaced by the pixel points of the previous video frame. In one embodiment, a mask can also be used to represent the pixel-level occlusion area. For example, "1" is used to indicate that the pixel points of the current video frame cannot be replaced by the optical flow map for the corresponding pixel points in the current video frame, and "0" is used to indicate that the pixel points of the current video frame can be replaced by the optical flow map for the corresponding pixel points in the current video frame. In this way, when decoding and outputting, occlusion area information composed of N masks can also be output, and the value of N is the total number of pixels contained in the current video frame.

[0067] In this embodiment, the inpainted image is a predicted output image obtained by processing the current video frame and the previous video frame by the subtitle removal model. The inpainted image is an image used to inpaint the subtitle region. Specifically, when inpainting the subtitle region using the inpainted image, pixels in the subtitle region corresponding to the pixels in the inpainted image are used to replace the pixels in the subtitle region, thereby inpainting the subtitle region.

[0068] In an exemplary embodiment, in order to make the subtitle position information, optical flow map, occlusion area information and repaired image obtained by the decoder more accurate, the decoder includes a convGRU neural network structure.

[0069] The convGRU neural network structure is a modified version of the ConvLSTM, converting the LSTM into a GRU for computation. The ConvLSTM replaces the fully connected layers in the LSTM with convolutional kernels, converting the fully connected layers into local connections. This is a neural network structure based on Torch for computation, using GRU for comparison.

[0070] In this embodiment, by adding a convGRU neural network structure in the decoder, more temporal information can be retained, so that the image obtained by using the previous frame information for subtitle removal and repair processing operations is more accurate.

[0071] Step S22: performing subtitle removal processing on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame, and the previous video frame to obtain an image with subtitles removed.

[0072] Specifically, after obtaining the subtitle position information, the position of the subtitles contained in the current video frame can be determined by the subtitle position information, and then the position pixels of the subtitles can be replaced for the first time according to the optical flow map to obtain the replaced image. After that, the target area belonging to the occlusion area in the position area corresponding to the subtitles can be determined according to the occlusion area information. Finally, the subtitle position pixels corresponding to the target area are replaced again according to the repaired image to obtain the final image without the subtitles.

[0073] In an exemplary embodiment, see Figure 3 , performing subtitle removal processing on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame, and the previous video frame to obtain an image with subtitles removed includes:

[0074] Step S30 : determining subtitle pixels contained in the current video frame according to the subtitle position information and the current video frame.

[0075] Specifically, since the subtitle position information includes the pixel-level positions of subtitle pixels, it is possible to determine which pixels in the current video frame belong to subtitle pixels based on the subtitle position information.

[0076] For example, the subtitle position information is [1, 0, 0, 1, 1, 0], which indicates that the first pixel, the fourth pixel, and the fifth pixel in the current video frame are subtitle pixels.

[0077] Step S31 : replacing subtitle pixels contained in the current video frame according to the optical flow map and the previous video frame to obtain a replaced image.

[0078] Specifically, since the optical flow map records which specific pixels in the current video frame have changed, after obtaining the optical flow map, we can first find out the optical flow information corresponding to the subtitle pixel points contained in the current video frame, and then determine whether it is necessary to obtain the corresponding pixel points from the previous video frame based on the optical flow information. For example, if it is determined based on the found optical flow information that subtitle pixel point a and subtitle pixel point b in the subtitle pixel points have not changed, then two subtitle pixel points corresponding to the subtitle pixel point a and subtitle pixel point b can be obtained from the previous video frame to replace the subtitle pixel point a and subtitle pixel point b, thereby obtaining the replaced image.

[0079] Step S32 : generating the subtitle-removed image according to the occlusion area information, the repaired image, and the replaced image.

[0080] Specifically, since the occlusion area information records which specific pixel points cannot be replaced by the optical flow map, after determining the target pixel points that cannot be replaced by the optical flow map, the pixel points in the repaired image can be used to replace the target pixel points, thereby generating the image with subtitles removed.

[0081] In an exemplary embodiment, see Figure 4 The step of generating the subtitle-removed image according to the occlusion area information, the repaired image, and the replaced image includes:

[0082] Step S40: determining replacement pixels according to the occlusion area information and the repaired image.

[0083] Step S41 : replacing the pixel points corresponding to the replacement pixel points in the replaced image with the replacement pixel points to generate the subtitle-removed image.

[0084] Specifically, since the occlusion area information records which specific subtitle pixel points cannot be replaced by the optical flow map in the current video frame, for example, it records that the subtitle pixel point c cannot be replaced by the optical flow map in the current video frame, a replacement pixel point C corresponding to the subtitle pixel point c will be obtained from the repaired image, and then the replacement pixel point C will replace the subtitle pixel point c, thereby generating the image with the subtitles removed.

[0085] In an exemplary embodiment, see Figure 5 , the subtitle removal model is pre-trained as follows:

[0086] Step S50 : Acquire a plurality of training sample images, wherein each training sample image includes a first video frame and a previous video frame of the first video frame.

[0087] Specifically, a training sample image may be obtained by performing a subtitle adding operation on a video frame in a video without subtitles.

[0088] In an exemplary embodiment, see Figure 6 Each training sample image is obtained as follows: Step S60, obtaining a second video frame and a previous video frame of the second video frame from a video without subtitles; Step S61, adding subtitles to the second video frame to obtain the first video frame; Step S62, using the first video frame and the previous video frame of the second video frame as a training sample image.

[0089] Specifically, for each training sample image, the second video frame and the previous video frame of the second video frame are taken from the video without subtitles, wherein the second video frame is any video frame other than the first frame in the video without subtitles. After obtaining the second video frame, subtitles can be added to any area in the video frame, and the second video frame with subtitles added can be used as the first video frame. After the second video frame is processed for adding subtitles, the first video frame and the previous video frame of the second video frame can be used as a training sample image. In one embodiment, before the first video frame and the previous video frame of the second video frame are used as a training sample image, the previous video frame of the second video frame can also be processed for adding subtitles, and then the previous video frame of the second video frame with subtitles added and the first video frame can be used as a training sample image.

[0090] In an exemplary embodiment, in order to improve the diversity of training samples, subtitles of various fonts and sizes may be added to various positions in the second video frame when generating training sample images.

[0091] Step S51 : generating training labels for the plurality of training sample images according to the subtitle adding positions in the first video frame of the plurality of training sample images and the original image corresponding to the first video frame.

[0092] Specifically, the original image is an image of the first video frame in the training sample image before subtitles are added, that is, the second video frame.

[0093] In this embodiment, when adding subtitles to the second video frame, the subtitle adding position is recorded.

[0094] When generating training labels, a first training label is generated according to the position where the subtitles are added, and a second training label is generated according to the original image. The first training label and the second training label constitute the training labels of the training sample images.

[0095] In one embodiment, a first training label in the form of a mask may be generated according to the location where the subtitles are added; and all pixels of the original image may be used as the second training label.

[0096] Step S52: training a preset neural network model based on a plurality of training sample images and their respective training labels to obtain the subtitle removal model.

[0097] Specifically, during the training process, the subtitle position information, optical flow map, and occlusion area information can be defined as classification problems for prediction output.

[0098] In this embodiment, during training, cross entropy may be used as a loss function to ensure that the end-to-end training results are as close to the original image as possible.

[0099] In this embodiment, the image is encoded by an encoder in a pre-trained subtitle removal model to obtain a feature map, and then the feature map is decoded by a decoder to predict the subtitle position information, optical flow map, occlusion area information, and repaired image of the current video frame from which the subtitles are to be removed. Then, the image after subtitle removal can be synthesized based on the subtitle position information, optical flow map, occlusion area information, repaired image, and the current video frame from which the subtitles are to be removed. Since the subtitle removal model trained in an end-to-end manner in this application can accurately predict the subtitle position information, optical flow map, occlusion area information, and repaired image, when performing pixel replacement and filling, the image after subtitle removal synthesized based on the subtitle position information, optical flow map, occlusion area information, repaired image, and the current video frame from which the subtitles are to be removed can be closer to the original video frame, thereby improving the user experience.

[0100] See Figure 7, which is a program module diagram of an embodiment of a video subtitle removal device 70 of the present application.

[0101] In this embodiment, the video subtitle removal device 70 includes a series of computer program instructions stored in a memory. When executed by a processor, the video subtitle removal functions of various embodiments of the present application can be implemented. In some embodiments, based on the specific operations implemented by each part of the computer program instructions, the video subtitle removal device 70 can be divided into one or more modules. Specifically, the modules that can be divided are as follows:

[0102] An acquisition module 71 is configured to acquire a current video frame from which subtitles are to be removed and a previous video frame of the current video frame from which subtitles are to be removed.

[0103] An input module 72 is configured to input the current video frame and the previous video frame into a subtitle removal model to obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion region information corresponding to the optical flow map, and a repaired image, wherein the occlusion region information is used to determine regions in the current video frame where subtitle pixels cannot be replaced using the optical flow map;

[0104] The subtitle processing module 73 is configured to perform subtitle removal processing on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame and the previous video frame to obtain an image with subtitles removed.

[0105] In an exemplary embodiment, the subtitle processing module 73 is further configured to determine subtitle pixels included in the current video frame based on the subtitle position information and the current video frame; replace the subtitle pixels included in the current video frame based on the optical flow map and the previous video frame to obtain a replaced image; and generate the subtitle-removed image based on the occlusion area information, the repaired image, and the replaced image.

[0106] In an exemplary embodiment, the subtitle processing module 73 is further configured to determine a replacement pixel point based on the occlusion area information and the repaired image; and replace the pixel point corresponding to the replacement pixel point in the replaced image with the replacement pixel point to generate the subtitle-removed image.

[0107] In an exemplary embodiment, the video subtitle removal apparatus 70 further includes a training module.

[0108] The training module is used to obtain multiple training sample images, wherein each training sample image includes a first video frame and a previous video frame of the first video frame; generate training labels for each of the multiple training sample images based on the subtitle addition position in the first video frame among the multiple training sample images and the original image corresponding to the first video frame; and train a preset neural network model based on the multiple training sample images and their respective training labels to obtain the subtitle removal model.

[0109] In an exemplary embodiment, the training module is further used to obtain a second video frame and a previous video frame of the second video frame from a video without subtitles; add subtitles to the second video frame to obtain the first video frame; and use the first video frame and the previous video frame of the second video frame as a training sample image.

[0110] In an exemplary embodiment, the acquisition module 71 is further configured to detect each frame of the video using a subtitle detection model, and when subtitles are detected, obtain the current video frame from which the subtitles are to be removed and the previous video frame of the current video frame from the video.

[0111] In an exemplary embodiment, the input module 72 is further used to input the current video frame and the previous video frame into the subtitle removal model, encode the current video frame and the previous video frame by the encoder to obtain a feature map, and decode the feature map by the decoder to obtain the subtitle position information contained in the current video frame, the optical flow map of the current video frame relative to the previous video frame, the occlusion area information corresponding to the optical flow map, and the repaired image, wherein the decoder includes a convGRU neural network structure.

[0112] Figure 8 The hardware architecture diagram of a computer device 8 suitable for implementing the video subtitle removal method according to an embodiment of the present application is schematically shown. In this embodiment, the computer device 8 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server or a server cluster composed of multiple servers). Figure 8 As shown, the computer device 8 includes at least but is not limited to: a memory 111, a processor 121, and a network interface 122 that can be interconnected via a system bus.

[0113] The memory 111 includes at least one type of computer-readable storage medium, which can be either volatile or non-volatile. Specifically, the computer-readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the memory 111 can be an internal storage module of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 111 can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Of course, the memory 111 can also include both the internal storage module of the computer device 8 and its external storage device. In this embodiment, the memory 111 is generally used to store the operating system and various application software installed on the computer device 8, such as the program code of the video subtitle removal method, etc. In addition, the memory 111 can also be used to temporarily store various data that has been output or is about to be output.

[0114] In some embodiments, the processor 121 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other video subtitle removal chip. The processor 121 is typically used to control the overall operation of the computer device 8, such as performing control and processing related to data exchange or communication with the computer device 8. In this embodiment, the processor 121 is used to execute program code stored in the memory 111 or process data.

[0115] The network interface 122 may include a wireless network interface or a wired network interface. The network interface 122 is generally used to establish a communication link between the computer device 8 and other computer devices. For example, the network interface 122 is used to connect the computer device 8 to an external terminal via a network, establishing a data transmission channel and a communication link between the computer device 8 and the external terminal. The network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0116] It should be pointed out that Figure 8 Only a computer device having components 111 - 122 is shown, but it should be understood that implementation of all of the shown components is not a requirement, and more or fewer components may alternatively be implemented.

[0117] In this embodiment, the video subtitle removal method stored in the memory 111 can be divided into one or more program modules and executed by one or more processors (processor 121 in this embodiment) to complete the present application.

[0118] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the video subtitle removal method of the embodiment are implemented.

[0119] In this embodiment, the computer-readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped with the computer device, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store the operating system and various application software installed on the computer device, such as the program code of the video subtitle removal method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are about to be output.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across at least two network units. Some or all of the modules may be selected based on actual needs to achieve the purpose of the embodiments of the present application. Those of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0121] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the above embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for removing subtitles from a video, characterized in that: include: Obtaining a current video frame from which subtitles are to be removed and a previous video frame of the current video frame from the video; Inputting the current video frame and the previous video frame into a subtitle removal model to obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion area information corresponding to the optical flow map, and a repaired image, wherein the occlusion area information is used to determine areas in the current video frame where subtitle pixels cannot be replaced using the optical flow map; Subtitle removal processing is performed on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame and the previous video frame to obtain an image with subtitles removed.

2. The video subtitle removal method according to claim 1, wherein: The performing subtitle removal processing on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame, and the previous video frame to obtain an image with subtitles removed includes: Determining subtitle pixels contained in the current video frame according to subtitle position information and the current video frame; replacing subtitle pixels contained in the current video frame according to the optical flow map and the previous video frame to obtain a replaced image; The subtitle-removed image is generated according to the occlusion area information, the repaired image, and the replaced image.

3. The video subtitle removal method according to claim 2, wherein: Generating the subtitle-removed image according to the occlusion area information, the repaired image, and the replaced image includes: Determining a replacement pixel point according to the occluded area information and the repaired image; The pixel points corresponding to the replacement pixel points in the replaced image are replaced with the replacement pixel points to generate the subtitle-removed image.

4. The video subtitle removal method according to claim 1, wherein: The subtitle removal model is pre-trained as follows: Acquire a plurality of training sample images, wherein each training sample image includes a first video frame and a video frame preceding the first video frame; generating training labels for the plurality of training sample images according to the subtitle addition position in the first video frame of the plurality of training sample images and the original image corresponding to the first video frame; A preset neural network model is trained based on a plurality of training sample images and respective training labels to obtain the subtitle removal model.

5. The video subtitle removal method according to claim 4, characterized in that: Each training sample image is obtained as follows: Obtaining a second video frame and a previous video frame of the second video frame from a video without subtitles; adding subtitles to the second video frame to obtain the first video frame; The first video frame and the previous video frame of the second video frame are used as a training sample image.

6. The video subtitle removal method according to claim 1, wherein: The step of obtaining a current video frame from which subtitles are to be removed and a previous video frame of the current video frame comprises: A subtitle detection model is used to detect each video frame of the video, and when subtitles are detected, a current video frame from which subtitles are to be removed and a previous video frame of the current video frame are obtained from the video.

7. The video subtitle removal method according to any one of claims 1 to 6, characterized in that: The subtitle removal model includes an encoder and a decoder. Inputting the current video frame and the previous video frame into the subtitle removal model to obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion area information corresponding to the optical flow map, and a repaired image includes: The current video frame and the previous video frame are input into a subtitle removal model, the current video frame and the previous video frame are encoded by the encoder to obtain a feature map, and the feature map is decoded by the decoder to obtain the subtitle position information contained in the current video frame, the optical flow map of the current video frame relative to the previous video frame, the occlusion area information corresponding to the optical flow map, and the repaired image, wherein the decoder contains a convGRU neural network structure.

8. A video subtitle removal device, characterized in that: include: An acquisition module is used to acquire a current video frame from which subtitles are to be removed and a previous video frame of the current video frame from which subtitles are to be removed. An input module, configured to input the current video frame and the previous video frame into a subtitle removal model, and obtain subtitle position information contained in the current video frame, an optical flow map of the current video frame relative to the previous video frame, occlusion area information corresponding to the optical flow map, and a repaired image, wherein the occlusion area information is used to determine areas in the current video frame where subtitle pixels cannot be replaced using the optical flow map; The subtitle processing module is used to perform subtitle removal processing on the current video frame according to the subtitle position information, the optical flow map, the occlusion area information, the repaired image, the current video frame and the previous video frame to obtain an image with subtitles removed.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images