Video subtitle removing method, chip, GPU and electronic device

Abstract

This disclosure relates to the field of computer technology, and in particular to a video subtitle removal method, chip, GPU, and electronic device. The method, executed on a graphics processing unit (GPU), includes: receiving an input video and decoding the input video into a frame sequence; detecting subtitle regions in the current frame of the frame sequence and generating a corresponding subtitle mask; responding to the presence of subtitles in the current frame, predicting the content of the subtitle region based on the subtitle mask and utilizing the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence; and filling the subtitle region with the predicted content to generate a video frame with subtitles removed. According to embodiments of this disclosure, by deploying a series of tasks such as video decoding, subtitle region detection, spatiotemporal correlation-based content prediction, and final filling onto the GPU, the massively parallel computing capabilities are utilized to accelerate the processing flow and improve the visual naturalness and spatiotemporal coherence of the repaired video.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for removing video subtitles, a chip, a GPU, and an electronic device. Background Technology

[0002] Subtitles are widely used as a crucial information carrier in the production, dissemination, and secondary creation of video content. However, some videos use "hard subtitles" or "burned subtitles," meaning the subtitles are permanently embedded in the image frames during the video production stage and cannot be individually hidden or extracted via toggle options. In certain application scenarios, such as video content re-encoding, translation, or accessibility, it is necessary to remove these hard subtitles from the video frame and naturally restore the background areas obscured by the subtitles to restore the integrity and visual appeal of the image.

[0003] However, existing subtitle restoration techniques often result in visual discontinuities or inconsistencies between the restored area and the surrounding image or preceding and following video frames. Furthermore, the execution of these techniques may rely on general-purpose computing architectures, which may face challenges in processing efficiency and throughput when handling high-resolution or long videos, making them unsuitable for applications requiring high processing speeds. Summary of the Invention

[0004] In view of this, this disclosure proposes a video caption removal technology solution.

[0005] According to one aspect of this disclosure, a video caption removal method is provided, the method being executed on a graphics processing unit (GPU), comprising:

[0006] Receive input video and decode the input video into a frame sequence;

[0007] The current frame in the frame sequence is used to detect subtitle regions and generate the corresponding subtitle mask;

[0008] In response to the presence of subtitles in the current frame, the content of the subtitle region is predicted based on the subtitle mask and the spatiotemporal correlation of the preceding and / or subsequent frames in the frame sequence.

[0009] The predicted content is filled into the subtitle area to generate a video frame with the subtitles removed.

[0010] In one possible implementation, the step of detecting subtitle regions in the current frame of the frame sequence and generating a corresponding subtitle mask includes:

[0011] Identify the subtitle region in the current frame;

[0012] Based on the subtitle region, a single-frame subtitle mask file corresponding to the current frame is generated, wherein the subtitle region is indicated by numerical values ​​in the single-frame subtitle mask file;

[0013] The values ​​in the single-frame subtitle mask files of consecutive video frames are accumulated to generate a global subtitle mask file, which is used to count the areas where subtitles appear multiple times.

[0014] In one possible implementation, the subtitle region detection of the current frame in the frame sequence includes:

[0015] The regions in the global subtitle mask file whose cumulative value exceeds the frequency threshold are identified as video subtitle regions to be removed.

[0016] In one possible implementation, predicting the content of the subtitle region based on the subtitle mask and utilizing the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence includes:

[0017] Determine the similarity between the current frame and at least one neighboring frame without subtitles;

[0018] In the case of a target frame without subtitles where the similarity is greater than the first threshold, the image of the subtitle region is predicted based on the target frame without subtitles to obtain the content of the predicted subtitle region.

[0019] In one possible implementation, the step of predicting the content of the subtitle region based on the target subtitle-free frame to obtain the predicted subtitle region content includes:

[0020] For the pixel blocks to be filled in the subtitle region, the corresponding candidate image blocks are extracted from each target subtitle-free frame;

[0021] Calculate the feature similarity between the surrounding region of the pixel block to be filled and the surrounding region of each candidate image block;

[0022] Based on the feature similarity, a contribution weight is assigned to each candidate image patch, and the contribution weight is positively correlated with the feature similarity;

[0023] Based on the contribution weights, the features of multiple candidate image blocks are weighted and fused to generate the predicted content of the pixel block to be filled.

[0024] In one possible implementation, extracting corresponding candidate image blocks from each target untitled frame includes:

[0025] Based on the optical flow field between the current frame and the target no-subtitle frame, the corresponding position of the pixel block to be filled in the target no-subtitle frame is determined;

[0026] The corresponding candidate image block is extracted with the corresponding position as the center.

[0027] In one possible implementation, based on the subtitle mask, the content of the subtitle region is predicted using the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence, including:

[0028] If subtitles are present in the preceding and / or subsequent frames, obtain local image blocks of the current frame and the preceding and / or subsequent frames in the subtitle area;

[0029] Based on the temporal positional changes of the local image patch, motion estimation is performed on the local image patch to determine the independent motion vector of the subtitle itself;

[0030] Based on the independent motion vector of the subtitle itself, the global optical flow field of the frame sequence is used to determine the background optical flow field without subtitles;

[0031] Using the background light flow field without subtitles, the background content in the preceding and / or subsequent frames is transformed to the perspective of the current frame to generate predicted content.

[0032] In one possible implementation, the method further includes:

[0033] Based on the numerical accuracy requirements of different calculation tasks in the subtitle removal process, different numerical calculation formats are dynamically assigned to different tasks.

[0034] In one possible implementation, based on the numerical precision requirements of different computational tasks in the subtitle removal process, different numerical calculation formats are dynamically assigned to different tasks, including:

[0035] A first precision format is assigned to the optical flow estimation task, the first precision format including: single-precision floating-point format FP32;

[0036] A second precision format is assigned to the image restoration task, the second precision format including: half-precision floating-point format FP16;

[0037] A third-precision format is assigned to the subtitle recognition task, and the third-precision format includes: integer format INT8;

[0038] The first precision format has a higher number of bits for calculation than the second precision format, and the second precision format has a higher number of bits for calculation than the third precision format.

[0039] According to another aspect of this disclosure, a video caption removal chip is provided, comprising:

[0040] The video processing module is used to receive input video and decode it into a frame sequence;

[0041] The subtitle recognition module is used to detect the subtitle region of the current frame in the frame sequence and generate the corresponding subtitle mask;

[0042] The image processing module is configured to, in response to the presence of subtitles in the current frame, predict the content of the subtitle region based on the subtitle mask and the spatiotemporal correlation of the preceding and / or subsequent frames in the frame sequence.

[0043] The image synthesis module fills the predicted content into the subtitle area to generate a video frame with the subtitles removed.

[0044] A hardware task scheduler is used to schedule the video processing module, the subtitle recognition module, the image processing module, and the image synthesis module to work together.

[0045] According to another aspect of this disclosure, a graphics processing unit (GPU) is provided, the GPU comprising the chip described above.

[0046] According to another aspect of this disclosure, an electronic device is provided, the electronic device including the chip described above.

[0047] In this embodiment, the method is executed on a graphics processing unit (GPU) and includes: receiving an input video and decoding the input video into a frame sequence; detecting subtitle regions in the current frame of the frame sequence and generating a corresponding subtitle mask; responding to the presence of subtitles in the current frame, predicting the content of the subtitle region based on the subtitle mask and utilizing the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence; and filling the subtitle region with the predicted content to generate a video frame with subtitles removed. Thus, by utilizing the parallel computing architecture of the GPU to execute the entire decoding, detection, and filling process, the overall throughput of video processing is improved. Simultaneously, by using the generated subtitle mask and comprehensively utilizing the spatiotemporal correlation of preceding and subsequent frames in the video stream for content prediction and filling, the repaired background content can maintain higher coherence and consistency with the surrounding image and the preceding and following time sequence, thereby effectively improving the visual naturalness and spatiotemporal coherence of the repaired video image while removing hard subtitles.

[0048] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0049] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0050] Figure 1 A flowchart illustrating a video caption removal method according to an embodiment of the present disclosure is shown.

[0051] Figure 2A schematic diagram of a frame sequence provided according to an embodiment of the present disclosure is shown.

[0052] Figure 3 A schematic diagram of a frame sequence provided according to an embodiment of the present disclosure is shown.

[0053] Figure 4 A flowchart illustrating a video caption removal method according to an embodiment of the present disclosure is shown.

[0054] Figure 5 A structural diagram of a video subtitle processing chip provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0055] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0056] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0057] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0058] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0059] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0060] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0061] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0062] In the production, post-processing, and re-creation of video content, the removal of hard captions (or burned captions) is one of the key steps in restoring image clarity, enabling multilingual versions, or stylized adaptations. Among related technologies, there are video restoration or content filling solutions based on software algorithms executed by general-purpose computing processors (such as CPUs), the core of which is running a pure software process on the host machine. This architecture may face limitations in both performance and effectiveness when handling video hard caption removal tasks.

[0063] First, the architecture of general-purpose processors is not optimized for high-throughput pixel-level video restoration and complex spatiotemporal prediction tasks. When processing high-resolution, high-frame-rate video streams, the entire computational process from decoding, region detection, and content prediction and filling consumes a significant amount of host computing power, which may lead to processing delays and affect system responsiveness.

[0064] Secondly, these software solutions may not fully adapt to the specific needs of subtitle removal scenarios in terms of repair strategies. For example, when predicting the background content obscured by subtitles, if the rich information contained in the preceding and following frames of the video stream in the temporal dimension is not systematically and efficiently utilized, the generated filling content may not blend naturally with the surrounding pixels in space, or there may be inconsistencies such as jumps and flickering with adjacent frames in time, affecting the visual experience and spatiotemporal coherence of the repaired content.

[0065] To improve the processing efficiency and restoration quality of video hard caption removal, this disclosure proposes a hardware-software co-processing method based on a graphics processing unit (GPU). By deploying a series of tasks, including video decoding, caption region detection, spatiotemporal correlation-based content prediction, and final filling, onto the GPU, the processing flow is accelerated by leveraging its massively parallel computing capabilities.

[0066] By utilizing the parallel computing architecture of GPUs to perform the entire process of decoding, detection, and filling, the overall throughput of video processing is improved. At the same time, by using the generated subtitle mask and comprehensively utilizing the spatiotemporal correlation between preceding and subsequent frames in the video stream for content prediction and filling, the restored background content can maintain a higher degree of coherence and consistency with the surrounding scenes and the preceding and following time sequences. Thus, while removing hard subtitles, the visual naturalness and spatiotemporal coherence of the restored video are effectively improved.

[0067] The following section, in conjunction with the accompanying drawings, will provide a detailed description of the process of the video subtitle removal method provided in this disclosure, the collaborative relationship between each step, the content prediction mechanism based on spatiotemporal correlation, and its various possible implementation methods.

[0068] Figure 1 A flowchart illustrating a video caption removal method according to an embodiment of the present disclosure is shown. The method is executed on a graphics processing unit (GPU), such as... Figure 1 As shown, the method includes:

[0069] In step S11, the input video is received and decoded into a frame sequence;

[0070] Graphics processing units (GPUs), as hardware specifically designed for parallel computing, have an architecture containing a large number of computing cores, making them suitable for performing high-throughput image and video processing tasks. In this disclosure, the GPU is used to perform the entire computational process from video decoding to subtitle removal, thereby improving processing efficiency. Specific implementations are detailed in this disclosure and will not be elaborated upon here.

[0071] Input video can originate from various storage media or transmission links, such as video files stored on local or network storage devices, or video streams received in real time via a data interface. This video data is typically encapsulated in specific encoding and container formats to reduce storage space and transmission bandwidth usage.

[0072] During video decoding, the compressed input video is decoded into a series of consecutive, uncompressed image frames. Specifically, the compressed pixel information is reconstructed into a complete image based on the corresponding video coding standard algorithm. The decoding operation can be performed by dedicated hardware circuitry in the GPU or by a decoder running on the GPU.

[0073] A frame sequence represents multiple still images arranged chronologically after decoding. Each image frame represents the content of the video at a specific moment. Figure 2 This diagram illustrates a frame sequence provided according to an embodiment of the present disclosure. In a frame sequence containing n (n is a positive integer) video frames, there may be frames containing subtitles, for example... Figure 2 Frames 3, 4, n-3, and n-2 all contain subtitles.

[0074] Figure 3 A schematic diagram illustrating a frame sequence provided according to an embodiment of this disclosure is shown. For example... Figure 3 As shown, video frame 1 and video frame 2 both contain subtitles, while video frame 3 does not contain subtitles.

[0075] The decoding operation can be performed by the video processing module within the GPU, which can receive input video from external systems or storage devices. In some implementations, the video processing module has a built-in or associated video decoder that performs the operation of decoding the input video into a sequence of frames.

[0076] The video processing module can identify the container format of the input video, parse and separate the video stream, and then call the corresponding decoding algorithm to decode the video stream. The decoding algorithm can be matched with the encoding standard used by the input video.

[0077] To improve processing efficiency, video processing modules may use dedicated hardware decoders to accelerate the decoding process. For example, in a GPU, the integrated hardware decoding engine can be used to decode video streams of specific formats, significantly reducing the load on the host processor and increasing decoding speed. The resulting frame sequence can then be placed in a designated memory region, such as the GPU's video memory, for subsequent processing.

[0078] In step S12, subtitle region detection is performed on the current frame in the frame sequence, and a corresponding subtitle mask is generated;

[0079] The caption region is the visual area in an image frame that contains text information. The caption region often exhibits different characteristics from the image background, such as a regular line arrangement, high edge density, or color and brightness that contrast sharply with the background. During the detection process, the specific location of these caption regions in the image coordinate system is determined.

[0080] A caption mask is a data layer that matches the size of the current frame's image. Its function is to quantitatively identify which parts of the image are designated as caption regions. Each location in the mask (typically corresponding to one pixel) stores a numerical value indicating whether that location belongs to the caption region to be processed. For example, binary values ​​can be used, with "1" representing caption pixels and "0" representing non-caption pixels. The caption mask provides clear spatial positioning information for subsequent image inpainting operations.

[0081] When performing subtitle region detection, the pixel information of the current frame can be extracted first and input into a pre-configured detection model. The internal calculation process of the model may involve convolution operations to extract multi-level features, and regression or classification can be used to determine the possible location and boundaries of the text. After detection, based on the obtained bounding box coordinates or subtitle pixels, preset identifier values ​​are filled into the corresponding positions in the mask data structure to complete the construction of the subtitle mask. For example, if the detection model determines that there is a horizontal rectangular area at the bottom of the image containing subtitles, then the corresponding values ​​in the mask matrix for all pixel coordinates covered by the rectangle are set to 1; in another example, if the detection model directly outputs the subtitle pixels, then the corresponding pixel values ​​in the mask matrix are directly set to 1.

[0082] The generated subtitle mask can be stored as a single-frame subtitle mask file. This file corresponds to the current frame, and the values ​​in the file indicate the position of the subtitle pixels. For example, for a 1080P resolution video frame, if the subtitle recognition module determines that there are subtitles within the pixel range of lines 540 to 600, then the corresponding line position in the mask file is marked as 1, and the remaining area is marked as 0.

[0083] In step S13, in response to the presence of subtitles in the current frame, the content of the subtitle region is predicted based on the subtitle mask and the spatiotemporal correlation of the preceding and / or subsequent frames in the frame sequence.

[0084] The preceding and subsequent frames represent images that are earlier or later than the current frame on the timeline, respectively. Spatiotemporal correlation represents the inherent connections and patterns in the time and spatial dimensions of a video sequence, such as the continuity of background content between adjacent frames, the motion trajectory of objects, and the consistency of lighting.

[0085] When it is determined that there are subtitles in the current frame, the content of the obscured area can be predicted based on the position indicated by the generated subtitle mask. Specifically, by analyzing the preceding and following frames that are temporally adjacent to the current frame, and based on the changes in pixel information at the same or adjacent spatial locations at different times, the visual content that should exist in the area covered by the subtitles at the current position can be inferred.

[0086] In some implementations, motion information between the current frame and adjacent frames can be calculated, for example, by using optical flow estimation to obtain the displacement vector of a pixel from one frame to another. Using this motion information, the corresponding background areas in adjacent frames that are not obscured by the subtitles can be aligned or distorted to the viewpoint of the current frame, thereby providing candidate pixel sources for the subtitle area.

[0087] For example, if it is detected that the position of the subtitle area is fixed in several consecutive frames, but the clouds in the background are moving slowly, the cloud texture located below the subtitle area in the previous frame can be calculated and mapped to the corresponding position in the current frame through the optical flow field, as part of the predicted content.

[0088] In other implementations, the prediction process can involve more complex feature matching and fusion. For example, image patches that spatially correspond to or semantically relate to the caption region in the current frame can be extracted from previous and subsequent frames as candidates. Then, by calculating the feature similarity between these candidate image patches and the context region of the current frame, appropriate weights are assigned to different candidates, and a naturally transitioning prediction result is finally generated through weighted fusion. For instance, for a small patch of grass covered by the caption, the texture of the grass blades might be borrowed from one frame, and the correct lighting and shadows might be borrowed from another frame, and a clear and lighting-appropriate fill pattern could be generated through fusion.

[0089] By comprehensively utilizing the bidirectional temporal context information provided by preceding and subsequent frames, the predicted content can better maintain the spatial texture continuity with the surrounding area and the logical consistency with the temporal movement of the preceding and following images, thereby providing higher quality source data for subsequent filling operations.

[0090] In step S14, the predicted content is filled into the subtitle area to generate a video frame with the subtitles removed.

[0091] The predicted content can be viewed as image data of the area obscured by the caption. This content may include pixel color values, texture blocks, or complete image fragments, with the goal of visually conforming to the surrounding background. The caption region represents the area of ​​the image that needs to be repaired, precisely defined by the caption mask.

[0092] The padding operation can be performed based on the correspondence between the predicted content and the caption mask. The caption mask provides a map of the pixel locations that need to be modified. In some implementations, the predicted pixel values ​​can be directly assigned to the areas marked with specific values ​​in the caption mask, overwriting the original caption pixels. This approach is suitable when the boundaries between the predicted content and the surrounding background are clear and no additional processing is required.

[0093] To ensure a natural transition between the filled area and the surrounding image, image fusion techniques can be used for filling. For example, pixel blending can be performed at the boundary between the predicted content and the original background to smooth seams and eliminate noticeable edge artifacts. For instance, for a predicted patch of sky texture, linear interpolation blending can be used on its edge pixels when filling the caption area to smoothly blend it with the color gradient of the surrounding sky.

[0094] After the filling process is complete, the subtitle area is replaced by new background content, resulting in a video frame without subtitles. This video frame can be preserved or input into subsequent video encoding processes to synthesize a complete subtitle-free video sequence. Through this filling operation, the video frame removes hard subtitles while maintaining the integrity and visual realism of the image content.

[0095] In this embodiment, the method is executed on a graphics processing unit (GPU) and includes: receiving an input video and decoding the input video into a frame sequence; detecting subtitle regions in the current frame of the frame sequence and generating a corresponding subtitle mask; responding to the presence of subtitles in the current frame, predicting the content of the subtitle region based on the subtitle mask and utilizing the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence; and filling the subtitle region with the predicted content to generate a video frame with subtitles removed. Thus, by utilizing the parallel computing architecture of the GPU to execute the entire decoding, detection, and filling process, the overall throughput of video processing is improved. Simultaneously, by using the generated subtitle mask and comprehensively utilizing the spatiotemporal correlation of preceding and subsequent frames in the video stream for content prediction and filling, the repaired background content can maintain higher coherence and consistency with the surrounding image and the preceding and following time sequence, thereby effectively improving the visual naturalness and spatiotemporal coherence of the repaired video image while removing hard subtitles.

[0096] In one possible implementation, the step of detecting subtitle regions in the current frame of the frame sequence and generating a corresponding subtitle mask includes:

[0097] Identify the subtitle region in the current frame;

[0098] Based on the subtitle region, a single-frame subtitle mask file corresponding to the current frame is generated, wherein the subtitle region is indicated by numerical values ​​in the single-frame subtitle mask file;

[0099] The values ​​in the single-frame subtitle mask files of consecutive video frames are accumulated to generate a global subtitle mask file, which is used to count the areas where subtitles appear multiple times.

[0100] A single-frame caption mask file is a data file used to record the location information of caption regions in a single frame of an image. This file typically has the same spatial dimensions as the original image frame, and its internal structure uses a specific numerical distribution to indicate the location of the captions. In some implementations, the single-frame caption mask file can exist in the form of a matrix or an image.

[0101] For example, in the implementation of the binary mask, the matrix element corresponding to the detected subtitle area can be set to a first value (e.g., 1), while the matrix element corresponding to the non-subtitle area can be set to a second value (e.g., 0), so as to clearly distinguish the coverage area of ​​the subtitle through the difference in values.

[0102] A global caption mask file is a cumulative data file generated by statistically integrating caption mask information from multiple consecutive frames. This file maintains the same spatial dimension as the video frames, but the values ​​at each location no longer represent only the state of a single frame; instead, they record the frequency or cumulative intensity of the caption at that spatial location.

[0103] As mentioned earlier, subtitle regions in the current frame can be identified based on image analysis and feature extraction. After determining the specific coordinates of the subtitle regions, a single-frame subtitle mask file can be constructed. Specifically, based on the coordinate information of the subtitle regions, in a matrix initialized to default values ​​(such as all 0s) and the same size as the frame, the values ​​of the corresponding coordinates are modified to preset identifier values ​​(such as 1). This accurately records the pixel-level position of the subtitles in that frame.

[0104] After obtaining a series of consecutive single-frame caption mask files, a global caption mask file can be generated by summing the values ​​at corresponding positions in these files element by element.

[0105] For example, when processing a video segment of length N frames (N being a positive integer), the values ​​of the same coordinates in the N single-frame mask files are summed. After accumulation, in the global caption mask file, positions where captions appear repeatedly across multiple frames will accumulate higher values, while positions where text or noise only appears occasionally will have lower values. For instance, for a dialogue caption area that is fixed at the bottom of the video, its corresponding position in each frame's single-frame mask is 1. After accumulating over hundreds of frames, the value at that position in the global mask may approach 100; while the accumulated value of occasional floating text will be much smaller. This accumulation mechanism naturally strengthens the signal of stable caption areas and weakens the interference of temporary text.

[0106] In this embodiment, the system identifies subtitle regions in the current frame; generates a single-frame subtitle mask file corresponding to the current frame based on the subtitle regions, with numerical values ​​indicating the subtitle regions within the single-frame subtitle mask file; and accumulates the values ​​in the single-frame subtitle mask files of consecutive video frames to generate a global subtitle mask file, which is used to statistically analyze regions where subtitles appear multiple times. Thus, by establishing precise single-frame mask records for each frame and generating a global mask reflecting the frequency of subtitle occurrences for each segment, the system can distinguish between stable, fixed subtitles and occasional temporary text in the video, thereby improving the robustness and accuracy of subtitle region detection and providing more reliable region guidance for subsequent content prediction and filling.

[0107] In one possible implementation, the subtitle region detection of the current frame in the frame sequence includes:

[0108] The regions in the global subtitle mask file whose cumulative value exceeds the frequency threshold are identified as video subtitle regions to be removed.

[0109] Text often appears in video content, such as road signs and shop signs. This text is not subtitles but part of the video content itself and should not be removed.

[0110] Hard text that needs to be removed from a video usually displays continuously within a scene or time period, thus accumulating a high value in the global mask; while other text or noise that appears temporarily in the picture has a lower cumulative value.

[0111] Therefore, a frequency threshold can be used to distinguish between text that appears occasionally in video content and subtitles that exist stably. This threshold can be an integer, and its specific value can be set according to the video content, subtitle characteristics, or processing requirements.

[0112] Setting a frequency threshold provides a quantitative standard for extracting stable regions from global statistics. For example, the threshold can be set as a percentage of the total number of frames or as an absolute cumulative count. Regions where the cumulative value exceeds the frequency threshold mean that the frequency of the subtitle appearing at that spatial location has reached a preset salience level.

[0113] Specifically, each pixel location or pre-divided image block in the global mask can be traversed, and its cumulative value can be compared with a preset frequency threshold. When the cumulative value at a certain location is greater than the threshold, that location is included in the set of video subtitle regions to be removed. For example, for a 30-frame video clip, if the frequency threshold is set to 20, only pixel locations where subtitles are detected in more than 20 frames will be ultimately identified as "subtitle regions" that need to be repaired. In this way, temporary text, such as text that appears in the first 5 frames and disappears in the last 25 frames, can be excluded from the areas to be repaired.

[0114] In this embodiment, regions in the global subtitle mask file whose cumulative values ​​exceed a frequency threshold are identified as video subtitle regions to be removed. Therefore, by using a frequency threshold to filter global cumulative statistics, it is possible to effectively distinguish between persistent, fixed subtitle regions in the video and text that appears occasionally or temporarily in the video content. This improves the accuracy and stability of subtitle region determination, avoids unnecessary repair processing of temporary text or detection noise, and allows subsequent prediction and filling operations to more accurately focus on the truly hard subtitle regions that need to be removed.

[0115] In one possible implementation, predicting the content of the subtitle region based on the subtitle mask and utilizing the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence includes:

[0116] Determine the similarity between the current frame and at least one neighboring frame without subtitles;

[0117] In the case of a target frame without subtitles where the similarity is greater than the first threshold, the image of the subtitle region is predicted based on the target frame without subtitles to obtain the content of the predicted subtitle region.

[0118] At least one adjacent frame without subtitles can be either the preceding frame or the following frame of the current frame. It should be noted that at least one adjacent frame without subtitles can also be separated from the current frame by at least one frame with subtitles. Here, "adjacent" can mean that the number of frames between them is less than a frame number threshold, for example, less than 5 or 10 frames.

[0119] A caption-free frame represents an image frame in a frame sequence that does not contain captions. These frames retain a relatively clean background image that is not obscured by text. Similarity is used to quantify the degree of similarity in content between two image frames. This metric can be obtained by calculating the feature differences between the two frames in a specific region or globally.

[0120] The first threshold is used to determine whether the similarity between two frames reaches a standard sufficient for prediction. Target no-subtitle frames represent specific no-subtitle frames whose similarity to the current frame exceeds the first threshold. Target no-subtitle frames can be considered as having a high degree of inheritance or consistency with the current frame in content.

[0121] The current frame can be compared with one or more temporally adjacent frames without subtitles. When calculating similarity, the area marked by the subtitle mask can be avoided, and the similarity of visual features in the non-subtitle area can be calculated. For example, the mean square error of pixel values ​​in the non-subtitle area of ​​two frames can be calculated, the structural similarity index can be calculated, or the cosine distance between deep feature vectors extracted by a neural network can be compared to obtain a similarity score.

[0122] When the calculated similarity is greater than the first threshold, the corresponding subtitle-free frame is considered to share similar background content with the current frame, providing reliable reference information for the obscured subtitle area. For ease of description, subtitle-free frames with a similarity greater than the first threshold to the current frame are described as target subtitle-free frames.

[0123] After identifying the target frame without subtitles, image prediction of the subtitle region in the current frame can be performed based on that frame. Since the backgrounds of the two frames are highly similar, it can be assumed that the spatial position in the target frame without subtitles corresponds to the subtitle region in the current frame, representing the original background of the current frame before it was covered by subtitles. Therefore, the prediction process can involve directly or appropriately transforming the image patch at the corresponding position in the target frame without subtitles as the predicted content for the subtitle region of the current frame.

[0124] For example, if there is a subtitle at the bottom of the current frame, and the target subtitle-free frame is almost static, the pixel blocks of the same rectangular area at the bottom of the target subtitle-free frame can be directly used as the predicted fill content for that subtitle area in the current frame. If there is slight global motion, local alignment can be performed first based on the estimated optical flow, and then the corresponding image blocks can be extracted as the predicted fill content. For specific details, please refer to the possible implementations provided in this disclosure, which will not be elaborated here.

[0125] In this embodiment, the similarity between the current frame and at least one neighboring subtitle-free frame is determined. If a target subtitle-free frame with a similarity greater than a first threshold exists, the subtitle region image is predicted based on the target subtitle-free frame to obtain the predicted subtitle region content. Therefore, by introducing a similarity calculation and threshold comparison mechanism, subtitle-free reference frames with a high degree of consistency with the background of the current frame can be intelligently selected, and content prediction is performed based on these reference frames. This makes the prediction results more closely match the real background of the original scene, reducing the risk of content errors or inconsistencies introduced due to improper reference frame selection, thereby improving the accuracy and reliability of subtitle region content prediction.

[0126] In one possible implementation, the step of predicting the content of the subtitle region based on the target subtitle-free frame to obtain the predicted subtitle region content includes:

[0127] For the pixel blocks to be filled in the subtitle region, the corresponding candidate image blocks are extracted from each target subtitle-free frame;

[0128] Calculate the feature similarity between the pixel block to be filled and each of the candidate image blocks;

[0129] Based on the feature similarity, a contribution weight is assigned to each candidate image patch, and the contribution weight is positively correlated with the feature similarity;

[0130] Based on the contribution weights, the features of multiple candidate image blocks are weighted and fused to generate the predicted content of the pixel block to be filled.

[0131] The pixel block to be filled is a local image unit obtained after further subdividing the subtitle region, which needs to be used for content prediction. The pixel block to be filled can be a rectangular region or a set of pixels with a specific shape (such as the shape of the subtitle). The candidate image block is an image region extracted from each target subtitle-free frame that corresponds spatially to the pixel block to be filled in the current frame.

[0132] During the prediction process, the following operations can be performed on each pixel block to be filled:

[0133] First, based on the area defined by the subtitle mask, it is divided into several smaller processing units, namely pixel blocks to be filled. For each pixel block to be filled, in each target frame without subtitles, image regions with the same spatial location are located and extracted as candidate image blocks based on temporal correspondence and possible motion compensation.

[0134] Subsequently, to evaluate the suitability of each candidate image patch, its feature similarity can be calculated. Feature similarity quantifies the visual similarity between the surrounding region of the pixel patch to be filled (i.e., its neighboring non-caption regions) and the corresponding surrounding region of the candidate image patch. The calculation does not directly compare the occluded patch itself, but rather compares the features between the occluded neighboring regions of the pixel patch to be filled in the current frame and the neighboring regions of the candidate image patch at the same position in its frame. These features can include color histograms, texture descriptors, gradient information, or deep semantic features extracted through neural networks. A higher similarity value indicates a better match between the local background environment of the candidate patch and the current frame.

[0135] Based on the calculated feature similarities, a contribution weight is assigned to each candidate image patch. The contribution weight represents the importance coefficient assigned to each candidate image patch during the fusion process, and this coefficient determines the magnitude of the candidate patch's contribution to the final prediction result. The allocation principle can include: candidate patches with higher feature similarity receive greater contribution weights. For example, the similarity can be normalized using the softmax function, and the normalized value can be directly used as the weight. In this way, the candidate patch that best matches the current frame's background will dominate the fusion process.

[0136] Finally, weighted fusion is performed to generate the predicted content. The features of each candidate image patch (which can be the original pixel values ​​or some intermediate representation) are multiplied by their corresponding contribution weight. Then, all weighted features are summed to obtain the synthetic features of the pixel patch to be filled, which are then decoded into the final pixel-level predicted content. For example, if there are two candidate patches, one with a weight of 0.7 and the other with a weight of 0.3, approximately 70% of the visual information in the predicted content comes from the first candidate patch, and 30% comes from the second candidate patch, achieving optimal information fusion.

[0137] In this embodiment, for the pixel block to be filled in the subtitle region, corresponding candidate image blocks are extracted from each target frame without subtitles; the feature similarity between the surrounding area of ​​the pixel block to be filled and the surrounding area of ​​each candidate image block is calculated; based on the feature similarity, a contribution weight is assigned to each candidate image block, the contribution weight being positively correlated with the feature similarity; based on the contribution weight, the features of multiple candidate image blocks are weighted and fused to generate the predicted content of the pixel block to be filled. Thus, by extracting candidates from multiple qualified reference frames for each local unit to be repaired and performing intelligent weighted fusion based on the matching degree of its surrounding area, the advantages of multiple frame information can be comprehensively utilized, effectively mitigating potential local deviations or defects in a single frame reference, thereby generating more accurate, natural, and highly coordinated subtitle region predicted content, improving the overall quality and reliability of the repair effect.

[0138] In one possible implementation, extracting corresponding candidate image blocks from each target untitled frame includes:

[0139] Based on the optical flow field between the current frame and the target no-subtitle frame, the corresponding position of the pixel block to be filled in the target no-subtitle frame is determined;

[0140] The corresponding candidate image block is extracted with the corresponding position as the center.

[0141] Optical flow is a vector field that characterizes the direction and velocity of motion of each pixel in an image between two consecutive frames. This field consists of a series of two-dimensional vectors, each indicating the displacement of a pixel from one frame to another.

[0142] The corresponding position represents the coordinates of the physical scene point represented by the pixel block to be filled in the current frame, within the target subtitle-free frame. This corresponding position can be determined based on motion information obtained from the optical flow field. Specifically, for a pixel block to be filled within the subtitle region of the current frame, the coordinates of the center point (or its feature points) of that pixel block in the current frame are first obtained. Then, the pre-calculated optical flow field pointing from the current frame to the target subtitle-free frame is queried to find the optical flow vector corresponding to that coordinate point. This vector indicates the offset of the scene point from the current frame to the target subtitle-free frame. Adding the current frame coordinates to this optical flow vector calculates the theoretical position of that point in the target subtitle-free frame, i.e., the corresponding position. For example, if the center coordinates of a pixel block in the current frame are (x, y), and the corresponding optical flow vector is (Δx, Δy), then its corresponding position in the target subtitle-free frame is (x+Δx, y+Δy).

[0143] Position mapping using optical flow fields can effectively compensate for pixel displacement caused by camera or object motion between frames. This allows the extracted candidate image patches to more accurately correspond in content to the same physical background area that is occluded in the current frame, rather than just the same position in the image coordinate system. For example, if subtitles in the current frame occlude the front of a moving car, and the car has moved forward in the target frame without subtitles, optical flow fields can be used to track the new position of the car's front in the target frame, thus extracting candidate patches from the correct position.

[0144] After determining the precise corresponding location, a region of a specific size is drawn out in the target frame without subtitles, centered on this location, to extract the corresponding candidate image block. The size of the region can be the same as or slightly larger than the size of the pixel block to be filled. Extraction centered on the corresponding location ensures that the candidate block surrounds the tracked physical point in terms of content, thereby maximizing its value as a prediction reference.

[0145] In this embodiment, based on the optical flow field between the current frame and the target textless frame, the corresponding position of the pixel block to be filled in the target textless frame is determined; and the corresponding candidate image block is extracted with the corresponding position as the center. Thus, by utilizing the optical flow field for motion estimation and pixel position tracking, the physical scene point corresponding to the occluded area of ​​the current frame can be accurately located in the target textless frame, and candidate image blocks can be extracted with this point as the center. This ensures that the extracted candidate content maintains a strict correspondence in scene semantics, effectively overcoming the direct position extraction error caused by inter-frame motion, thereby providing more accurate and reliable input information for subsequent weighted fusion and improving the spatial accuracy of the predicted content.

[0146] In one possible implementation, based on the subtitle mask, the content of the subtitle region is predicted using the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence, including:

[0147] If subtitles are present in the preceding and / or subsequent frames, obtain local image blocks of the current frame and the preceding and / or subsequent frames in the subtitle area;

[0148] Based on the temporal positional changes of the local image patch, motion estimation is performed on the local image patch to determine the independent motion vector of the subtitle itself;

[0149] Based on the independent motion vector of the subtitle itself, the global optical flow field of the frame sequence is used to determine the background optical flow field without subtitles;

[0150] Using the background light flow field without subtitles, the background content in the preceding and / or subsequent frames is transformed to the perspective of the current frame to generate predicted content.

[0151] When analyzing consecutive frames and finding that the subtitle region persists, local image patches within the same region defined by the subtitle mask can be obtained from the current frame and the frames before and after it. These patches constitute an image sequence spanning several frames, focused on the subtitle-covered area. Local image patches represent small image units extracted from the subtitle region of multiple consecutive frames for analysis. These patches temporally correspond to the same physical area covered by the subtitle, but their specific image content may differ due to changes in the subtitle text or background movement.

[0152] Subsequently, by analyzing the temporal changes in appearance and positional shifts of these local image blocks, the potential motion patterns of the subtitle content (text) itself can be estimated, yielding the subtitle's independent motion vector. The subtitle's independent motion vector represents the displacement vector of the subtitle (such as scrolling subtitles) relative to the background image between consecutive frames.

[0153] For example, for horizontally scrolling subtitles, the text moves continuously from right to left. By using block matching, feature point tracking, or local optical flow calculation for that region, an independent motion vector representing the overall movement trend of the subtitle text can be determined.

[0154] After obtaining the independent motion vectors of the subtitles, the background optical flow field without subtitles can be determined by combining it with the pre-calculated inter-frame global optical flow field. The global optical flow field is a vector field calculated by analyzing the entire frame sequence, containing the comprehensive motion information of all moving objects in the scene (including the background and subtitles). The background optical flow field without subtitles is a pure motion field that reflects only the motion of the background scenery after removing or compensating for the independent motion components of the subtitles from the global optical flow field.

[0155] When determining the background optical flow field without subtitles, in one example, the independent motion vector of the subtitle can be subtracted from or inversely compensated for by the optical flow vector of each pixel within the subtitle region in the global optical flow field, thereby obtaining the true motion estimate of the background portion at that point. For example, if the global optical flow shows that a point has moved 10 pixels to the right, while the independent motion vector of the subtitle is 8 pixels to the right, it can be inferred that the background at that point has actually only moved 2 pixels to the right.

[0156] Once a clean background optical flow field is obtained, it can be used to accurately transform the background areas in previous or subsequent frames that are not obscured by subtitles into the coordinate system of the current frame. Since the optical flow field used at this point has been stripped of interference from subtitle motion, the transformed background content can achieve precise spatial alignment with the non-subtitle areas of the current frame. By integrating the portions of these aligned background contents that correspond to the subtitle areas of the current frame, high-quality predicted content can be generated.

[0157] In this embodiment, when subtitles exist in preceding and / or subsequent frames, local image patches in the subtitle region of the current frame and preceding and / or subsequent frames are acquired. Based on the temporal positional changes of the local image patches, motion estimation is performed on the local image patches to determine the independent motion vector of the subtitle itself. Based on the independent motion vector of the subtitle itself and the global optical flow field of the frame sequence, the background optical flow field without subtitles is determined. Using the background optical flow field without subtitles, the background content in the preceding and / or subsequent frames is converted to the perspective of the current frame to generate predicted content. Thus, by actively estimating and separating the independent motion components of the subtitles themselves in complex scenes where subtitles continuously appear, and removing them from the global motion, a pure optical flow field that accurately reflects the background motion is obtained. Based on this, background content conversion can be performed, effectively overcoming the interference caused by subtitle motion on background motion estimation, and significantly improving the accuracy of background content prediction and spatial alignment precision in dynamic subtitle scenes.

[0158] In one possible implementation, predicting the content of the subtitle region by utilizing the spatiotemporal correlation between preceding and subsequent frames in the frame sequence includes:

[0159] In the absence of similar unsubtitled frames, the predicted content for the subtitle region is generated based on the known content outside the subtitle region in the current frame.

[0160] The situation described here, where there are no similar untitled frames, can be due to the failure to find an untitled image that is sufficiently similar to the background of the current frame and can serve as a reliable reference in a temporally adjacent frame sequence through the aforementioned similarity calculation and threshold comparison mechanism. This usually occurs when the scene changes drastically, the camera switches quickly, or the background of the subtitle-covered area itself lacks temporal continuity.

[0161] When it is determined that valid reference information cannot be obtained from temporally adjacent frames, reference information can be determined from the spatial context information of the current frame itself. That is, by using the known content outside the subtitle area in the current frame that is not obscured by the subtitle, the content of the subtitle area can be predicted.

[0162] In practical implementation, predictions can be made based on this known content using a content generation model. A content generation model is a trained machine learning model that can automatically synthesize or complete reasonable content for missing parts based on given contextual information (such as partially known regions of an image), such as image inpainting models based on generative adversarial networks or diffusion models.

[0163] This approach does not rely on finding a specific, complete reference frame, but rather on the model's general visual priors and completion capabilities learned from a large amount of data. Therefore, even when the content of consecutive frames differs greatly or the background of the subtitle area is unique, the model can still generate visually reasonable predictions that are consistent with the surrounding content.

[0164] In this embodiment of the disclosure, when no similar untitled frames exist, the predicted content of the subtitle region is generated based on known content outside the subtitle region in the current frame. Therefore, by utilizing the spatial context information of the current frame itself to predict the subtitle region in scenarios where temporal reference fails, the robustness and generalization ability of the subtitle removal method in complex and variable video scenarios are enhanced, ensuring acceptable repair results even in the absence of ideal reference frames.

[0165] In one possible implementation, the method further includes:

[0166] Based on the numerical accuracy requirements of different calculation tasks in the subtitle removal process, different numerical calculation formats are dynamically assigned to different tasks.

[0167] The subtitle removal process encompasses multiple computational tasks, from decoding and region detection to motion estimation and content prediction and filling. The inherent algorithmic characteristics of these tasks determine their varying sensitivities to numerical precision. Numerical precision requirements characterize the different demands placed on the accuracy of data representation, the range of values, and the tolerance for rounding errors by different computational processing steps.

[0168] For example, optical flow estimation tasks involving large amounts of accumulation and tiny gradient updates may require high numerical precision to maintain computational stability and the accuracy of the final result; while some classification or feature matching tasks may be more robust to numerical errors, allowing the use of lower precision but more computationally efficient formats.

[0169] Therefore, computation formats can be dynamically assigned to different tasks. A numerical computation format represents the specific data format used to represent and manipulate numbers in a computing system, such as floating-point formats or integer formats with different bit widths. These formats each have their own characteristics in terms of numerical representation range, precision, and computational efficiency.

[0170] In practical implementation, when tasks are scheduled or data flows through different processing units, the corresponding hardware computing units (such as tensor cores or stream processors in a GPU) can be configured to run in a specified precision format. For example, the scheduler can configure the module that is about to perform optical flow estimation to use a single-precision floating-point format, while configuring the module that is about to perform optical character recognition (OCR) to use an 8-bit integer format.

[0171] The principle behind allocation is a trade-off between precision and performance. High-precision formats are allocated to error-sensitive, critical tasks to ensure output quality; low-precision formats are allocated to tasks that can tolerate some loss of precision, in exchange for higher computational throughput and lower memory bandwidth usage. This dynamic allocation makes the use of computing resources more targeted.

[0172] In this embodiment of the disclosure, different numerical calculation formats are dynamically allocated to different tasks based on their numerical precision requirements in the subtitle removal process. Therefore, by adaptively matching the most suitable numerical precision format to the characteristics of different calculation stages in the process, the computational load and storage overhead of non-critical tasks can be effectively reduced while meeting the precision requirements of critical tasks. This optimizes the overall utilization efficiency of computing resources and improves the processing speed and energy efficiency of the subtitle removal process.

[0173] In one possible implementation, based on the numerical precision requirements of different computational tasks in the subtitle removal process, different numerical calculation formats are dynamically assigned to different tasks, including:

[0174] A first precision format is assigned to the optical flow estimation task, and the first precision format includes: FP32 single-precision floating-point format;

[0175] A second precision format is assigned to the image restoration task, the second precision format including: FP16 half-precision floating-point format;

[0176] A third precision format is assigned to the subtitle recognition task, and the third precision format includes: INT8 integer format.

[0177] The first precision format has a higher number of bits for calculation than the second precision format, and the second precision format has a higher number of bits for calculation than the third precision format.

[0178] First-precision, second-precision, and third-precision formats represent data formats with different numerical representation capabilities and computational efficiencies assigned to specific computational tasks. These formats are directly supported by the computing system and are used to represent weights, activation values, and intermediate computation results when executing specific algorithms. The number of bits used in computation indicates the number of binary bits used to represent a value; generally, the higher the number of bits, the higher the range and precision of the values ​​that can be represented.

[0179] In the subtitle removal process, optical flow estimation involves computational estimation steps based on optical flow. This involves complex gradient calculations, iterative optimizations, and estimation of minute motion vectors, making it extremely sensitive to numerical precision. Even small rounding errors can accumulate and lead to significant deviations in the final optical flow field. Therefore, a first-precision format is assigned to the optical flow estimation task, such as the FP32 single-precision floating-point format. This format uses 32-bit binary numbers to represent a floating-point number, providing a large dynamic range and high precision to ensure the convergence stability and accuracy of the optical flow calculations.

[0180] Image inpainting tasks involve filling predicted content into the subtitle region to generate video frames without subtitles. This typically involves forward inference from deep neural networks to generate or fuse pixel content. Such tasks have a certain tolerance for numerical errors during the inference phase, but are computationally intensive, requiring high memory bandwidth and computational throughput. A second-precision format, such as the FP16 half-precision floating-point format, can be assigned to image inpainting tasks. This format uses 16 bits to represent floating-point numbers, significantly reducing memory usage and increasing computational speed while generally maintaining the model output accuracy that meets visual perception quality.

[0181] The subtitle recognition task involves detecting subtitle regions in the current frame of the frame sequence and generating the corresponding subtitle mask. It primarily includes feature extraction and classification based on convolutional neural networks, with highly parallel computational patterns, and the final output is a discrete category or character. This type of task has relatively low dependence on extremely high-bit floating-point precision and can benefit from more efficient integer computation. A third-precision format, such as the INT8 integer format, can be assigned to the subtitle recognition task. This format uses 8 bits to represent an integer, and by quantizing model weights and activation values ​​to low-bit integers, the computation process can be greatly accelerated and power consumption reduced, while generally maintaining acceptable recognition accuracy.

[0182] There is a clear hierarchy of precision among these three formats. The first precision format (FP32) uses 32 bits for computation, which is higher than the second precision format (FP16) which uses 16 bits; the second precision format (FP16) has a higher number of bits for computation than the third precision format (INT8) which uses 8 bits. This strategy of allocating precision from high to low constitutes a gradient computing resource allocation scheme that balances quality and efficiency.

[0183] In this embodiment, a first precision format is assigned to the optical flow estimation task, including an FP32 single-precision floating-point format; a second precision format is assigned to the image restoration task, including an FP16 half-precision floating-point format; and a third precision format is assigned to the subtitle recognition task, including an INT8 integer format. The first precision format has a higher computational bit depth than the second precision format, and the second precision format has a higher computational bit depth than the third precision format. Therefore, by precisely matching differentiated numerical computational formats from high to low based on the characteristics of each task—optical flow estimation, image restoration, and subtitle recognition—high precision optical flow calculation and high-quality restoration results can be ensured while significantly improving the computational efficiency and energy efficiency of the image restoration and subtitle recognition stages. This achieves refined and optimized allocation of computational resources for the overall subtitle removal process.

[0184] In one possible implementation, the method further includes:

[0185] The frame sequence is divided into multiple batches;

[0186] The GPU executes each batch in parallel, and performs at least one step of subtitle region detection, subtitle region content prediction, and subtitle region content filling on multiple video frames within each batch.

[0187] Multiple batches can be simultaneously scheduled to different compute units or stream processors on the GPU for processing. Furthermore, within each batch, for steps such as caption region detection, content prediction, or content padding, the GPU can leverage its thread-level parallelism to perform the same operations on multiple frames within the batch simultaneously. For example, if a batch contains 32 frames, the GPU can allocate 32 thread blocks, each responsible for caption region detection in one frame; alternatively, for parallelizable computations within a single frame (such as convolution operations), the GPU can further decouple them at a finer granularity.

[0188] This multi-level parallel strategy fully leverages the hardware characteristics of the GPU. At the inter-batch parallelism level, multiple batches can fill the GPU's computation pipeline, hiding memory access latency. At the intra-batch inter-frame parallelism level, simultaneous processing of multiple frames significantly improves the frame processing throughput per unit time. At the intra-frame computational parallelism level, intensive computations for each frame (such as neural network inference) are mapped onto thousands of parallel threads for accelerated execution. For example, two batches can be loaded into the GPU memory simultaneously; one batch performs subtitle detection while the other performs content prediction based on the detected results, forming a pipelined parallelism.

[0189] In this embodiment, the frame sequence is divided into multiple batches; the GPU executes each batch in parallel, and within each batch, multiple video frames perform at least one step of subtitle region detection, subtitle region content prediction, and subtitle region content filling. Therefore, by rationally dividing the video frame sequence into batches and fully utilizing the parallel computing capabilities of the GPU at multiple levels of computation—between batches, within batches, between frames, and within frames—the overall data processing throughput of the video subtitle removal process can be significantly improved. This effectively utilizes GPU computing resources, reduces idle waiting time, and thus achieves more efficient and faster processing of long or high frame rate videos.

[0190] The following describes an embodiment of this disclosure in a specific application scenario. Figure 4 A flowchart illustrating a video caption removal method according to an embodiment of the present disclosure is shown, such as... Figure 4 As shown, the method includes:

[0191] The frame monitoring subtask can monitor video frame sequences in real time. When a video frame is detected, it enters the processing of the current frame, reads the image data of the frame, performs OCR (Optical Character Recognition) processing on the current frame to identify the text positions in the frame, and generates corresponding text position information. Based on the text position information, a mask file for the current frame is generated, in which integer values ​​(such as 1) are used to mark subtitle pixels, while non-subtitle areas are marked with another integer value (such as 0). Then, a global mask file is generated by accumulating the mask information of multiple consecutive frames. The cumulative value at each position in this file represents the frequency of the subtitle appearing at that position, used to distinguish between fixed subtitles and text that appears occasionally in the picture.

[0192] Based on the global mask file, the subtitle regions where subtitles frequently appear can be identified. Then, based on the results of OCR text location recognition, it is determined whether subtitles exist in the subtitle region. If no subtitles are found, i.e., the current frame has no subtitles, the content of that frame is cached or marked as a "no subtitle frame" for use as a reference frame in subsequent predictions. Then, the next frame is processed.

[0193] If the determination is yes, meaning the current frame contains subtitles, then OCR text recognition is performed, and the recognition result is output. Simultaneously, the current frame subtitle file and the global subtitle file are updated. Then, the subtitle frame repair process begins:

[0194] First, a forward search is performed to determine if the current frame is similar to the previous f frames. This involves calculating the similarity between the current frame and the previous f-th frame (where f is an integer, usually initialized to 1) in the non-subtitle region. If the similarity is high enough, the previous f frames are further checked for subtitles. If the previous f frames are subtitle-free, image prediction is performed based on these subtitle-free frames. If the previous f frames have subtitles or are not similar, the integer f is incremented (f = f + 1), attempting to look back at earlier frames (e.g., f = 2, 3...). This process is repeated until a suitable reference is found or the search limit is exceeded.

[0195] Next, a backward search is performed. If the forward search fails, a backward search is initiated, with logic completely symmetrical to the forward search. It checks if the current frame is similar to the content of the next b frames (b is an integer, usually initialized to 1). If similar, it checks if the next b frames have no subtitles. If so, it performs image prediction based on the subsequent b frames without subtitles. If not, it increments the integer b (b = b + 1) and attempts to view further subsequent frames until a suitable reference is found or the search limit is exceeded.

[0196] If no matching frame without subtitles is found through forward (traversing f) and backward (traversing b) searches, artificial intelligence (AI) prediction can be performed based on the image within the current frame, generating content using the contextual information of this frame. That is, by utilizing known background content outside the subtitle area in the current frame, and through image restoration or generation models, reasonable content for the occluded area can be directly inferred and generated without relying on other frames.

[0197] Figure 5 A structural diagram of a video caption processing chip provided according to an embodiment of this disclosure is shown. The chip may be located within a graphics processing unit (GPU) or be a dedicated video processing chip. This chip can be used to execute the video caption processing method provided in this disclosure. It is a collaborative processing system integrating parallel computing, dedicated hardware acceleration, and hierarchical storage, efficiently supporting the entire process from video decoding, caption detection, content prediction to final filling at the hardware level.

[0198] The entire chip's data and control flow is scheduled by a hardware task scheduler. The hardware task scheduler receives instructions from the CPU / host, parses the subtitle removal task, and dynamically decomposes and schedules it to the various dedicated hardware modules below. The host side is equipped with memory and a hard disk to store input video, output results, and necessary system software and model parameters.

[0199] The hardware task scheduler first schedules the video parsing task to the video processing module. The video processing module has multiple built-in decoders (decoder 1 to decoder X, where X is a positive integer) used to receive the input video stream in parallel and efficiently decode it into an uncompressed raw frame sequence using hardware decoding capabilities. The decoded frame sequence is then sent to storage units, such as a buffer. The buffer temporarily stores frequently accessed currently processed frames and their neighboring frames, while memory (e.g., GPU memory) is used to store the complete frame sequence, intermediate data, and mask files generated in subsequent steps.

[0200] The AI ​​computing module can read frame sequences stored in the cache. This module includes hardware units optimized for deep learning inference—subtitle recognition modules (subtitle recognition module 1 and subtitle recognition module 2, etc.). The subtitle recognition module loads and runs a pre-trained text detection and recognition model, analyzes the current frame read from the cache, locates the subtitle region, and generates a binary single-frame subtitle mask file corresponding to the current frame (subtitle regions are marked with specific integers such as 1, and non-subtitle regions are marked with 0). Based on the mask files of consecutive frames, a global subtitle mask file is generated, where the cumulative integer value at each position reflects the frequency of subtitle occurrence. In addition, the subtitle recognition module can also recognize the text content of the subtitles.

[0201] The image processing modules (Image Processing Module 1 and Image Processing Module 2, etc.) in the AI ​​computing module can be used to predict the content to be filled in the subtitle region. Specifically, the image processing modules perform optical flow estimation based on the subtitle mask, using the decoded previous and subsequent frame sequences stored in memory to calculate inter-frame motion, and evaluate the similarity between the current frame and neighboring subtitle-free frames. If a target subtitle-free frame with a similarity exceeding a preset threshold is found, the background content of the occluded area is predicted based on that frame using feature matching and weighted fusion algorithms. If the subtitle appears in consecutive frames, AI prediction can be performed based on the image within the current frame, generating predicted content using the context information of this frame. In addition, the independent motion of the subtitle itself can be estimated, and the pure background optical flow can be separated from the global optical flow, and then warped from other frames to generate predicted content. The entire prediction process extensively uses matrix and tensor operations, which are accelerated in parallel by multiple image processing modules.

[0202] After obtaining the predicted content, the image calculation module uses multiple (M in the diagram, where M is a positive integer) image synthesis modules (Image Synthesis Module 1, Image Synthesis Module 2... Image Synthesis Module M) to perform the filling of the predicted content and image synthesis. The image calculation module reads the current frame, the corresponding subtitle mask, and the predicted content output by the AI ​​calculation module, and performs pixel-level filling operations. The predicted content is precisely fitted to the area indicated by the mask, and image fusion techniques (such as edge blending) are used to ensure that the filled area transitions naturally with the surrounding background in terms of texture and color, generating the final subtitle-free video frame.

[0203] The processed, unsubtitled frame sequence is buffered and sent back to the video processing module. This module calls its multiple encoders (encoder 1 to encoder Y, where Y is a positive integer) to re-encode the repaired frame sequence and synthesize it into a complete output video stream, which is then written back to memory or directly output.

[0204] This modular, pipelined, and highly parallel hardware architecture significantly accelerates each subtask of video caption removal, enabling high-quality and efficient processing.

[0205] According to another aspect of this disclosure, a video caption removal chip is provided, comprising:

[0206] The video processing module is used to receive input video and decode it into a frame sequence;

[0207] The subtitle recognition module is used to detect the subtitle region of the current frame in the frame sequence and generate the corresponding subtitle mask;

[0208] The image processing module is configured to, in response to the presence of subtitles in the current frame, predict the content of the subtitle region based on the subtitle mask and the spatiotemporal correlation of the preceding and / or subsequent frames in the frame sequence.

[0209] The image synthesis module fills the predicted content into the subtitle area to generate a video frame with the subtitles removed.

[0210] A hardware task scheduler is used to schedule the video processing module, the subtitle recognition module, the image processing module, and the image synthesis module to work together.

[0211] In one possible implementation, the subtitle recognition module is used for:

[0212] Identify the subtitle region in the current frame;

[0213] Based on the subtitle region, a single-frame subtitle mask file corresponding to the current frame is generated, wherein the subtitle region is indicated by numerical values ​​in the single-frame subtitle mask file;

[0214] The values ​​in the single-frame subtitle mask files of consecutive video frames are accumulated to generate a global subtitle mask file, which is used to count the areas where subtitles appear multiple times.

[0215] In one possible implementation, the subtitle recognition module is used to identify regions in the global subtitle mask file whose cumulative value exceeds a frequency threshold as video subtitle regions to be removed.

[0216] In one possible implementation, the image processing module is configured to:

[0217] Determine the similarity between the current frame and at least one neighboring frame without subtitles;

[0218] In the case of a target frame without subtitles where the similarity is greater than the first threshold, the image of the subtitle region is predicted based on the target frame without subtitles to obtain the content of the predicted subtitle region.

[0219] In one possible implementation, the image processing module is configured to:

[0220] For the pixel blocks to be filled in the subtitle region, the corresponding candidate image blocks are extracted from each target subtitle-free frame;

[0221] Calculate the feature similarity between the surrounding region of the pixel block to be filled and the surrounding region of each candidate image block;

[0222] Based on the feature similarity, a contribution weight is assigned to each candidate image patch, and the contribution weight is positively correlated with the feature similarity;

[0223] Based on the contribution weights, the features of multiple candidate image blocks are weighted and fused to generate the predicted content of the pixel block to be filled.

[0224] In one possible implementation, the image processing module is configured to:

[0225] Based on the optical flow field between the current frame and the target no-subtitle frame, the corresponding position of the pixel block to be filled in the target no-subtitle frame is determined;

[0226] The corresponding candidate image block is extracted with the corresponding position as the center.

[0227] In one possible implementation, the image processing module is used for:

[0228] If subtitles are present in the preceding and / or subsequent frames, obtain local image blocks of the current frame and the preceding and / or subsequent frames in the subtitle area;

[0229] Based on the temporal positional changes of the local image patch, motion estimation is performed on the local image patch to determine the independent motion vector of the subtitle itself;

[0230] Based on the independent motion vector of the subtitle itself, the global optical flow field of the frame sequence is used to determine the background optical flow field without subtitles;

[0231] Using the background light flow field without subtitles, the background content in the preceding and / or subsequent frames is transformed to the perspective of the current frame to generate predicted content.

[0232] In one possible implementation, the hardware task scheduler is used to dynamically allocate different numerical calculation formats to different tasks based on their numerical accuracy requirements in the subtitle removal process.

[0233] In one possible implementation, the hardware task scheduler is used to:

[0234] A first precision format is assigned to the optical flow estimation task, the first precision format including: single-precision floating-point format FP32;

[0235] A second precision format is assigned to the image restoration task, the second precision format including: half-precision floating-point format FP16;

[0236] A third-precision format is assigned to the subtitle recognition task, and the third-precision format includes: integer format INT8;

[0237] The first precision format has a higher number of bits for calculation than the second precision format, and the second precision format has a higher number of bits for calculation than the third precision format.

[0238] According to another aspect of this disclosure, a graphics processing unit (GPU) is provided, the GPU comprising the chip described above.

[0239] According to another aspect of this disclosure, an electronic device is provided, the electronic device including the chip described above.

[0240] In some embodiments, the functions or modules of the chip provided in this disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0241] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0242] In the description of this specification, it should be understood that the terms "center," "longitudinal," "transverse," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this disclosure.

[0243] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means two or more, unless otherwise explicitly specified.

[0244] In this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a communication connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0245] In this disclosure, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0246] The foregoing disclosure provides many different implementations or examples for carrying out different structures of this disclosure. To simplify the disclosure, specific examples of components and arrangements have been described above. Of course, these are merely examples and are not intended to limit the scope of this disclosure. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0247] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this disclosure, and these should all be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for removing video subtitles, characterized in that, The method is executed on a graphics processing unit (GPU) and includes: Receive input video and decode the input video into a frame sequence; The current frame in the frame sequence is used to detect subtitle regions and generate the corresponding subtitle mask; In response to the presence of subtitles in the current frame, the content of the subtitle region is predicted based on the subtitle mask and the spatiotemporal correlation of the preceding and / or subsequent frames in the frame sequence. The predicted content is filled into the subtitle area to generate a video frame with the subtitles removed; Based on the subtitle mask, and utilizing the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence, the content of the subtitle region is predicted, including: If subtitles are present in the preceding and / or subsequent frames, obtain local image blocks of the current frame and the preceding and / or subsequent frames in the subtitle area; Based on the temporal positional changes of the local image patch, motion estimation is performed on the local image patch to determine the independent motion vector of the subtitle itself; Based on the independent motion vector of the subtitle itself, the global optical flow field of the frame sequence is used to determine the background optical flow field without subtitles; Using the background light flow field without subtitles, the background content in the preceding and / or subsequent frames is transformed to the perspective of the current frame to generate predicted content.

2. The method according to claim 1, characterized in that, The step of detecting subtitle regions in the current frame of the frame sequence and generating a corresponding subtitle mask includes: Identify the subtitle region in the current frame; Based on the subtitle region, a single-frame subtitle mask file corresponding to the current frame is generated, wherein the subtitle region is indicated by numerical values ​​in the single-frame subtitle mask file; The values ​​in the single-frame subtitle mask files of consecutive video frames are accumulated to generate a global subtitle mask file, which is used to count the areas where subtitles appear multiple times.

3. The method according to claim 2, characterized in that, The step of detecting subtitle regions in the current frame of the frame sequence includes: The regions in the global subtitle mask file whose cumulative value exceeds the frequency threshold are identified as video subtitle regions to be removed.

4. The method according to claim 1, characterized in that, The step of predicting the content of the subtitle region based on the subtitle mask and utilizing the spatiotemporal correlation of preceding and / or subsequent frames in the frame sequence includes: Determine the similarity between the current frame and at least one neighboring frame without subtitles; In the case of a target frame without subtitles where the similarity is greater than the first threshold, the image of the subtitle region is predicted based on the target frame without subtitles to obtain the content of the predicted subtitle region.

5. The method according to claim 4, characterized in that, The step of predicting the content of the subtitle region based on the target subtitle-free frame includes: For the pixel blocks to be filled in the subtitle region, the corresponding candidate image blocks are extracted from each target subtitle-free frame; Calculate the feature similarity between the surrounding region of the pixel block to be filled and the surrounding region of each candidate image block; Based on the feature similarity, a contribution weight is assigned to each candidate image patch, and the contribution weight is positively correlated with the feature similarity; Based on the contribution weights, the features of multiple candidate image blocks are weighted and fused to generate the predicted content of the pixel block to be filled.

6. The method according to claim 5, characterized in that, The step of extracting corresponding candidate image blocks from each target frame without subtitles includes: Based on the optical flow field between the current frame and the target no-subtitle frame, the corresponding position of the pixel block to be filled in the target no-subtitle frame is determined; The corresponding candidate image block is extracted with the corresponding position as the center.

7. The method according to claim 1, characterized in that, The method further includes: Based on the numerical accuracy requirements of different calculation tasks in the subtitle removal process, different numerical calculation formats are dynamically assigned to different tasks.

8. The method according to claim 7, characterized in that, Based on the numerical precision requirements of different computational tasks in the subtitle removal process, different numerical calculation formats are dynamically assigned to different tasks, including: A first precision format is assigned to the optical flow estimation task, the first precision format including: single-precision floating-point format FP32; A second precision format is assigned to the image restoration task, the second precision format including: half-precision floating-point format FP16; A third-precision format is assigned to the subtitle recognition task, and the third-precision format includes: integer format INT8; The first precision format has a higher number of bits for calculation than the second precision format, and the second precision format has a higher number of bits for calculation than the third precision format.

9. A video subtitle removal chip, characterized in that, include: The video processing module is used to receive input video and decode it into a frame sequence; The subtitle recognition module is used to detect the subtitle region of the current frame in the frame sequence and generate the corresponding subtitle mask; The image processing module is configured to, in response to the presence of subtitles in the current frame, predict the content of the subtitle region based on the subtitle mask and the spatiotemporal correlation of the preceding and / or subsequent frames in the frame sequence. The image synthesis module fills the predicted content into the subtitle area to generate a video frame with the subtitles removed. A hardware task scheduler is used to schedule the video processing module, the subtitle recognition module, the image processing module, and the image synthesis module to work together. Image processing module, used for: If subtitles are present in the preceding and / or subsequent frames, obtain local image blocks of the current frame and the preceding and / or subsequent frames in the subtitle area; Based on the temporal positional changes of the local image patch, motion estimation is performed on the local image patch to determine the independent motion vector of the subtitle itself; Based on the independent motion vector of the subtitle itself, the global optical flow field of the frame sequence is used to determine the background optical flow field without subtitles; Using the background light flow field without subtitles, the background content in the preceding and / or subsequent frames is transformed to the perspective of the current frame to generate predicted content.

10. A graphics processing unit (GPU), characterized in that, The GPU includes the chip as described in claim 9.

11. An electronic device, characterized in that, The electronic device includes the chip as described in claim 9.

Images