A missing word compensation processing apparatus and method for video subtitle extraction

Abstract

This invention discloses a device and method for compensating for missing characters in video subtitle extraction, effectively addressing the problem of missing characters caused by video image quality, complex background interference, limitations of OCR recognition, and dynamic changes in subtitles. This invention employs key technologies such as adaptive subtitle denoising, semantic compensation, and multi-frame fusion to significantly improve the completeness and accuracy of subtitle extraction. This solution is applicable to various types of videos, including movies, TV series, short videos, and conference videos. Its core innovation lies in its ability to effectively detect, compensate for, and correct missing or erroneous characters by intelligently analyzing context and integrating multi-frame information, thereby significantly improving the quality of subtitle output, especially in scenarios where individual characters are easily lost.

Description

Technical Field

[0001] This invention relates to the field of video subtitle technology, and more specifically, to a technique for extracting subtitle text from videos, particularly an apparatus and method for detecting, analyzing, and compensating for character omissions (missing characters) that may occur during the extraction process, aiming to improve the completeness and accuracy of subtitle extraction. Background Technology

[0002] Subtitles are an essential component of video content, playing an indispensable role in information delivery, cross-language understanding, and assistance for the hearing impaired. Video Subtitle Extraction (VDE) technology has emerged to address this need, aiming to automatically identify and extract embedded subtitle text from video footage using computer vision and Optical Character Recognition (OCR) technologies.

[0003] Existing video caption extraction methods typically include two main steps: caption region detection and caption text recognition. Caption region detection is responsible for locating the rectangular region containing the caption in the video frame, while caption text recognition uses OCR technology to convert the detected caption region image into a text sequence.

[0004] However, in practical applications, video subtitle extraction faces many challenges, leading to inaccurate or incomplete results. One common and troublesome problem is the "missing characters" phenomenon. Missing characters can be caused by a variety of reasons:

[0005] Video quality issues: Low resolution, compression artifacts, noise interference (such as salt and pepper noise, Gaussian noise), motion blur, etc. may cause the image quality of the subtitle area to degrade, making some characters blurry, incomplete at the edges, or difficult to distinguish from the background, making them difficult to be correctly recognized by the OCR engine, or even ignored.

[0006] Complex background interference: The subtitle area may be similar to or overlap with the video background color or texture (for example, the subtitle is superimposed on a complex scene, pattern or rapidly changing background), resulting in inaccurate subtitle segmentation and loss of some character information due to insufficient contrast or being occluded by background elements.

[0007] OCR Recognition Limitations: OCR engines have limited ability to recognize specific fonts (such as handwriting and artistic fonts), font sizes, slant, excessively small or large character spacing, and non-standard character arrangements. Furthermore, they may miss or misrecognize characters when processing rapidly changing subtitles, semi-transparent subtitles, or subtitles with outlines / shadows.

[0008] Dynamic changes in subtitles: The rapid appearance, disappearance, scrolling, scaling, or special effects (such as fade-in / fade-out, word-by-word display) of subtitles may result in incomplete or low-confidence information for some characters in certain frames, or they may be completely missed when the sampling frame rate is insufficient.

[0009] Traditional subtitle extraction methods often focus on improving the accuracy of single-frame OCR or performing simple post-processing (such as spell correction based on a fixed dictionary or simple rule filtering). However, for the "loss" of characters at the physical level (insufficient image information) or at the recognition level (OCR engine cannot handle it) due to the aforementioned reasons, there is a lack of effective intelligent compensation mechanisms based on context and multi-frame information. This results in missing content in the final output subtitle text sequence, seriously affecting the integrity of information and the user's viewing experience. Especially for language information that relies on contextual understanding (such as Chinese, Japanese, Korean, etc.), the loss of a single character often leads to a huge deviation or even a complete error in the semantics of the entire sentence.

[0010] Therefore, there is an urgent need for a technical solution that can effectively detect and intelligently compensate for missing characters during video subtitle extraction, so as to significantly improve the completeness, accuracy and time consistency of subtitle extraction. Summary of the Invention

[0011] This invention aims to solve the problem of missing characters in existing video subtitle extraction technologies due to factors such as video quality, complex backgrounds, OCR limitations, and dynamic changes in subtitles. It provides a video subtitle extraction and processing device and method that can detect, analyze, and intelligently compensate for missing characters, thereby improving the completeness, accuracy, and time consistency of the final output subtitles.

[0012] To address the aforementioned technical problems, the present invention provides a missing character compensation processing device for video subtitle extraction, the device comprising:

[0013] Video input interface: Used to receive input video containing video data streams;

[0014] Subtitle detection module: Configured to process video data streams, identify subtitle regions within video frames by analyzing image features of video frames (such as edges, colors, textures, connected components, etc.) or by using pre-trained deep learning models, and optionally track the position and size changes of subtitle regions in consecutive frames;

[0015] OCR Recognition Module: Configured to perform optical character recognition (OCR) on the subtitle region image identified by the subtitle detection module, generate an initial subtitle text sequence containing character and position information and confidence information, and optionally combine a basic language model or dictionary for preliminary spelling correction or correction of common recognition errors;

[0016] Adaptive subtitle denoising module: configured to adaptively select and apply one or more denoising and enhancement algorithms based on the image characteristics or extracted features of the subtitle area to reduce background noise, artifacts or interference, sharpen character edges, optimize the extraction quality of subtitle information, and provide higher quality input for subsequent semantic compensation and multi-frame fusion;

[0017] Semantic compensation module: One of the core modules of this invention, configured to analyze and predict the location and content of any missing characters (omitted words) based on the initial subtitle text sequence generated by the OCR recognition module and its contextual information (such as word collocation, syntactic structure, and thematic coherence) using a large language model. This module can automatically complete the missing characters based on the prediction results and confidence level, generating a compensated subtitle text sequence;

[0018] Multi-frame fusion module: Another core module of this invention, configured to analyze subtitle information (which may be image features, OCR recognition results, or semantically compensated text fragments) from multiple temporally adjacent or content-related video frames. Through inter-frame alignment and information weighted fusion strategies, this module can utilize temporal redundancy to verify, correct, or supplement characters, further improving the completeness, accuracy, and temporal consistency of the final output subtitles, and can assist the semantic compensation module in decision-making or result optimization;

[0019] Subtitle output module: Configured to receive the final subtitle data processed by the semantic compensation module and / or multi-frame fusion module, and generate subtitle files according to preset or user-specified formats (such as SRT, ASS, VTT, etc.), or render the subtitle text into images and overlay them onto the corresponding video output frames.

[0020] Another aspect of the present invention provides a method for compensating for missing characters in video subtitle extraction, the method comprising the following steps:

[0021] Video Input and Subtitle Detection: Receives and processes input video frames, identifies subtitle regions in the video frames through the subtitle detection process, and optionally tracks their spatiotemporal changes in consecutive frames;

[0022] Initial subtitle recognition: Perform OCR on the identified subtitle area to generate an initial subtitle text sequence containing character, position and confidence information, and optionally perform preliminary language model correction;

[0023] Adaptive denoising and enhancement: Adaptive denoising and enhancement processing is performed on the image data of the subtitle area or its derived features to optimize the visual quality and recognizability of the subtitles;

[0024] Semantic analysis and missing character compensation: Analyze the semantic context, structural features and language model probabilities of the initial subtitle text sequence, detect potential missing character locations, use the language model to predict and automatically fill in one or more of the most likely missing characters, and generate a compensated subtitle text sequence;

[0025] Multi-frame information fusion and optimization: Integrate and analyze subtitle information from multiple video frames that are temporally adjacent or content-related. Through inter-frame alignment, information weighting, and decision fusion, further verify, correct, or supplement subtitle content to improve the completeness, accuracy, and temporal consistency of subtitles.

[0026] Subtitle formatting and output: The final subtitle text sequence, after being processed and optimized in the above steps, is formatted into a standard subtitle file or rendered and overlaid on the video for output.

[0027] Compared with the prior art, the apparatus and method provided by the present invention have the following beneficial effects:

[0028] Effective compensation for missing characters: Through the core semantic compensation module, it can intelligently predict and complete characters lost during the OCR process due to various reasons by leveraging powerful language models and context analysis capabilities, significantly improving the integrity of subtitle text and solving the problem of physical or recognizable character loss that is difficult to handle by traditional methods.

[0029] Improved accuracy and robustness: By combining the adaptive denoising module to optimize input quality and the multi-frame fusion module to use temporal redundancy information for cross-validation and correction, it can effectively cope with adverse factors such as video noise, complex backgrounds, and dynamic changes in subtitles, thereby improving the overall accuracy of subtitle extraction and its robustness to harsh conditions.

[0030] Enhanced temporal consistency: The multi-frame fusion module integrates information in the temporal dimension to ensure the stability and continuity of the output subtitles in time, reducing the impact of single-frame recognition errors, missing characters, or fluctuations in recognition results, making the subtitle stream smoother and more natural;

[0031] The processing workflow is flexible and intelligent: the device and method allow for the selective or dynamic application of modules and their internal algorithms, such as denoising, semantic compensation, and multi-frame fusion, based on the characteristics of the video and subtitles. For example, the activation of compensation and fusion modules can be determined based on the quality assessment of the initial OCR results, thus balancing processing effectiveness and computational efficiency.

[0032] High degree of automation, enhancing user experience: The entire processing flow is highly automated, greatly reducing the need for manual proofreading and correction of subtitles, saving labor costs, and enabling the delivery of high-quality subtitles faster, thereby improving the user's viewing and usage experience. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a structural block diagram of a character loss compensation processing device according to an embodiment of the present invention;

[0035] Figure 2 This is a basic flowchart of a character loss compensation processing method according to an embodiment of the present invention;

[0036] Figure 3 This is a dynamic adaptive flowchart of a character loss compensation processing method according to another embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of a possible internal structure of the adaptive subtitle denoising module according to an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of the internal processing flow of the semantic compensation module according to an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram of the internal processing flow of the multi-frame fusion module according to an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1: Missing Character Compensation Processing Device

[0042] Reference Figure 1 This invention provides a missing character compensation processing device 100 for video subtitle extraction. This device can be a hardware entity, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a dedicated computing device including a central processing unit (CPU), a graphics processing unit (GPU), and memory; or it can be a combination of software modules running on a general-purpose computing device (such as a server or personal computer). Device 100 includes:

[0043] Video input interface 110: Used to receive input video. The input video can come from various sources, including local files (such as MP4, AVI, MKV, etc.), real-time network video streams (such as RTSP, HLS, RTMP), and camera capture devices, and includes encoded video data streams. This interface may also include decoding functionality to decode the video data into a sequence of raw video frames.

[0044] The subtitle detection module 120 is responsible for processing the decoded video frames. It analyzes each frame or selected keyframe image, employing methods such as edge detection (e.g., Canny edge detection), color segmentation (e.g., clustering based on subtitle color characteristics), texture analysis (e.g., MSER - Maximally Stable Extremal Regions), or deep learning-based object detection models (e.g., Faster R-CNN, YOLO, SSD, or models optimized for text detection such as EAST, DBNet) to identify and locate one or more rectangular regions (bounding boxes) that may contain subtitles. This module can also be configured to track the position and size changes of these subtitle regions between consecutive frames, for example, using Kalman filters, particle filters, or simple inter-frame region overlap calculations, to cope with subtitle scrolling, translation, scaling, or video jitter, ensuring stable tracking of the subtitle regions.

[0045] OCR recognition module 130 receives the subtitle region image blocks located by subtitle detection module 120. It applies an Optical Character Recognition (OCR) engine (which can be a traditional method based on feature extraction and classifiers, or a more advanced end-to-end OCR model based on deep learning, such as CRNN+CTC or Attention-OCR) to convert these images into initial text sequences. The output typically includes the recognized characters, the confidence score for each character, and the character's position information in the image. To improve initial accuracy, this module can selectively integrate a base language model (such as an N-gram model or a small neural network language model) or a dictionary to perform preliminary spell checking, correction of common OCR error pairs (e.g., "1" vs. "l", "0" vs. "O"), or preliminary filtering of illogical character sequences based on a simple vocabulary.

[0046] Adaptive Subtitle Denoising Module 140: This module aims to improve the quality of the subtitle region image input to the OCR module (if applied beforehand), or optimize the extracted subtitle region image / features, providing more reliable input for subsequent semantic compensation and multi-frame fusion. Its "adaptive" nature is reflected in its ability to dynamically adjust processing strategies or parameters based on the detected noise type, subtitle characteristics (such as color, contrast, font), or background complexity. Figure 4 As shown, this module may contain one or more of the following sub-units:

[0047] Frequency domain filtering unit 141: Analyzes the spectral characteristics of the subtitle region image through Fourier transform, wavelet transform or cosine transform, etc., and designs and applies filters (such as band-stop filters, Wiener filters) to suppress noise of specific frequencies (such as periodic noise, some high-frequency noise).

[0048] Color / Spatial Segmentation and Enhancement Unit 142: Utilizing the characteristic that subtitles typically have a relatively single color and high contrast with the background, color clustering (such as K-Means), adaptive thresholding (such as Otsu's method, Niblack's method), or graph cut-based segmentation methods are employed to more accurately separate subtitle pixels from the background. Local contrast enhancement algorithms (such as CLAHE) can be combined to enhance the sharpness of character edges;

[0049] Image restoration and enhancement unit 143: applies techniques such as brightness / contrast adjustment, histogram equalization, deblurring algorithms (such as blind deconvolution and Wiener filtering, especially for motion blur or out-of-focus blur), and super-resolution techniques (such as SRCNN and ESPCN, if the subtitle resolution is low) to improve the overall visual quality and OCR recognizability of the image.

[0050] Morphological processing unit 144: Uses mathematical morphological operations such as dilation, erosion, opening, and closing to smooth character edges, fill holes that may exist inside characters, remove small noise spots, or connect broken strokes.

[0051] Adaptive filtering unit 145: Adaptively selects filtering methods (such as mean filtering, median filtering, Gaussian filtering, bilateral filtering) and parameters based on local statistical characteristics of the image (such as mean, variance, gradient), striving to preserve the edge and detail information of the subtitles to the maximum extent while effectively denoising;

[0052] Temporal filtering unit 146: When processing multiple frames, if the subtitle content is relatively stable in a short period of time, the time dimension filtering (such as time median filtering, weighted averaging) can be performed using the aligned subtitle region images of adjacent frames to suppress random noise and enhance the stability of the subtitle image.

[0053] Semantic compensation module 150: This is the key module for solving the missing character problem. It receives the initial subtitle text sequence generated by the OCR recognition module 130. For example... Figure 5 As shown, its internal processing flow may include:

[0054] Context and Structure Analysis Unit 151:

[0055] Contextual analysis: Analyze the local context (such as adjacent words, N-grams) and global context (such as entire sentences, paragraphs, or domain knowledge based on video topics) of a text sequence.

[0056] Structural analysis: A structural model is established by combining the layout characteristics of the subtitle area (such as average character spacing, line spacing, alignment, and expected number of characters—which can be inferred from similar subtitle lines);

[0057] Missing character location detection: By comparing the significant difference between the actual character spacing and the expected spacing, detecting the locations of semantic incoherence or grammatical errors in the text sequence, or using a language model to evaluate the probability of each segment in the sequence, locations with significantly low probability or structural abnormalities (such as excessively large gaps) are identified and marked as potential missing character locations.

[0058] Language model prediction unit 152:

[0059] Powerful pre-trained language models (such as BERT, GPT series, T5, etc., which can be fine-tuned for subtitle corpora) are used to calculate the transition probability or sequence generation probability between characters / words;

[0060] For each detected potential missing word location, it is treated as a "mask" or "fill-in-the-blank" task. The model predicts one or more characters (or word fragments) that are most likely to be missing at that location based on its left and right context, and gives the probability or confidence score for each candidate character / fragment.

[0061] Semantic verification and optimization unit 153:

[0062] The candidate characters / fragments predicted by the language model are validated a second time. This involves checking whether the completed sequence conforms to syntactic rules (such as part-of-speech collocation and sentence structure), whether the semantics are coherent, and whether the logic is sound.

[0063] External knowledge bases (such as commonsense bases, domain dictionaries, and thesaurus) can be used for auxiliary verification and screening;

[0064] Based on the verification results, the predicted probability of candidate characters, and their contribution to the overall sequence fluency, candidate characters are filtered and sorted, and the optimal one or a few completion schemes are selected.

[0065] Finally, based on a preset confidence threshold or selection strategy, the selected characters / fragments are inserted into the missing positions of the original sequence to generate a compensated subtitle text sequence.

[0066] Multi-frame fusion module 160: This module utilizes the temporal redundancy and continuity of video subtitles to further improve the stability and completeness of the final result. It can also assist in verifying or correcting the results of semantic compensation, or directly perform character loss compensation. For example... Figure 6 As shown, its internal processing flow may include:

[0067] Inter-frame feature extraction and alignment units 161 and 162:

[0068] Feature extraction unit 161: Extracts robust features from detected subtitle regions in multiple temporally adjacent or content-related video frames (e.g., several frames within a time window, or a group of frames selected based on content change detection). These features may be:

[0069] Image-level features: such as local invariant features like SIFT, SURF, and ORB, or image embedding vectors based on deep learning;

[0070] OCR result-level features include: the identified character sequence, the confidence score of each character, and the geometric position of the character (bounding box).

[0071] Frame Alignment Unit 162: Based on extracted features, it uses techniques such as template matching, optical flow (e.g., Lucas-Kanade), feature point matching and transformation estimation (e.g., affine transformation or homography transformation under RANSAC optimization) to accurately align the subtitle regions or their recognized content in different frames. The goal is to compensate for displacement and deformation caused by camera movement, object motion, or subtitle scrolling / scaling.

[0072] Character / Information Weighted Fusion Unit 163: Fuses the aligned multi-frame information. This can occur at different levels:

[0073] Image-level fusion (optional): If the alignment accuracy is high, the aligned multi-frame caption image blocks can be averaged at the pixel level or weighted average to improve the signal-to-noise ratio, and then a higher quality OCR can be performed on the fused image.

[0074] OCR result fusion (core):

[0075] Collect OCR recognition results (characters, confidence scores) at alignment positions across multiple frames;

[0076] For a certain character position, if the OCR result of the current frame has low confidence or is empty (missing character, which may have been marked by the semantic compensation module or determined by confidence), then the recognition results of other aligned frames at that position are referenced.

[0077] During fusion, different weights are assigned based on factors such as the confidence level of each frame result, temporal distance (closeness to the current frame), image quality assessment score (such as sharpness and contrast), and whether semantic compensation has been performed.

[0078] The final character is determined using mechanisms such as majority voting, selection based on the highest confidence level, or calculation of a weighted average confidence level. This process helps smooth out random recognition errors in a single frame and utilizes temporal continuity to confirm or fill in (compensate for) lost characters. For example, if a character is stably recognized at the same location in multiple consecutive frames, it can be filled in even if it is lost in a few frames.

[0079] The subtitle output module 170 receives the final, complete, and consistent subtitle data processed by the semantic compensation module 150 and / or the multi-frame fusion module 160. Based on user requirements or preset configurations, it formats this text data (including timestamp information, typically determined and optimized by the subtitle detection module or multi-frame fusion module) into a standard subtitle file format (such as SRT, ASS, VTT, SAMI, etc.). Alternatively, it renders it as an image containing the text and overlays it onto the corresponding video output frame in real-time or during post-processing to generate a video with hard subtitles.

[0080] Example 2: Missing Character Compensation Method

[0081] Reference Figure 2 This invention provides a method for compensating for missing characters in video subtitle extraction. Corresponding to the apparatus in Embodiment 1, its basic process includes the following steps:

[0082] Step S210: Video Input and Subtitle Detection. Receive the input video data stream and decode the video frames. Process the video frames (or keyframes) and identify the bounding boxes of one or more subtitle regions using subtitle detection algorithms (such as MSER, EAST, DBNet, etc.). Optionally, establish tracking relationships between subtitle regions in consecutive frames (e.g., using Kalman filtering).

[0083] Step S220: Initial Subtitle Recognition. Perform OCR (e.g., CRNN+CTC) on the image data within each subtitle region identified in Step S210. Generate an initial subtitle text sequence, which typically includes the text content of each character, its position coordinates in the image, and a confidence score. Optionally, use a base language model or dictionary for preliminary spell correction or correction of common OCR errors.

[0084] Step S230: Adaptive Subtitle Denoising and Enhancement. Based on the characteristics of the subtitle region image (such as noise level, contrast, and sharpness) or extracted features, one or more denoising and enhancement techniques (such as frequency domain filtering, color / space segmentation, morphological operations, image restoration algorithms, adaptive filtering, temporal filtering, etc.) are adaptively selected and applied to improve the processing effect of subsequent semantic compensation and multi-frame fusion modules. This step can be applied to the original subtitle region image or to the extracted features.

[0085] Step S240: Semantic analysis and word loss compensation.

[0086] Context and Structure Analysis: Analyze the local and global context information of the initial subtitle text sequence obtained in step S220 (or the OCR result corresponding to the input optimized by S230), as well as the layout structure features of the subtitles (character spacing, alignment, etc.).

[0087] Missing word location detection: Combining language model probability evaluation and structural anomaly detection, we can identify potential missing word locations in a text sequence.

[0088] Language model prediction and completion: Utilizing powerful pre-trained language models (such as BERT, GPT series, T5, etc.), based on the context of the identified characters, predict one or more characters most likely to be missing at the detected missing character position, and provide the confidence score of the candidate characters.

[0089] Semantic verification and optimization: Perform syntactic and semantic verification on candidate completion characters, select the optimal completion scheme, insert it into the original sequence, and generate a semantically compensated subtitle text sequence.

[0090] Step S250: Multi-frame information fusion and optimization.

[0091] Feature extraction and alignment: Extract subtitle-related features (image features or OCR result features) from the current frame and multiple temporally adjacent or content-related video frames, and accurately align the subtitle information of these frames.

[0092] Weighted fusion: The aligned multi-frame information (such as OCR characters, confidence scores, and semantic compensation results) is weighted and fused. The weights can be determined based on factors such as the confidence score of each frame, image quality, and temporal proximity.

[0093] Decision and Refinement: The final characters are determined through voting, selecting the highest confidence level, or combining weighted information. This step can further correct single-frame errors, fill in missing characters that semantic compensation could not handle (if the information in adjacent frames is complete), and ensure the temporal consistency and smoothness of the subtitle sequence.

[0094] Step S260: Subtitle Formatting and Output. The final subtitle text sequence, processed and optimized by semantic compensation S240 and multi-frame fusion S250, along with its corresponding timestamp information, is converted into a user-specified standard subtitle file format (such as SRT, ASS, VTT), or rendered as a text image and overlaid on the corresponding video output frame.

[0095] Changes and optimizations in method execution flow

[0096] Reference Figure 3 ,Apart from Figure 2In addition to the sequential execution flow shown, this method can also employ more flexible and intelligent execution strategies:

[0097] Selective execution process (e.g.) Figure 3 (Partial path shown in the diagram): After steps S310 (video input and detection) and S320 (initial OCR), a quality / completeness assessment step can be introduced (not labeled in the diagram, but implicit in the decision logic). This assessment is based on indicators such as the average confidence of the initial OCR results, the natural language fluency of the text sequence (e.g., perplexity assessed using a language model), and the presence of obvious signs of missing characters (e.g., abnormally large character spacing, grammatical breaks).

[0098] Based on the evaluation results, the system decides whether to perform and which subsequent enhancement steps to perform: S330 (denoising), S340 (semantic compensation), and S350 (multi-frame fusion).

[0099] For example: if the initial quality is high and there are no obvious missing characters, you can directly jump to S360 (output); if significant image noise is detected but the text is basically complete, then mainly perform S330 (denoising), and then you may re-perform OCR or output directly; if obvious signs of missing characters are detected (low fluency, structural abnormalities), then S340 (semantic compensation) must be performed; if the subtitles have temporal jitter or single-frame recognition is unstable, then S350 (multi-frame fusion) is performed to improve consistency.

[0100] Finally, the results of the selected steps are output (S360).

[0101] Dynamic adaptive process (e.g.) Figure 3 (The complete logic shown) is as follows: After S310 (input and detection) and S320 (initial OCR), an explicit decision model S327 is introduced.

[0102] The decision model S327 can be a rule-based expert system or a trained machine learning model (such as a decision tree, support vector machine, or small neural network).

[0103] Input features: The decision model receives a variety of input features, including: (1) Video metadata: resolution, frame rate, compression rate; (2) Scene / content analysis results: scene complexity, intensity of motion; (3) Detected subtitle characteristics: stability of subtitle area, font complexity (can be roughly estimated), color, contrast with background, average duration; (4) Quality indicators of initial OCR results: average character confidence, proportion of low confidence characters, preliminary estimate of the number of suspected errors / missing characters (e.g. based on simple rules or lightweight language models); (5) System resources and performance requirements: such as real-time requirements.

[0104] Decision output: Based on these inputs, the decision model S327 dynamically determines: (1) whether one or more of the adaptive denoising module S330, semantic compensation module S340, and multi-frame fusion module S350 need to be called; (2) if called, which sub-algorithms in these modules are used or which key parameters are set (e.g., the selection and strength of the denoising algorithm, the complexity of the language model in semantic compensation, the size of the time window for multi-frame fusion, and the fusion strategy).

[0105] Then, the system executes the corresponding modules (one or more combinations and configurations of S330, S340, and S350) according to the instructions of the decision model S327, and effectively integrates the results processed by each module (which may require iteration or collaborative work) to generate the final subtitles, which are then output in S360.

[0106] This dynamic adaptive workflow is relatively the most flexible and efficient, providing near-optimal processing solutions for video content and subtitle characteristics of different qualities and types, achieving a good balance between effect and resource consumption.

[0107] Further explanation of specific technologies

[0108] Adaptive subtitle denoising module 140: In specific implementation, a small model or rule set can be established to select appropriate denoising algorithm combinations and parameters based on the statistical characteristics of the input subtitle region image (such as noise variance, edge intensity, and contrast). For example, for Gaussian noise, Gaussian filtering or bilateral filtering is preferred; for salt-and-pepper noise, median filtering is more effective; for blurred subtitles, deblurring algorithms are tried. Temporal filtering (146) is very effective when the content of multiple frames of subtitles is basically the same (such as static subtitles or slowly scrolling subtitles, and they are already aligned).

[0109] Semantic Compensation Module 150: (1) Language Model Selection: The selection and training of language models (LMs) are crucial to the effect. Large-scale pre-trained models (such as BERT series, GPT series, DeepSeek series, Qwen series) can be used as the basis, and fine-tuning can be performed on video subtitle corpora in specific fields (such as news, movies, educational videos) to adapt to the specific language style and vocabulary distribution of the subtitles. (2) Missing Character Detection: In addition to analyzing the probability of the language model and the character spacing, the bounding box information of each character output by the OCR engine can also be combined. If there is a gap between the bounding boxes of adjacent characters that is much larger than the average character width, and the gap area does not correspond to any recognizable pixel pattern on the image, it may be a missing character. (3) Candidate Character Generation and Screening: When the language model generates multiple candidate characters for completion, in addition to probability, the perplexity of the entire sentence after completion, grammatical correctness (which can be connected to grammar checking tools), and relevance to the overall theme of the video (if there is theme information) can also be considered.

[0110] Multi-frame fusion module 160: (1) Frame alignment accuracy: Alignment is the key to successful multi-frame fusion. For subtitles with fast movement or complex deformation, a more robust optical flow estimation algorithm or a deep learning-based alignment network may be required. (2) Fusion weight design: The weight design of weighted fusion needs to be refined. For example, frames with high OCR confidence, high image clarity, and high compensation confidence after semantic compensation should be given higher weights. Frames that are closer in time are usually more relevant, but for periodically appearing subtitles, a longer time span may need to be considered. (3) Fusion strategy: Fusion can not only be used to fill in missing characters, but also to correct single-frame OCR errors (selecting the most consistent and highest-rated recognition result among multiple frames), and even to optimize the timestamps of subtitles (e.g., determining the precise start and end times of subtitles based on information from multiple frames).

[0111] The video subtitle extraction and character loss compensation processing device and method proposed in this invention innovatively introduces key technology modules such as adaptive subtitle denoising, powerful semantic compensation, and robust multi-frame fusion, and designs a flexible and efficient execution process (including a dynamic adaptive process). This significantly improves the shortcomings of traditional subtitle extraction techniques in handling various complex situations (especially character loss scenarios). This not only effectively improves the completeness and accuracy of the output subtitles but also enhances temporal consistency, which has significant theoretical and practical value for improving the accessibility of video information, assisting cross-language communication, and improving the user viewing experience.

[0112] It should be noted that the various embodiments described in this specification may have different focuses in their specific implementation. Features not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. The technical features in each embodiment can be combined, replaced, or added or deleted in any reasonable way according to actual application needs. As long as such combination, replacement, or addition or deletion does not conflict with the core ideas and basic principles of the present invention, it should be considered to be within the scope of disclosure of the present invention.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, or combinations of features made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. Those skilled in the art will understand that the present invention can be implemented in many other specific forms without departing from its core spirit or essential characteristics. Therefore, the embodiments and descriptions of the present invention should be considered illustrative rather than restrictive, and the present invention is not limited to the specific details given.

Claims

1. A device for compensating for missing characters in video subtitle extraction, characterized in that, include: A video input interface for receiving input video containing video data streams; The subtitle detection module is used to process the video data stream to identify one or more subtitle regions within a video frame and track the positional changes of the subtitle regions between consecutive frames; the OCR recognition module is used to perform optical character recognition on the image data within the identified subtitle regions to generate an initial subtitle text sequence, and optionally combine a language model to perform preliminary correction of potential misidentified characters in the initial subtitle text sequence. An adaptive subtitle denoising module is used to process the image data or features of the subtitle area to reduce the impact of background noise or interference on OCR recognition or subtitle processing and optimize the extraction quality of subtitle information. The semantic compensation module is used to analyze and predict possible missing characters based on the context information of the initial subtitle text sequence, and automatically fill in the missing characters according to the prediction results to generate a compensated subtitle text sequence; the multi-frame fusion module is used to analyze subtitle information from multiple temporally adjacent or related video frames to improve the completeness and temporal consistency of the final output subtitles. The subtitle output module receives the final subtitle data processed by the semantic compensation module and / or the multi-frame fusion module, converts it into one or more preset target subtitle file formats, or renders it and overlays it onto the corresponding video output frame. After the subtitle detection module and the OCR recognition module, a decision model is used to dynamically determine which modules among the adaptive subtitle denoising module, semantic compensation module, and multi-frame fusion module to call, as well as their processing strength or parameters, based on the input video quality and resolution, subtitle stability, subtitle type, subtitle contrast, and environmental noise interference. The model then integrates the processing results of each selected module to output the final subtitle data, which finally enters the subtitle output module.

2. The missing character compensation processing device for video subtitle extraction according to claim 1, characterized in that, The adaptive subtitle denoising module includes at least one or a combination of the following units: a frequency domain filtering unit, a color / space segmentation unit, and an image enhancement unit. The frequency domain filtering unit uses Fourier transform and frequency domain analysis to identify and suppress background noise at specific frequencies. The color / space segmentation unit performs adaptive and precise segmentation of the subtitle region based on the subtitle's color features, brightness contrast, or spatial layout features to enhance the separation between the subtitle and the background. The image enhancement unit improves the readability and recognizability of low-quality or low-resolution subtitle regions by applying brightness / contrast adjustment, deblurring, and super-resolution techniques.

3. The missing character compensation processing device for video subtitle extraction according to claim 1, characterized in that, The semantic compensation module is divided into a context analysis unit, a language probability analysis unit, and a semantic verification and optimization unit. The context analysis unit identifies potential missing character locations based on local or global context information of the subtitle text sequence. The language probability analysis unit uses a language model to calculate the correlation or sequence probability between adjacent characters, assesses the probability of missing characters at specific locations, and predicts and initially completes the missing characters based on this. The semantic verification and optimization unit performs secondary verification and screening of the initially completed characters based on syntactic structure, semantic coherence, and an optional external knowledge base to improve the accuracy of the completion and reduce erroneous completion.

4. The missing character compensation processing device for video subtitle extraction according to claim 1, characterized in that, The multi-frame fusion module is divided into an inter-frame feature extraction unit, a frame alignment unit, and a character / information weighted fusion unit. The inter-frame feature extraction unit is used to extract the spatial, temporal, or textual features of subtitles appearing in consecutive or related video frames. The frame alignment unit is used to align the subtitle region images or recognized text in different frames according to the extracted features to compensate for inter-frame displacement or jitter. The character / information weighted fusion unit calculates the weighted combination result based on the confidence and detection stability of the subtitle OCR characters in each frame to generate more stable and complete fused subtitle information.

5. A method for compensating for missing characters in video subtitle extraction, characterized in that, Includes the following steps: 5.1) Receive and process input video frames, identify one or more subtitle regions within the video frame through a subtitle detection process, and track the positional changes of the subtitle regions between consecutive frames; 5.2) Perform optical character recognition on the image data within the identified subtitle regions to generate an initial subtitle text sequence, and optionally combine it with a language model for preliminary correction; 5.3) Perform adaptive subtitle denoising, processing the image data or features of the subtitle regions to reduce background interference and optimize the extraction quality of subtitle information; 5.4) Apply semantic compensation, predicting and automatically completing any missing characters that may exist by analyzing the contextual information of the initial subtitle text sequence; 5.5) Perform multiple... Frame fusion analyzes subtitle information from multiple temporally adjacent or related video frames to improve the integrity and temporal consistency of the final output subtitles; 5.6) Outputs the processed final subtitle data, converting it to the target subtitle format or overlaying it onto the video output frame; After executing steps 5.1) and 5.2), a decision model is used to dynamically determine which steps in steps 5.3), 5.4), and 5.5) and their processing intensity or parameters to call based on the input video quality and resolution, subtitle stability, subtitle type, subtitle contrast, and environmental noise interference, and integrates the processing results of each selected step to generate the final subtitle data, and finally executes step 5.6).

6. The method for compensating for missing characters in video subtitle extraction according to claim 5, characterized in that, Steps 5.1) to 5.6) are executed sequentially and are suitable for handling subtitle scenarios with relatively fixed positions and stable appearances.

7. The method for compensating for missing characters in video subtitle extraction according to claim 5, characterized in that, After performing steps 5.1) and 5.2), based on the evaluation results of the initial subtitle text sequence or subtitle region image quality, one or more processing steps in steps 5.3), 5.4), and 5.5) are selectively applied, and finally step 5.6 is performed.

8. The method for compensating for missing characters in video subtitle extraction according to claim 5, characterized in that, The specific processing steps of step 5.3) adaptive subtitle denoising include at least one or a combination of the following operations: 8.1) Fourier transform or wavelet transform is used to process the subtitle region image to identify and suppress background noise in a specific frequency range; 8.2) Utilize the color characteristics of subtitles, combined with color clustering or adaptive threshold segmentation, to effectively extract pixels in the subtitle area and enhance the contrast and clarity of subtitle edges; 8.3) Apply morphological operations to optimize the shape of the subtitle area, fill internal holes or remove small noise spots, and improve the readability of low-resolution or damaged subtitles; 8.4) Apply an adaptive filtering algorithm to the subtitle area and adjust the filtering intensity according to the local image characteristics to reduce the interference of complex background textures while preserving subtitle details; 8.5) Combine multi-frame information for temporal filtering and use temporal redundancy to further suppress random noise and ensure that the denoised subtitle area has good consistency in the time dimension.

9. A method for compensating for missing characters in video subtitle extraction according to claim 5, characterized in that, The specific processing steps of semantic compensation in step 5.4) include: 9.1) Analyzing the text layout features of the current frame subtitle area and establishing a subtitle structure model; 9.2) Detecting potential missing character positions in the OCR recognition results by analyzing areas that do not conform to the expected layout or have low probability in the language model; 9.3) Analyzing the correlation strength between adjacent characters or words based on the language model and calculating the probability distribution of missing characters appearing in the detected potential positions; 9.4) Combining local or global contextual semantic information, using a text generation or prediction algorithm based on natural language processing to generate one or more candidate completion characters and their confidence scores; 9.5) Applying syntactic rules and contextual coherence checks to screen, sort, and re-evaluate the confidence scores of the candidate completion characters, and selecting the optimal completion character; 9.6) Inserting the optimal completion character into the original OCR recognition results based on its confidence score to form a compensated subtitle text sequence, ensuring its integrity and semantic rationality.

10. A method for compensating for missing characters in video subtitle extraction according to claim 5, characterized in that, The specific processing steps of step 5.5) multi-frame fusion include: 10.1) Calculating the similarity of subtitle regions in adjacent or related frames, and using an alignment algorithm to accurately align the subtitle regions or their content; 10.2) Extracting OCR recognition results from multiple consecutive frames, and calculating the confidence information of each recognized character or the entire sequence; 10.3) For missing characters or low-confidence characters detected in a single frame, retrieving corresponding character information from aligned adjacent frames as potential sources of completion or correction; 10.4) Verifying character information from different frames by combining time series analysis; 10.5) Using a weighted fusion algorithm, determining the weights based on the confidence of characters in each frame, time distance, or other relevant factors, calculating a weighted average or selecting the optimal character to generate the final fused subtitle content; 10.6) During the fusion process, using multi-frame information to smooth or remove random recognition errors or temporary missing characters, enhancing the temporal stability and overall integrity of the final subtitle.

Images