Video embedded subtitle detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and provides a video embedded subtitle detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an audio stream associated with a video stream, carrying out the voice recognition of the audio stream, and generating first time sequence data comprising a transliteration text and corresponding time information; extracting a video frame sequence from the video stream, and performing character recognition on each video frame in the video frame sequence to generate second time sequence data comprising a recognition text and corresponding position information; and performing cross-modal time sequence alignment and text content comparison based on the first time sequence data and the second time sequence data, and determining target subtitle information embedded in the video stream according to a comparison result. According to the method, the cross-modal analysis of the audio and the video is introduced, so that the scene characters in the embedded subtitles and the video images can be effectively distinguished, the false detection rate is greatly reduced, and the subtitle detection accuracy is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting embedded subtitles in videos. Background Technology

[0002] With the development of multimedia technology, embedded subtitles are widely used in various videos. Accurately detecting and locating these embedded subtitles is crucial for secondary editing of video content (such as subtitle erasure and replacement) and for providing clean training data for computer vision tasks (such as action recognition).

[0003] Existing caption detection technologies are mainly divided into two categories. One category is based on traditional image processing methods, such as locating text lines by combining edge detection with projection analysis. However, this type of method is very sensitive to video quality, caption style, and complex backgrounds. When the image is blurry, has low contrast, or the caption shape is irregular, the detection accuracy will drop significantly.

[0004] Another type is based on deep learning, typically employing a joint architecture of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) to detect scene text in a single frame of an image. While this type of method improves recognition capabilities to some extent, it still faces significant challenges. The CNN architecture struggles to capture temporal correlations across frames, and the serial computation characteristic of RNNs limits processing speed. Furthermore, neither of these methods can effectively distinguish between embedded subtitles and inherent scene text within the image (such as billboards, road signs, etc.), easily confusing the two and resulting in a high false detection rate. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for detecting embedded subtitles in videos, in order to overcome the shortcomings of the prior art in the recognition and detection of embedded subtitles in videos.

[0006] This invention provides a method for detecting embedded subtitles in videos, comprising: Acquire the audio stream associated with the video stream, and perform speech recognition on the audio stream to generate first time-series data including transcribed text and corresponding time information; Extract a video frame sequence from the video stream, and perform character recognition on each video frame in the video frame sequence to generate second time-series data including recognized text and corresponding position information; Based on the first time-series data and the second time-series data, cross-modal time-series alignment and text content comparison are performed, and the target subtitle information embedded in the video stream is determined according to the comparison results. The target subtitle information includes target recognition text and the location information corresponding to the target recognition text. The target recognition text is the recognition text located in the second time-series data that matches the transcribed text in the first time-series data.

[0007] According to a video embedded subtitle detection method provided by the present invention, the step of acquiring an audio stream associated with a video stream and performing speech recognition on the audio stream to generate first time-series data including transcribed text and corresponding time information includes: Extract the audio stream from the video stream and perform format standardization on the audio stream; Speech activity detection is performed on the processed audio stream to segment speech segments; The speech segment is processed using a speech recognition model to generate first time-series data that includes the transcribed text and the corresponding time information of the transcribed text.

[0008] According to a video embedded subtitle detection method provided by the present invention, the step of extracting a video frame sequence from the video stream and performing character recognition on each video frame in the video frame sequence to generate second temporal data including recognized text and corresponding position information includes: Based on a preset frame rate, video frames are extracted from the video stream to obtain the video frame sequence; Based on prior knowledge of subtitle position, candidate subtitle regions for each video frame in the video frame sequence are determined; An optical character recognition model is used to detect and recognize text in any video frame, obtaining multiple candidate recognition results including text content and coordinate position; The position coordinates of the multiple candidate recognition results are matched with the subtitle candidate region, and the target recognition result of any video frame is selected from the multiple candidate recognition results. The overlap between the target recognition result and the subtitle candidate region meets a preset condition. The second time-series data is generated based on the target recognition results of each video frame.

[0009] According to a video embedded subtitle detection method provided by the present invention, the step of performing cross-modal temporal alignment and text content comparison based on the first temporal data and the second temporal data, and determining the target subtitle information embedded in the video stream according to the comparison result, includes: Based on the first time series data and the second time series data, cross-modal time series alignment and text content comparison are performed, and candidate subtitle information for each video frame in the video stream is determined according to the comparison results; Based on the candidate subtitle information, the position information of embedded subtitles in multiple consecutive video frames in time sequence is checked for consistency, and the target subtitle information is determined according to the check result.

[0010] According to a video embedded subtitle detection method provided by the present invention, the step of performing cross-modal temporal alignment and text content comparison based on the first temporal data and the second temporal data, and determining candidate subtitle information for each video frame in the video stream based on the comparison results, includes: The first time series data is converted into a first time series feature vector, and the second time series data is converted into a second time series feature vector; A timing alignment algorithm is used to calculate the timing correspondence between the first timing feature vector and the second timing feature vector; Based on the time sequence correspondence, text similarity is calculated between the transcribed text in the first time sequence data and the recognized text in the second time sequence data; Based on the transcribed text and the identified text whose text similarity meets a preset similarity threshold, candidate subtitle information including the identified text and corresponding location information is determined.

[0011] According to the present invention, a method for detecting embedded subtitles in a video includes, based on the candidate subtitle information, performing consistency verification on the position information of embedded subtitles in multiple temporally consecutive video frames, comprising: The candidate caption information is tracked in multiple consecutive video frames in a temporal sequence, and the displacement of the position information in the candidate caption information between adjacent frames is calculated. If the displacement calculated in multiple consecutive frames exceeds a preset displacement threshold, the candidate subtitle information of the corresponding frame is determined to be abnormal and filtered out.

[0012] A video embedded subtitle detection method provided by the present invention further includes: If the video stream does not contain audio information, the steps of generating the first time-series data and performing cross-modal time-series alignment and text content comparison are skipped, and the identified text and corresponding position information in the second time-series data are used as the candidate subtitle information.

[0013] The present invention also provides a video embedded subtitle detection device, comprising: An audio processing unit is used to acquire an audio stream associated with a video stream, and to perform speech recognition on the audio stream to generate first time-series data including transcribed text and corresponding time information. The video processing unit is used to extract a video frame sequence from the video stream, and to perform character recognition on each video frame in the video frame sequence to generate second time-series data including the recognized text and corresponding position information. The subtitle determination unit is used to perform cross-modal temporal alignment and text content comparison based on the first temporal data and the second temporal data, and determine the target subtitle information embedded in the video stream according to the comparison result. The target subtitle information includes target recognition text and the position information corresponding to the target recognition text. The target recognition text is the recognition text located in the second temporal data that matches the transcribed text in the first temporal data.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the video embedded subtitle detection method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video embedded subtitle detection method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video embedded subtitle detection method as described above.

[0017] This invention provides a video embedded subtitle detection method, apparatus, electronic device, and storage medium that establishes a verification mechanism based on audio-text content consistency by introducing cross-modal analysis of audio and video. Specifically, this invention not only identifies all possible text from video frames using character recognition technology (i.e., second-time-series data), but more importantly, it also obtains audio-transcribed text synchronized with the video (i.e., first-time-series data) through speech recognition. By performing temporal alignment and content comparison on these two different modal data, the invention can accurately filter out text matching the audio content from the complex text information in the video frame as the true target subtitle, using the audio content as a reference. This cross-modal cross-verification method can effectively distinguish embedded subtitles from scene text in the video frame, thereby greatly reducing the false detection rate and significantly improving the accuracy of subtitle detection. Furthermore, because it comprehensively utilizes both audio and video information sources, when the information quality of a single modality is poor, the other modality can still provide a valid reference, thus enhancing the robustness of the entire detection system and its adaptability to complex video environments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the video embedded subtitle detection method provided by the present invention; Figure 2 This is a schematic diagram of the audio processing flow provided by the present invention; Figure 3 This is a schematic diagram of the video frame processing flow provided by the present invention; Figure 4 This is a flowchart illustrating the cross-modal text temporal alignment provided by the present invention; Figure 5 This is a schematic diagram of the multi-frame consistency verification process provided by the present invention; Figure 6 This is a schematic diagram of the video embedded subtitle detection device provided by the present invention; Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] With the rapid development of internet technology and multimedia applications, video data has become a core carrier of information dissemination. Among various video content, embedded subtitles (also known as hard subtitles), a technology that directly encodes and burns text information into the video frame, are widely used in movies, TV series, online education courses, and short videos. These subtitles are deeply integrated with the video image frames and cannot be independently selected, turned off, or edited by the user during playback.

[0022] To perform in-depth analysis and utilization of massive amounts of video data, such as cross-modal model training, video content retrieval, or secondary creation and editing, it is first necessary to perform structured analysis of the content elements in the video. Among these steps, accurately detecting and locating the position of embedded subtitles is crucial. Accurate subtitle location information is not only the foundation for editing operations such as subtitle erasure, replacement, or style modification, but also a key prerequisite for purifying visual data and avoiding feature confusion and semantic noise interference from subtitle text in computer vision tasks (such as action recognition and scene understanding). Failure to effectively separate subtitles from the background will severely impact the performance and reliability of subsequent advanced visual algorithms.

[0023] Currently, the detection technologies for embedded subtitles in videos can be mainly divided into two categories: The first category is detection methods based on traditional image processing. These methods typically utilize the inherent physical features of the subtitle text region for localization. A typical workflow includes: first, preprocessing the video frames, such as grayscale conversion and binarization; then, using edge detection operators (such as Canny and Sobel) to extract high-frequency edge information from the image; finally, through horizontal or vertical projection analysis, searching for peaks matching the text line characteristics on the projection histogram to determine candidate subtitle regions.

[0024] The second category is detection methods based on deep learning. With the development of artificial intelligence technology, end-to-end scene text detection and recognition models have been applied to caption detection tasks. These methods typically use convolutional neural networks (CNNs) to extract deep spatial features of images, and then combine them with recurrent neural networks (RNNs) to capture the temporal dependencies of text sequences. They also use mechanisms such as connectionist temporal classification (CTC) to solve the sequence alignment problem, thereby achieving automatic recognition of caption regions within a single frame of an image.

[0025] However, in practical applications, it has been found that the above detection methods still have certain shortcomings. First, traditional image processing methods have poor environmental adaptability. These methods rely heavily on manually designed features and fixed thresholds, making them very sensitive to the complexity of video images. When the video has motion blur, low resolution, low contrast, or when the subtitle text has artistic fonts, perspective distortion, or non-horizontal arrangement, the effectiveness of edge detection and projection analysis will drop sharply, leading to insufficient positioning accuracy or even failure.

[0026] Secondly, single-modal visual analysis methods based solely on deep learning have inherent limitations. While deep learning methods improve text detection capabilities within a single frame, they still face significant challenges: First, they cannot effectively distinguish between target subtitles and scene text. Besides embedded subtitles used for narration or dialogue, video footage may also contain text inherent to the scene, such as billboards, banners, and road signs. Detection methods relying solely on visual features struggle to differentiate between the two semantically and functionally, leading to numerous false positives and misidentifying non-subtitle scene text as target subtitles. Second, they fail to fully utilize the inherent cross-modal correlation information in video. The content of embedded subtitles is often highly synchronized temporally and semantically with the video's audio stream (such as human dialogue). Traditional methods mostly treat video as an independent sequence of image frames, ignoring audio as an important information source, resulting in limited robustness and accuracy.

[0027] To address this, the present invention provides a method for detecting embedded subtitles in videos, thereby overcoming the aforementioned shortcomings. This method can be applied to various scenarios such as training data preprocessing for computer vision models, video content analysis, and multilingual video translation; however, the embodiments of the present invention do not specifically limit its application.

[0028] Figure 1 This is a flowchart illustrating the video embedded subtitle detection method provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step 110: Obtain the audio stream associated with the video stream, and perform speech recognition on the audio stream to generate first time-series data including transcribed text and corresponding time information.

[0029] Specifically, a video stream refers to a digital data stream containing a series of image frames arranged in chronological order, with common formats such as MP4, AVI, MKV, and MOV. The audio stream associated with the video stream refers to the audio track played synchronously with the video. In most multimedia container formats (such as MP4), the audio and video streams are encapsulated as different data tracks within the same file container. Of course, in some applications, the audio stream can also be an independent audio file separate from the video file but with a temporal correspondence (e.g., WAV, MP3, AAC formats). The purpose of this step is to convert the speech content in the audio into timestamped text, providing a core basis for subsequent cross-modal verification.

[0030] Specifically, after acquiring the audio stream, it can be processed using Automatic Speech Recognition (ASR) technology. The core function of speech recognition technology is to convert speech signals into computer-readable text sequences. In this embodiment of the invention, any mature speech recognition engine or model can be used. For example, end-to-end models based on deep learning, such as the Conformer model and the Whisper model, can be used. These models can directly map the original audio waveform to a text sequence and output the time information corresponding to each word or sentence.

[0031] After processing, the generated first-time-series data is structured data. This data binds the text content recognized by speech (i.e., the transcribed text) to the time periods in which the text appears in the original audio stream (i.e., the corresponding time information). For example, the first-time-series data can be a collection of data entries, each with the format {start time, end time, transcribed text}. For instance, a first-time-series data entry could be {00:01:05.250, 00:01:07.800, "The weather is really nice today"}, indicating that the audio content is "The weather is really nice today" between 1 minute 5 seconds 250 milliseconds and 1 minute 7 seconds 800 milliseconds in the video.

[0032] Step 120: Extract a video frame sequence from the video stream, and perform character recognition on each video frame in the video frame sequence to generate second time-series data including recognized text and corresponding position information.

[0033] Specifically, this step aims to extract all possible text information from the video footage. First, a series of video frames, i.e., static images, needs to be extracted from the video stream. Frame extraction can be performed at a fixed frame rate (e.g., 1, 3, or 5 frames per second) or using a dynamic strategy (e.g., intelligent frame extraction based on scene changes) to form a video frame sequence. Each frame retains its timestamp information within the video for temporal alignment with the audio information.

[0034] Subsequently, character recognition is performed on each frame of the video frame sequence. This character recognition can be Optical Character Recognition (OCR). OCR technology can detect text regions in an image and recognize the text content within them. In this embodiment of the invention, a general-purpose OCR engine, such as Tesseract or PaddleOCR, can be used, or a dedicated OCR model optimized for video subtitle scenarios can be employed. The character recognition process scans each frame of the image, identifies all regions containing text, and outputs the content of this text and its position within the frame.

[0035] After processing, the generated second time-series data is also structured. This data records what text appeared at which location and time point (corresponding to a specific video frame). For example, the second time-series data can be a collection of data entries, each in the format {frame number / timestamp, recognized text, corresponding location information}. The recognized text is the text content identified by the OCR engine from the image. The corresponding location information describes the precise location of the text in the image coordinate system, which can be the coordinates of the four vertices of the bounding box (x1, y1, x2, y2) or the polygonal outline coordinates of the text region. A single frame may contain multiple text regions due to the presence of road signs, advertising slogans, scene text, etc. Therefore, multiple recognition results may correspond to a single frame (or a single timestamp).

[0036] Step 130: Based on the first time-series data and the second time-series data, perform cross-modal time-series alignment and text content comparison, and determine the target subtitle information embedded in the video stream according to the comparison results. The target subtitle information includes target recognition text and the location information corresponding to the target recognition text. The target recognition text is the recognition text located in the second time-series data that matches the transcribed text in the first time-series data.

[0037] Specifically, this step achieves accurate identification of embedded subtitles by integrating information from both audio and video modalities, effectively distinguishing genuine subtitles from other irrelevant text in the image.

[0038] Cross-modal temporal alignment refers to matching first-series data (from audio) and second-series data (from video) along the timeline. The basic principle is that a piece of speech and its corresponding embedded subtitles are highly synchronized in time. By comparing the time information in the first-series data and the timestamps in the second-series data, speech-to-text and image recognition text that co-occur within similar time periods can be identified.

[0039] Text content comparison, building upon temporal alignment, further compares the similarity in content between the speech-to-text and the image-recognized text. For example, if at time point T, the speech recognition result is "The weather is really nice today," and simultaneously, in the lower area of ​​the video frame corresponding to time point T, the OCR recognition result is also "The weather is really nice today," then it can be highly confident that the OCR recognition result is the genuine embedded subtitle.

[0040] Based on the comparison results, the target subtitle information is finally determined. Target recognized text refers to the recognized text derived from the second time-series data that matches the audio-transcribed text in content. Therefore, the target subtitle information includes this target recognized text and its corresponding position information in the image. Through this step, texts that are recognized by OCR in the image but do not match the synchronous audio in content (such as advertising slogans, scoreboards, road signs, and other scene text) are effectively filtered out, thus retaining only the embedded subtitles that match the audio content and their positions.

[0041] The method provided in this invention establishes a verification mechanism based on audio-text content consistency by introducing cross-modal analysis of audio and video. Specifically, this invention not only identifies all possible text from video frames using character recognition technology (i.e., second temporal data), but more importantly, it also obtains audio-transcribed text synchronized with the video (i.e., first temporal data) through speech recognition. By performing temporal alignment and content comparison on these two different modal data, the method can accurately filter out text matching the speech content from the complex text information in the video frame as the true target subtitle, using the audio content as a reference. This cross-modal verification method can effectively distinguish embedded subtitles from scene text in the video frame, thereby greatly reducing the false detection rate and significantly improving the accuracy of subtitle detection. Furthermore, since it comprehensively utilizes both audio and video information sources, when the information quality of a single modality is poor, the other modality can still provide a valid reference, thereby enhancing the robustness of the entire detection system and its adaptability to complex video environments.

[0042] Based on the above embodiments, Figure 2 This is a schematic diagram of the audio processing flow provided by the present invention, such as... Figure 2 As shown, step 110 specifically includes: Step 111: Extract the audio stream from the video stream and perform format standardization processing on the audio stream.

[0043] Specifically, the first step is to extract (also known as demultiplex) the audio stream from the input video stream. This can be accomplished using mature multimedia processing frameworks, such as FFmpeg or GStreamer. These tools can parse various mainstream video file containers, such as MP4, AVI, and MKV, and separate the encapsulated audio tracks.

[0044] After extracting the audio stream, it is usually necessary to perform format normalization. This is because different video sources may use different audio encodings (such as AAC, MP3, AC3, etc.), sampling rates (such as 44.1kHz, 48kHz, etc.), bit depths, or number of channels. Subsequent speech activity detection and speech recognition models often have specific requirements for the input format. Therefore, normalization is a crucial step in ensuring system stability and recognition accuracy. For example, audio codec libraries (such as libavcodec) can be used to decode various audio formats and uniformly convert them to a 16kHz sampling rate, 16-bit bit depth, mono linear PCM (Pulse-Code Modulation) format. Of course, these specific parameters can be adjusted according to the optimal input requirements of the selected subsequent models, and this embodiment of the invention does not impose specific limitations on this.

[0045] Step 112: Perform speech activity detection on the processed audio stream to segment speech segments.

[0046] It's important to note that the purpose of this step is to accurately identify segments containing human speech (i.e., speech segments) from a continuous audio stream and distinguish them from non-speech segments such as silence, background noise, or instrumental music. This approach has two main advantages: first, it significantly reduces computational costs, as expensive speech recognition models do not need to process the entire audio stream, only the segments containing speech; second, it effectively reduces interference from non-speech signals on the speech recognition results, thereby improving transcription accuracy.

[0047] Specifically, a deep learning-based end-to-end Voice Activity Detection (VAD) model can be used. For example, the WebRTC VAD model or the Silo-VAD model can be used. These models typically combine traditional acoustic features (such as short-time energy and zero-crossing rate) with modern neural network architectures (such as Long Short-Term Memory networks, LSTM), enabling fast and accurate detection of the start and end boundaries of speech. Through this step, the complete audio stream is segmented into a series of speech segments with start and end times.

[0048] Step 113: The speech segment is processed using a speech recognition model to generate first time-series data including the transcribed text and the corresponding time information of the transcribed text.

[0049] Specifically, after obtaining clean speech segments, this step involves transcribing and timestamping these segments. Each segmented speech is used as input and fed into a high-performance Automatic Speech Recognition (ASR) model. State-of-the-art ASR models, such as the Conformer or Whisper models, can be selected. These models not only have high transcription accuracy but, more importantly, provide timestamp information accurate to the word or sentence level. After processing, the model generates a corresponding transcribed text for each input speech segment, annotating the precise time information (start and end times) of that text (or its words) within the original audio stream.

[0050] Finally, by summarizing the processing results of all speech segments, a structured first time-series data is generated, which is a set of {start time, end time, transcribed text}, providing a high-quality, audio-derived data foundation for subsequent cross-modal alignment steps.

[0051] The method provided in this invention first ensures data consistency through standardization, then efficiently filters irrelevant audio information using VAD technology, and finally employs an advanced ASR model to accurately transcribe and time-annotate effective speech segments. This not only improves overall processing efficiency but also lays a solid foundation for obtaining high-precision subtitle information in subsequent steps, ensuring that the information extracted from the audio modality is both accurate and timely.

[0052] Based on any of the above embodiments Figure 3 This is a schematic diagram of the video frame processing flow provided by the present invention, such as... Figure 3 As shown, step 120 specifically includes: Step 121: Based on a preset frame rate, extract video frames from the video stream to obtain the video frame sequence.

[0053] Specifically, the purpose of this step is to discretize the dynamic video stream into a series of static images that can be analyzed independently. The preset frame rate refers to the number of image frames extracted from the video per second. Setting this value requires a trade-off between processing efficiency and information integrity. If the frame rate is too high, it will generate a large number of redundant frames, increasing the computational burden; if the frame rate is too low, some quickly flashing subtitles may be missed. In practical applications, this frame rate is usually set between 1fps and 5fps, and can also be flexibly set according to actual needs. For example, setting it to 3fps means that 3 image frames are uniformly extracted from the video per second. When extracting each frame, its timestamp in the original video must be recorded, which is the basis for subsequent time-series alignment with audio information. After this step, a video frame sequence with timestamps is obtained.

[0054] Step 122: Based on prior knowledge of subtitle position, determine the candidate subtitle region for each video frame in the video frame sequence.

[0055] Specifically, this step aims to leverage industry practices and statistical patterns to significantly narrow down the processing scope of subsequent text recognition, thereby improving efficiency and reducing the false recognition rate. Prior knowledge of subtitle position refers to empirical patterns regarding the common locations of embedded subtitles in the frame, derived from the analysis of a large amount of video content. For example, statistics show that over 80% of video subtitles appear in the bottom 20% horizontal area of ​​the frame; in some bilingual videos, subtitles in another language may also appear simultaneously in the top 20% area of ​​the frame.

[0056] Therefore, this step can pre-define one or more candidate caption regions when processing each frame of the image. This region can be one or more rectangular boxes; for example, a box covering the bottom 20% of the image and a box covering the top 20% of the image can be defined simultaneously. Subsequent text recognition will primarily be performed within these candidate regions, or the results of full-image recognition can be filtered based on these regions.

[0057] Step 123: Use an optical character recognition model to detect and recognize text in any video frame to obtain multiple candidate recognition results, including text content and coordinate position.

[0058] Specifically, for each frame in the video frame sequence (i.e., any video frame), an optical character recognition (OCR) model is invoked for processing. For example, OCR models such as PaddleOCR or Tesseract can be used. To handle complex situations such as small font size, low contrast, and diverse fonts that may occur in the video, specially optimized or improved OCR models can also be used.

[0059] OCR processing typically involves two stages: text detection (locating all regions in an image that may contain text) and text recognition (recognizing the content of the located regions). Since a single video frame may contain other text besides the target subtitle, such as banners, advertising slogans, scoreboards, and signs within the scene, the processing result of a single frame by an OCR model is usually a collection of multiple candidate recognition results. Each candidate recognition result includes the recognized text content and its coordinate position in the image (e.g., bounding box coordinates).

[0060] Step 124: Match the position coordinates of the multiple candidate recognition results with the subtitle candidate region, and filter out the target recognition result of any video frame from the multiple candidate recognition results. The overlap between the target recognition result and the subtitle candidate region meets the preset conditions.

[0061] Specifically, this step refines the rough results obtained in the previous step, filtering out most of the non-subtitle interference text. The matching process compares the position coordinates of each candidate recognition result obtained in step 123 with the subtitle candidate region determined in step 122 in terms of spatial location.

[0062] The comparison is based on overlap, a metric that measures the degree of overlap between two regions. In computer vision, the Intersection over Union (IoU) ratio is commonly used to calculate overlap. A preset IoU threshold is set, for example, 50%. Specifically, for a text box recognized by OCR, its IoU value is calculated with a pre-defined candidate caption region (e.g., the bottom 20% of the screen). If the value is greater than 50%, the text box is considered highly likely to be a caption and is retained as the target recognition result for that video frame; if the IoU value is less than 50%, it is considered to be located in a non-caption region (e.g., the center of the screen) and is considered interference information, which should be discarded. In this way, advertising slogans, scene text, etc., located in non-caption positions on the screen can be effectively filtered out.

[0063] Step 125: Generate the second time-series data based on the target recognition results of each video frame.

[0064] Specifically, this step involves compiling the filtered, high-quality single-frame recognition results into a complete time-series data. The target recognition results (including frame timestamps, recognized text, and location coordinates) obtained from each frame in the video frame sequence after filtering in step 124 are organized in chronological order to form structured second time-series data for use in subsequent cross-modal alignment steps.

[0065] The method provided in this invention, by pre-setting candidate regions and filtering based on overlap, can efficiently and accurately complete preliminary screening before cross-modal comparison. This not only effectively reduces the computational load of subsequent processing but also initially filters out a large amount of scene text interference unrelated to subtitles, thereby providing cleaner and more relevant visual text data for subsequent high-precision cross-modal alignment and improving the performance of the entire subtitle detection system.

[0066] Based on any of the above embodiments, step 130 specifically includes: Step 131: Based on the first time-series data and the second time-series data, perform cross-modal time-series alignment and text content comparison, and determine the candidate subtitle information for each video frame in the video stream according to the comparison results.

[0067] It should be noted that, in order to distinguish between real subtitles and interfering text in the video and to ensure that the final result is stable and reliable, the embodiments of the present invention can be divided into two main stages: the cross-modal alignment and comparison stage and the multi-frame consistency verification stage.

[0068] Specifically, this step involves using audio information to verify the text identified in the video frames, obtaining a highly reliable intermediate result: candidate caption information. Candidate caption information refers to visual text and its location information that highly match the audio speech in both time and content. Please refer to... Figure 4 This is a schematic diagram of the cross-modal text temporal alignment process provided by the present invention, as shown below. Figure 4 As shown, this step can be specifically implemented as follows: Step 1311: Convert the first time series data into a first time series feature vector, and convert the second time series data into a second time series feature vector; Step 1312: The temporal correspondence between the first temporal feature vector and the second temporal feature vector is calculated using a temporal alignment algorithm; Step 1313: Based on the time sequence correspondence, perform text similarity calculation on the transcribed text in the first time sequence data and the recognized text in the second time sequence data; Step 1314: Based on the transcribed text and the identified text whose text similarity meets a preset similarity threshold, determine the candidate subtitle information including the identified text and corresponding location information.

[0069] Specifically, in order to facilitate algorithm processing, the structured data generated in the preceding steps needs to be converted into numerical time-series feature vectors.

[0070] For the first time-series data (derived from audio), it can be viewed as a time series, where each element contains {start time, end time, transcribed text}. Simply use time as the axis, treating each text segment as an event point or time period. Its features can include time information and the vectorized representation of the text (such as word embedding vectors converted by models like BERT).

[0071] The second time-series data (derived from video) can also be viewed as a time series, with each element containing {frame number / timestamp, recognized text, location information}. Its feature vector can include the timestamp and a vectorized representation of the text content.

[0072] Because the rhythm, pauses, and frame skipping intervals of speech can cause two sequences to not correspond one-to-one on the timeline, an alignment algorithm that can tolerate local temporal distortion is needed. In this embodiment of the invention, Dynamic Time Warping (DTW) algorithm can be used. DTW is a method for measuring the similarity between two time series of different lengths. By constructing a distance matrix (the matrix elements can be the similarity or distance of text features at two time points) and using dynamic programming to find a path through the matrix with the minimum total cost, the optimal temporal correspondence between the two modal data is established.

[0073] After finding temporal matching pairs (i.e., speech within a certain time period corresponding to certain image frames) using algorithms such as DTW, further verification of content matching is needed. For each matching pair, the transcribed text (from audio) and recognized text (from video) are extracted, and their text similarity is calculated. Similarity calculation can employ various methods, such as edit distance, which calculates the minimum number of single-character edits (insertion, deletion, or replacement) required to convert one string into another; or semantic similarity, which uses models such as Word2Vec and BERT to convert the text into vectors and then calculates the cosine similarity between the vectors.

[0074] Set a preset similarity threshold (e.g., edit distance similarity greater than 0.8, or cosine similarity greater than 0.9). For each text pair, if its calculated similarity exceeds this threshold, the text recognized by OCR is considered a genuine subtitle synchronized with the speech. In this case, the recognized text and its corresponding location information are confirmed as a candidate subtitle. Conversely, if the similarity does not meet the threshold, the OCR recognition result is considered a false positive or irrelevant scene text and is discarded. Through this step, a highly reliable list of candidate subtitles, validated through audio modal cross-validation, is obtained.

[0075] Step 132: Based on the candidate subtitle information, perform consistency verification on the position information of embedded subtitles in multiple consecutive video frames in time sequence, and determine the target subtitle information based on the verification result.

[0076] Specifically, the second stage of processing takes place. This step leverages the spatial continuity of subtitles over time to refine and filter the candidate results, eliminating any potential, momentary false positives. Typically, a subtitle is displayed continuously on the screen for several frames, its position remaining stable or moving slowly (e.g., scrolling subtitles), without abrupt changes between adjacent frames. Please refer to [link / reference]. Figure 5 This is a schematic diagram of the multi-frame consistency verification process provided by the present invention, as shown below. Figure 5 As shown, the verification process can be specifically implemented as follows: Step 1321: Track the candidate caption information in multiple consecutive video frames in time sequence, and calculate the displacement of the position information in the candidate caption information between adjacent frames; Step 1322: If the displacement calculated in multiple consecutive frames exceeds a preset displacement threshold, the candidate subtitle information of the corresponding frame is determined to be abnormal and filtered out.

[0077] Specifically, first, the candidate caption information sequence generated in step 131 is processed. Multiple temporally consecutive video frames are traced, for example, considering a sliding window containing N frames (e.g., N=5). For each pair of adjacent frames within the window (e.g., frame i and frame i+1), if they both contain candidate caption information, the displacement between these two caption positions is calculated. The displacement can be calculated as the Euclidean distance between the center points of the caption bounding box, or the average displacement of the corresponding vertex coordinates of the bounding box.

[0078] Set a reasonable preset displacement threshold (e.g., a distance of a few pixels). When analyzing multiple consecutive frames within a sliding window, if the displacement of the subtitle position in a certain frame (e.g., frame i+1) relative to the previous frame (frame i) suddenly exceeds this threshold, it indicates that a positional abrupt change may have occurred. It's important to note that anomaly detection is not based solely on two frames, but rather on the context of multiple consecutive frames. For example, if the position of frame i+1 differs significantly from both frames i and i+2, then the candidate subtitle information in frame i+1 can be identified as an anomaly and filtered out. This process effectively removes skipped subtitles caused by momentary OCR recognition errors (e.g., a frame misrecognizing text in other positions as text matching the speech), ensuring the temporal stability and continuity of the final output subtitle position.

[0079] After the above two stages (i.e., steps 131 and 132), the remaining candidate subtitle information is the final, high-precision target subtitle information.

[0080] This invention, through cross-modal alignment and text content comparison, utilizes audio information as a strong reference to achieve the first critical screening of visual text information, greatly improving the accuracy of subtitle recognition. Building upon this, a multi-frame consistency verification mechanism is introduced, leveraging the spatiotemporal continuity of subtitles to further refine the screening results, effectively filtering out transient noise and abnormal detection points. These two stages complement each other, jointly constructing a robust and accurate subtitle detection process, resulting in highly reliable target subtitle information across content, time, and space dimensions.

[0081] Based on any of the above embodiments, the method further includes: If the video stream does not contain audio information, the steps of generating the first time-series data and performing cross-modal time-series alignment and text content comparison are skipped, and the identified text and corresponding position information in the second time-series data are used as the candidate subtitle information.

[0082] It should be noted that in practical applications, some video files are inherently silent, such as some surveillance recordings, early silent films, or videos whose audio tracks were removed or damaged during production. To address this, this invention provides a degradation processing mechanism.

[0083] Specifically, when this method begins execution, it first checks whether the input video stream contains available audio tracks or associated audio information. If the video stream does not contain audio information, the system automatically switches to a vision-first processing mode. In this mode, the audio processing flow is skipped, meaning all steps related to generating the first time-series data are completely skipped. Consequently, the cross-modal temporal alignment and text content comparison steps, which rely on the first time-series data, are also skipped. Because of the lack of reference information from the audio modality, temporal alignment and text similarity comparison cannot be performed.

[0084] In this scenario, the system will rely entirely on the processing results of purely visual information. That is, the system will execute the entire video frame processing flow. The second temporal data generated by the aforementioned visual processing flow (containing recognized text and corresponding positional information) will be directly considered as candidate caption information for subsequent steps. This second temporal data, considered candidate caption information, will continue to be used as input in the multi-frame consistency verification step. The system will still further refine the results by tracking the stability of caption positions in consecutive frames and filtering out jump captions with abnormal displacement.

[0085] The method provided in this invention can still improve the accuracy of subtitle detection even under the challenge of lacking audio modal information by using location-based prior knowledge filtering and multi-frame spatiotemporal consistency verification. Its performance is still better than traditional methods that only perform simple OCR recognition on a single frame image.

[0086] Based on any of the above embodiments, this invention provides an efficient and accurate method for automatically detecting embedded subtitles in videos. This method uses multimodal fusion technology to achieve intelligent recognition of subtitle positions. The implementation process is as follows: First, the audio processing module extracts the audio track signal from the video. The audio stream is segmented into speech segments and non-speech segments using a Voice Activity Detection (VAD) algorithm. A deep learning-based speech recognition model (such as Conformer or Whisper) is used to accurately timestamp and transcribe each speech segment, generating a time-stamped text sequence. Simultaneously, the video processing module decomposes the video stream into a continuous sequence of image frames at a preset frame rate (usually 1-5 fps). Combined with prior knowledge of subtitle positions (e.g., 80% of subtitles are located in the lower 20% of the screen), the initial search range is determined. An improved OCR engine (such as PaddleOCR or Tesseract) is used to perform multi-scale text detection and recognition on the candidate regions, with optimized recognition accuracy for complex scenarios such as small font sizes and low contrast. In the text alignment stage, a Dynamic Time Warping (DTW) algorithm is constructed to match the OCR recognition results with the transcribed text along the timeline. Finally, through multi-frame consistency verification, the spatiotemporal continuity of subtitles between adjacent frames is analyzed to achieve precise positioning and tracking of the embedded subtitle region. The specific steps are as follows: Step S1, Audio Processing Steps: This step employs multimodal fusion technology to extract and transcribe the audio from the video, serving as a crucial basis for determining the embedded subtitle region. If the video itself does not contain audio information, this step can be ignored.

[0087] Step S11, Audio Extraction and Format Standardization: Use multimedia frameworks such as FFmpeg or GStreamer to demultiplex and extract audio tracks from video files, supporting common formats such as MP4, AVI, and MKV. Use libavcodec to decode the audio and convert it into a 16kHz sampling rate, 16-bit depth, mono linear PCM format to ensure the standardization of ASR model input. S12, Speech Activity Detection and Segmentation: An end-to-end VAD model based on deep learning (such as WebRTC's VAD or Silero-VAD) is used to detect speech boundaries by combining short-time energy / zero-crossing rate with LSTM, thereby segmenting the audio stream into speech segments and non-speech segments; S13, ASR transcription and temporal alignment: Use ASR models such as Conformer or Whisper to accurately timestamp each speech segment and transcribe the content into text. S14, Structured Data Generation: Generate structured audio processing labeled data, such as {start time, end time, transcribed text}.

[0088] S2, Video frame image processing steps: This step processes video frames, combining traditional image processing methods with deep learning methods, and effectively reduces processing time and false recognition rate based on prior knowledge; in addition, a series of comprehensive anti-interference mechanisms are introduced to further improve the detection rate and effectiveness of subtitle areas.

[0089] S21, Video frame extraction: Extract video frames according to a preset frame rate, decode the video stream into a continuous RGB image frame sequence, and retain the timestamp information corresponding to each frame. S22, Subtitle Area Location: First, based on prior knowledge, define the subtitle search area bounding box (such as the upper 20% area of ​​the screen and the lower 20% area of ​​the screen). S23, OCR recognition and search region matching: Use OCR models such as PaddleOCR or Tesseract to perform multi-scale text detection and recognition on each image frame, and output the recognition result: {text content, coordinate position box}. Match the multiple candidate coordinate position boxes output with the search range region box. If the overlap rate (IoU) exceeds 50%, it is retained; otherwise, it is discarded, thereby eliminating irrelevant text interference. S24, Structured Data Generation: Generate labeled data after processing of structured video frames, such as {frame number, coordinate bounding box, recognized text}.

[0090] S3, cross-modal text temporal alignment steps: Based on the structured output results of steps S1 and S2, the labeled data generated from audio processing and video processing are converted into time-series feature vectors, respectively. Then, a Dynamic Time Warping (DTW) distance matrix is ​​constructed, and a dynamic programming algorithm is used for optimal path search and backtracking to establish the optimal temporal correspondence between the two modalities. After alignment, the similarity between the Automatic Speech Recognition (ASR) transcribed text and the Optical Character Recognition (OCR) results is compared. Matches with similarity exceeding a preset threshold are retained, while the rest are discarded.

[0091] S4, Multi-frame Consistency Check: Considering that the position of subtitle text in adjacent video frames does not change significantly, the displacement of the subtitle region in adjacent frames is calculated based on the subtitle position coordinate data output from the timing alignment step by tracking the subtitle region position between consecutive frames. Abnormal frames with displacement exceeding a set threshold are filtered to ensure the continuity and stability of the subtitle spatial position.

[0092] Compared with traditional image processing and detection methods based on deep learning, this invention has the following advantages: (1) Traditional image processing methods, such as edge detection algorithms and projection analysis methods, are affected by image noise and character deformation or incompleteness, resulting in low subtitle recognition rates. This invention first determines the initial search range by combining prior knowledge of subtitle position, and then uses an improved deep learning model-based OCR engine to perform multi-scale text detection and recognition on candidate regions, effectively improving the accuracy of subtitle recognition in complex scenarios such as low image contrast, character deformation or incompleteness, and small font size.

[0093] (2) Although deep learning-based solutions improve the accuracy of subtitle recognition for single-frame images compared to traditional image processing methods, they still have problems such as difficulty in capturing temporal correlations across frames and insufficient adaptability to changes in character spatial distribution, resulting in a high rate of false subtitle detection. This invention proposes cross-modal comprehensive analysis capabilities, utilizing audio and frame images in the video to complement each other's information, and matching the OCR recognition results with the speech-to-text text on the time axis, thereby significantly reducing the probability of false subtitle detection.

[0094] (3) The present invention also proposes a method for multi-frame consistency verification. Taking into account the characteristic that the position of subtitle text in adjacent video frames does not change significantly, the probability of false detection of subtitle text is further reduced by real-time consistency verification of subtitle position in adjacent multiple frames.

[0095] The video embedded subtitle detection device provided by the present invention is described below. The video embedded subtitle detection device described below can be referred to in correspondence with the video embedded subtitle detection method described above.

[0096] Based on any of the above embodiments Figure 6 This is a schematic diagram of the video embedded subtitle detection device provided by the present invention, as shown below. Figure 6 As shown, the device includes: The audio processing unit 610 is used to acquire an audio stream associated with the video stream, and to perform speech recognition on the audio stream to generate first time-series data including transcribed text and corresponding time information. The video processing unit 620 is used to extract a video frame sequence from the video stream, and perform character recognition on each video frame in the video frame sequence to generate second time-series data including recognized text and corresponding position information. The subtitle determination unit 630 is used to perform cross-modal temporal alignment and text content comparison based on the first temporal data and the second temporal data, and determine the target subtitle information embedded in the video stream according to the comparison result. The target subtitle information includes target recognition text and the position information corresponding to the target recognition text. The target recognition text is the recognition text located in the second temporal data that matches the transcribed text in the first temporal data.

[0097] The apparatus provided in this invention establishes a verification mechanism based on audio-text content consistency by introducing cross-modal analysis of audio and video. Specifically, this invention not only identifies all possible text from video frames using character recognition technology (i.e., second temporal data), but more importantly, it also obtains audio-transcribed text synchronized with the video (i.e., first temporal data) through speech recognition. By performing temporal alignment and content comparison on these two different modal data, the invention can accurately filter out text matching the speech content from the complex text information in the video frame as the true target subtitle, using the audio content as a reference. This cross-modal verification method can effectively distinguish embedded subtitles from scene text in the video frame, thereby greatly reducing the false detection rate and significantly improving the accuracy of subtitle detection. Furthermore, since it comprehensively utilizes both audio and video information sources, when the information quality of a single modality is poor, the other modality can still provide a valid reference, thereby enhancing the robustness of the entire detection system and its adaptability to complex video environments.

[0098] Based on any of the above embodiments, the audio processing unit 610 is specifically used for: Extract the audio stream from the video stream and perform format standardization on the audio stream; Speech activity detection is performed on the processed audio stream to segment speech segments; The speech segment is processed using a speech recognition model to generate first time-series data that includes the transcribed text and the corresponding time information of the transcribed text.

[0099] Based on any of the above embodiments, the video processing unit 620 is specifically used for: Based on a preset frame rate, video frames are extracted from the video stream to obtain the video frame sequence; Based on prior knowledge of subtitle position, candidate subtitle regions for each video frame in the video frame sequence are determined; An optical character recognition model is used to detect and recognize text in any video frame, obtaining multiple candidate recognition results including text content and coordinate position; The position coordinates of the multiple candidate recognition results are matched with the subtitle candidate region, and the target recognition result of any video frame is selected from the multiple candidate recognition results. The overlap between the target recognition result and the subtitle candidate region meets a preset condition. The second time-series data is generated based on the target recognition results of each video frame.

[0100] Based on any of the above embodiments, the subtitle determination unit 630 includes: The comparison subunit is used to perform cross-modal temporal alignment and text content comparison based on the first temporal data and the second temporal data, and determine the candidate subtitle information of each video frame in the video stream according to the comparison results; The verification subunit is used to perform consistency verification on the position information of embedded subtitles in multiple consecutive video frames in time based on the candidate subtitle information, and determine the target subtitle information based on the verification result.

[0101] Based on any of the above embodiments, the comparison subunit is specifically used for: The first time series data is converted into a first time series feature vector, and the second time series data is converted into a second time series feature vector; A timing alignment algorithm is used to calculate the timing correspondence between the first timing feature vector and the second timing feature vector; Based on the time sequence correspondence, text similarity is calculated between the transcribed text in the first time sequence data and the recognized text in the second time sequence data; Based on the transcribed text and the identified text whose text similarity meets a preset similarity threshold, candidate subtitle information including the identified text and corresponding location information is determined.

[0102] Based on any of the above embodiments, the verification subunit is specifically used for: The candidate caption information is tracked in multiple consecutive video frames in a temporal sequence, and the displacement of the position information in the candidate caption information between adjacent frames is calculated. If the displacement calculated in multiple consecutive frames exceeds a preset displacement threshold, the candidate subtitle information of the corresponding frame is determined to be abnormal and filtered out.

[0103] Based on any of the above embodiments, the device further includes a skipping unit, the skipping unit being used for: If the video stream does not contain audio information, the steps of generating the first time-series data and performing cross-modal time-series alignment and text content comparison are skipped, and the identified text and corresponding position information in the second time-series data are used as the candidate subtitle information.

[0104] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a video embedded subtitle detection method. The method includes: acquiring an audio stream associated with a video stream, performing speech recognition on the audio stream, and generating first time-series data including transcribed text and corresponding time information; extracting a video frame sequence from the video stream, and performing character recognition on each video frame in the video frame sequence, generating second time-series data including recognized text and corresponding position information; performing cross-modal time-series alignment and text content comparison based on the first time-series data and the second time-series data, and determining target subtitle information embedded in the video stream according to the comparison result. The target subtitle information includes target recognized text and the position information corresponding to the target recognized text. The target recognized text is the recognized text located in the second time-series data that matches the transcribed text in the first time-series data.

[0105] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video embedded subtitle detection method provided by the above methods. The method includes: acquiring an audio stream associated with a video stream and performing speech recognition on the audio stream to generate first time-series data including transcribed text and corresponding time information; extracting a video frame sequence from the video stream and performing character recognition on each video frame in the video frame sequence to generate second time-series data including recognized text and corresponding position information; performing cross-modal time-series alignment and text content comparison based on the first time-series data and the second time-series data, and determining target subtitle information embedded in the video stream according to the comparison result. The target subtitle information includes target recognized text and position information corresponding to the target recognized text. The target recognized text is the recognized text located in the second time-series data that matches the transcribed text in the first time-series data.

[0107] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the video embedded subtitle detection method provided by the above methods. The method includes: acquiring an audio stream associated with a video stream, and performing speech recognition on the audio stream to generate first time-series data including transcribed text and corresponding time information; extracting a video frame sequence from the video stream, and performing character recognition on each video frame in the video frame sequence to generate second time-series data including recognized text and corresponding position information; performing cross-modal time-series alignment and text content comparison based on the first time-series data and the second time-series data, and determining target subtitle information embedded in the video stream according to the comparison result, wherein the target subtitle information includes target recognized text and position information corresponding to the target recognized text, and the target recognized text is recognized text located in the second time-series data that matches the transcribed text in the first time-series data.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for video closed caption detection, the method comprising: The method comprises: obtaining an audio stream associated with a video stream, and performing speech recognition on the audio stream to generate first timing data comprising transcribed text and corresponding time information; extracting a sequence of video frames from the video stream, and performing character recognition on each video frame in the sequence of video frames to generate second timing data comprising recognized text and corresponding position information; based on the first timing data and the second timing data, performing cross-modal timing alignment and text content comparison, and determining target subtitle information embedded in the video stream according to the comparison result, the target subtitle information comprising target recognized text and position information corresponding to the target recognized text, the target recognized text being recognized text in the second timing data that matches the transcribed text in the first timing data.

2. The video closed caption detection method of claim 1, wherein, The method comprises: extracting the audio stream from the video stream, and performing format standardization processing on the audio stream; performing speech activity detection on the processed audio stream to segment out speech segments; processing the speech segments using a speech recognition model to generate first timing data comprising the transcribed text and time information corresponding to the transcribed text.

3. The video closed caption detection method of claim 1, wherein, The method comprises: based on a preset frame rate, extracting video frames from the video stream to obtain the sequence of video frames; based on prior knowledge of subtitle positions, determining subtitle candidate regions for each video frame in the sequence of video frames; performing text detection and recognition on any video frame using an optical character recognition model to obtain a plurality of candidate recognition results comprising text content and coordinate positions; matching the position coordinates of the plurality of candidate recognition results with the subtitle candidate regions to filter out target recognition results for the any video frame from the plurality of candidate recognition results, the target recognition results satisfying a preset condition in terms of overlap with the subtitle candidate regions; based on the target recognition results for each video frame, generating the second timing data.

4. The video closed caption detection method of claim 1, wherein, The method comprises: based on the first timing data and the second timing data, performing cross-modal timing alignment and text content comparison, and determining candidate subtitle information for each video frame in the video stream according to the comparison result; based on the candidate subtitle information, performing consistency verification on the position information of embedded subtitles in a plurality of consecutive video frames in terms of timing, and determining the target subtitle information according to the verification result.

5. The video closed caption detection method of claim 4, wherein, The method comprises: based on the first timing data and the second timing data, performing cross-modal timing alignment and text content comparison, and determining candidate subtitle information for each video frame in the video stream according to the comparison result. convert the first time-series data into a first time-series feature vector and convert the second time-series data into a second time-series feature vector; calculate a time-series correspondence between the first time-series feature vector and the second time-series feature vector using a time-series alignment algorithm; perform text similarity calculation on the transcribed text in the first time-series data and the recognized text in the second time-series data based on the time-series correspondence; determine the candidate subtitle information including the recognized text and corresponding position information based on the transcribed text and the recognized text that satisfy a preset similarity threshold.

6. The video closed caption detection method of claim 4, wherein, the consistency verification on the position information of the embedded subtitles in the multiple frames of video that are continuous in time sequence based on the candidate subtitle information includes: tracking the candidate subtitle information in the multiple frames of video that are continuous in time sequence and calculating the displacement amount of the position information in the candidate subtitle information between adjacent frames; in the case that the displacement amount calculated in the continuous multiple frames exceeds a preset displacement threshold, determining the candidate subtitle information of the corresponding frame as abnormal and filtering out.

7. The video closed caption detection method of any of claims 4 to 6, wherein, further comprising: in the case that the video stream does not contain audio information, skipping the steps of generating the first time-series data and performing cross-modal time-series alignment and text content comparison, and taking the recognized text and corresponding position information in the second time-series data as the candidate subtitle information.

8. An apparatus for video closed caption detection, the apparatus comprising: comprise: an audio processing unit configured to acquire an audio stream associated with a video stream, perform speech recognition on the audio stream, and generate first time-series data including transcribed text and corresponding time information; a video processing unit configured to extract a sequence of video frames from the video stream, perform character recognition on each video frame in the sequence of video frames, and generate second time-series data including recognized text and corresponding position information; a subtitle determination unit configured to perform cross-modal time-series alignment and text content comparison based on the first time-series data and the second time-series data, and determine target subtitle information embedded in the video stream according to the comparison result, the target subtitle information including target recognized text and position information corresponding to the target recognized text, the target recognized text being the recognized text in the second time-series data that matches the transcribed text in the first time-series data.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the video embedded subtitle detection method according to any one of claims 1 to 7. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the video embedded subtitle detection method according to any one of claims 1 to 7.

Images