Video subtitle extraction method

Abstract

The invention discloses a video subtitle extraction method. The problems that in the prior art, the recognition noise is large, a subtitle area is difficult to accurately position, the subtitle cross-frame stability is poor, structured output is lacked, and the adaptability of a clustering algorithm is insufficient are solved. The method comprises the following steps: performing frame sampling on input video data to form a video frame sequence; carrying out OCR (Optical Character Recognition) on the video frame sequence to obtain an initial recognition result; multi-dimensional filtering processing is executed on the initial recognition result to remove non-subtitle text interference, and the multi-dimensional filtering processing at least comprises cleaning based on text content and area preliminary screening based on a global position rule; performing clustering analysis based on spatial density on the text subjected to the multi-dimensional filtering processing so as to identify a subtitle text cluster from the preliminary screening region; and performing time sequence processing on the recognized subtitle text cluster, and outputting structured subtitle data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a video subtitle extraction method. BACKGROUND

[0002] In recent years, with the popularity of Internet video platforms and the rapid development of intelligent terminals, video has become one of the main carriers of information dissemination, entertainment consumption and knowledge acquisition. The scale and complexity of video content are increasing, and the demand for efficient and accurate content understanding and processing is also increasingly prominent. As an important form of video, subtitles carry voice content, semantic information and auxiliary explanations, and play an irreplaceable role in various application scenarios.

[0003] For the audience, subtitles help understand voice content and improve accessibility (e.g. for the hearing impaired or foreign language learners). For platforms and regulators, subtitles are important clues for video content review, compliance checks, and copyright protection. For secondary applications, subtitles are the core data source for video search, knowledge extraction, automatic translation, and summary generation. Therefore, how to quickly and accurately extract subtitles from massive videos has become a key problem in the field of computer vision and multimedia processing.

[0004] At present, the inventors realize that the extraction of video subtitles mainly relies on optical character recognition (OCR) technology. OCR can convert the text area in the video frame into editable text. However, in practical applications, there are the following outstanding problems in simply relying on OCR to extract subtitles: 1) high recognition noise: video pictures often contain non-subtitle text (such as advertisements, signs, interface elements), which will be recognized by OCR and interfere with subtitle extraction; 2) difficulty in accurately positioning the subtitle area: subtitles are usually located at the bottom of the picture, but under different resolutions and different video proportions, the area division is difficult to unify; 3) poor stability of subtitles across frames: the same subtitle often appears in consecutive frames, but the OCR result may be identified as multiple different text boxes due to slight shaking and position changes, causing redundancy; 4) lack of structured output: existing methods mostly directly output text, lacking frame-level timing and position information, which is not conducive to subsequent video processing; 5) insufficient adaptability of clustering algorithms: general clustering methods (such as standard DBSCAN) have the problems of misclustering or over-splitting in the subtitle scene, which need to be improved. SUMMARY

[0005] The present application provides a video subtitle extraction method, which aims to solve the problems of existing technology, such as high recognition noise, difficulty in accurately positioning the subtitle area, poor stability of subtitles across frames, lack of structured output, and insufficient adaptability of clustering algorithms.

[0006] The present application provides a video subtitle extraction method, comprising:

[0007] Step S1, frame sampling is performed on the input video data to form a video frame sequence; wherein the video frame sequence has at least one frame of picture with subtitles;

[0008] Step S2, the video frame sequence is subjected to OCR recognition to obtain an initial recognition result;

[0009] Step S3, multi-dimensional filtering processing is performed on the initial recognition result to remove non-subtitle text interference, and the multi-dimensional filtering processing at least includes cleaning based on text content and region preliminary screening based on global position rules;

[0010] Step S4, after the multi-dimensional filtering processing, spatial density-based clustering analysis is performed on the text to identify a subtitle text cluster from the preliminary screening region;

[0011] Step S5, time sequence processing is performed on the identified subtitle text cluster to output structured subtitle data.

[0012] In the above scheme, optionally, in step S4, the subtitle text cluster is identified from the preliminary screening region, comprising:

[0013] calculating the average horizontal center position of each cluster;

[0014] based on the offset of the average horizontal center position from the picture reference position and the number of text boxes contained in each cluster, filtering the cluster that simultaneously satisfies the centering condition and the maximum size condition from all clusters as the subtitle text cluster.

[0015] In the above scheme, optionally, the step S4 specifically comprises:

[0016] Step S41, the frame sequence obtained after the multi-dimensional filtering processing in step S3 is constructed one by one to form a coordinate set and a picture reference parameter:

[0017] Step S411, the center point coordinates of all candidate text boxes are arranged into a two-dimensional array in the format of:

[0018] ;

[0019] Step S412, all candidate text boxes are traversed, and half of the maximum horizontal boundary is taken as the picture horizontal center , that is ;

[0020] Step S42, setting the DBSCAN clustering parameters:

[0021] neighborhood radius eps: if the center point distance of two text boxes is a preset pixel value, it is considered as a neighborhood point;

[0022] core point minimum neighbor number : a text box needs at least A points in the neighborhood to be determined as a core point; wherein A is a preset value;

[0023] Step S43, using DBSCAN algorithm to cluster the text boxes reserved after step S3 processing:

[0024] Step S431, randomly select a point P from all unvisited candidate text boxes;

[0025] Step S432, check the neighborhood of the point P, calculate the number of points contained in the neighborhood; if the number of points in the neighborhood is greater than or equal to the minimum boundary point number, the point P is taken as a core point;

[0026] Step S433, add all points in the neighborhood of the core point to the cluster, for each point in the neighborhood, if it is also a core point, continue to expand the neighborhood of the point, and constantly expand until no new core point can be expanded;

[0027] Step S434, if a point is not a core point and does not belong to the neighborhood of any core point, mark it as noise;

[0028] Step S435, constantly repeat, execute the above steps S431-S434 on all points of the candidate text boxes until all points are visited;

[0029] Step S44, for the multiple non-noise clusters obtained after step S43 clustering execution, select the best subtitle area:

[0030] Step S441, calculate the average horizontal center of all text boxes in each cluster , the formula is as follows:

[0031]

[0032] wherein, represents the number of text boxes in the cluster, represents the horizontal center point coordinate of the mth text box in the cluster;

[0033] Step S442, screen the centralized cluster center: set the center offset threshold , calculate the deviation of the horizontal center of each cluster from the horizontal center of the picture , if , it is considered that the deviation is within an acceptable range, and it is taken as a candidate subtitle cluster;

[0034] ​In step S443, for the clusters meeting the centering condition, the cluster with the largest number of text boxes is selected as the final subtitle region.

[0035] In the above solution, optionally, in the step S3, the region preliminary screening based on the global position rule comprises:

[0036] calculating the aspect ratio of the video picture;

[0037] if the aspect ratio is greater than a preset proportion threshold, selecting a region with a first preset proportion below the screen as the preliminary screening region;

[0038] if the aspect ratio is less than or equal to the preset proportion threshold, selecting a region with a second preset proportion below the screen as the preliminary screening region, wherein the first preset proportion is less than the second preset proportion.

[0039] In the above solution, optionally, in the step S3, the cleaning based on the text content comprises at least one of the following operations:

[0040] invalid character filtering, empty character filtering, low confidence character filtering, vertical character filtering, and cross-frame dynamic interference character filtering.

[0041] In the above solution, further optionally, the cross-frame dynamic interference character filtering specifically comprises:

[0042] calculating the similarity of the text content of two continuous frames, and if the similarity is greater than or equal to a first predetermined threshold, determining that the text is similar;

[0043] calculating the horizontal center point moving distance of the similar text, and if the moving distance is greater than a second predetermined threshold, filtering out the text.

[0044] In the above solution, optionally, the initial recognition result obtained in the step S2 is structured, and contains frame-level timing information and text position information.

[0045] In the above solution, optionally, the method further comprises: organizing the initial recognition result obtained in the step S2 and / or the structured subtitle data obtained in the step S5 into JSON format data according to frames.

[0046] In the above solution, optionally, the step S5 specifically comprises:

[0047] sorting the subtitle text clusters obtained in the step S4 according to frame indexes to form a time sequence;

[0048] de-duplicating adjacent texts with the same content in the time sequence;

[0049] outputting structured subtitle data comprising frame indexes, subtitle content, and position information.

[0050] Optionally, the step S5 further comprises:

[0051] The structured subtitle data is further converted into an SRT standard format.

[0052] Compared with the prior art, the present application has at least the following beneficial effects:

[0053] Based on further analysis and research on the problems of the prior art, the present application realizes systematic solution to the above problems by constructing a collaborative processing procedure including multi-dimensional filtering and intelligent clustering.

[0054] Specifically, for the problem of difficult accurate positioning of the subtitle area, the present application realizes adaptive processing of videos with different resolutions and different picture ratios by introducing region preliminary screening based on global position rules, and can intelligently demarcate candidate subtitle areas; further, through clustering analysis based on spatial density, text clusters conforming to the spatial distribution characteristics of subtitles are accurately identified from the preliminary screening areas, so as to stably position the subtitle area in complex pictures. For the problem of poor cross-frame stability of subtitles, the present application identifies multiple text boxes appearing in similar positions as the same text cluster through clustering analysis, effectively eliminating redundant identification caused by picture jitter; in combination with the sorting and deduplication mechanism in time sequence processing, the continuity and consistency of the final output subtitles in the time dimension are ensured.

[0055] In addition, the present application effectively reduces identification noise through text content-based cleaning; meets the needs of subsequent processing by outputting structured subtitle data containing timing information and position information; and improves the adaptability of the clustering algorithm in the subtitle scene by incorporating domain knowledge in clustering analysis.

[0056] In summary, through the combination of the above technical means, the present application significantly improves the accuracy, purity and automation of video subtitle extraction, and provides a more reliable technical foundation for video content understanding and application. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 A flowchart of a video subtitle extraction method provided by an embodiment of the present application.

[0058] Figure 2 A subtitle area clustering result graph of a video subtitle extraction method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0060] In the description of the present application: unless otherwise specified, the meaning of "multiple" is two or more. The terms "first", "second", "third" and the like in the present application are intended to distinguish the objects referred to, and do not have a special meaning in the technical connotation aspect (for example, it should not be understood as emphasizing importance or order, etc.). The expressions "include", "contain", "have" and the like also mean "not limited to" (some units, components, materials, steps, etc.).

[0061] The present application proposes a video subtitle extraction method, aiming to solve the problems of large noise interference, difficult accurate positioning of subtitle area, serious cross-frame redundancy and lack of structured output in the prior art in video subtitle recognition. By introducing text cleaning, position screening, similarity filtering and subtitle clustering analysis method, the present application can efficiently and accurately extract subtitle information in complex video scenes, and ensure the consistency and integrity of the subtitles in the cross-frame sequence. This method not only improves the accuracy and robustness of subtitle recognition, but also outputs structured results with timing and position information, which is convenient for subsequent application processing. The related technology can be widely applied to video content analysis, automatic subtitle generation, intelligent retrieval, film and television review and supervision and other scenes.

[0062] In one embodiment, referring to Figure 1 , a video subtitle extraction method is provided, comprising steps S1-S5.

[0063] Step S1, frame sampling is performed on the input video data to form a video frame sequence; wherein the video frame sequence has at least one frame of picture with subtitles;

[0064] Step S2, performing OCR recognition on the video frame sequence to obtain an initial recognition result;

[0065] Step S3, performing multi-dimensional filtering processing on the initial recognition result to remove non-subtitle text interference, the multi-dimensional filtering processing at least includes cleaning based on text content and region preliminary screening based on global position rules;

[0066] Step S4, performing clustering analysis based on spatial density on the text after the multi-dimensional filtering processing to identify a subtitle text cluster from the preliminary screening region;

[0067] Step S5, performing timing processing on the identified subtitle text cluster to output structured subtitle data.

[0068] The present application generates structured and frame-ordered subtitle data through text recognition, region screening, noise removal, and subtitle clustering of video frames. This method improves the accuracy and stability of subtitle extraction and is suitable for video content analysis, automatic subtitle generation, and intelligent retrieval scenarios. The flowchart of the entire method is shown in Figure 1

[0069] The entire video subtitle extraction process starts with video input. First, the input video is frame-sampled to select appropriate video frames. Then, optical character recognition (OCR) is performed on the sampled frames to identify the text content in the frames. The recognized text and related information are then organized in JSON format. Next, text cleaning is performed to remove interfering characters and retain valid text. Region filtering is then performed to select text within the screen area that may contain subtitles. Subsequently, clustering is performed on these texts to aggregate texts belonging to the same subtitle. Sorting and deduplication are then performed to arrange the subtitles in order and remove duplicate content. Finally, structured subtitle output is obtained to obtain standardized subtitle data.

[0070] The scheme of the present application is described step by step as follows.

[0071] In one embodiment, regarding step S1 video input and frame sampling: the input video data is frame-sampled to form a video frame sequence; wherein the video frame sequence has at least one frame of picture with subtitles;

[0072] The original video data to be processed is obtained, which can be in various formats (such as MP4, AVI, MKV, etc.) and contains at least several frames of pictures with subtitles. The input video is frame-sampled to collect continuous video frames at one frame per second from the video, forming a frame sequence, and assigning a frame index index to each frame. This ensures complete capture of subtitle information, reduces redundant computation, and improves processing efficiency.

[0073] In one embodiment, step S2, optical character recognition (OCR): the video frame sequence is subjected to OCR recognition to obtain initial recognition results;

[0074] The initial recognition results obtained in step S2 are structured, containing frame-level timing information and text position information. The initial recognition results obtained in step S2 and / or the structured subtitle data obtained in step S5 are organized into JSON format data by frame.

[0075] A general character recognition model PaddleOCR is used to recognize the text of each frame image obtained, and the recognition results are organized into JSON format data by frame. The data structure is as follows: [

[0077] {​

[0078] "index": frame index,

[0079] "texts": [

[0080] {"text": "example subtitle", "location": {"widthInPixel": width, "heightInPixel": height, "topOffsetInPixel": top edge distance, "leftOffsetInPixel": left edge distance}, "conf": "confidence"} ]

[0082] }, ... ]

[0085] This step saves the recognition result in a structured manner, ensures the correspondence between the subtitle data and the time axis and position information, and provides a basis for subsequent filtering and clustering.

[0086] In one embodiment, step S3, multi-dimensional filtering processing. Perform multi-dimensional filtering processing on the initial recognition result to remove non-subtitle text interference, the multi-dimensional filtering processing at least includes cleaning based on text content and area preliminary screening based on global position rules;

[0087] In step S3, the cleaning based on text content includes at least one of the following operations:

[0088] Invalid character filtering, null character filtering, low confidence character filtering, vertical character filtering, and cross-frame dynamic interference character filtering.

[0089] The cross-frame dynamic interference character filtering specifically includes:

[0090] Calculate the similarity of the text content of two consecutive frames, and if the similarity is greater than or equal to a first predetermined threshold, determine that the text is similar.

[0091] Calculate the horizontal center point movement distance of the similar text, and if the movement distance is greater than a second predetermined threshold, filter out the text.

[0092] In step S3a, text cleaning.

[0093] Because the general character recognition model inevitably has character recognition errors and noise text interference in the video background, the text content in the JSON data is cleaned, and the specific steps are as follows:

[0094] 1. Invalid character filtering: Only Chinese characters and Arabic numerals are retained by using regular expressions, and abnormal symbols are removed.​

[0095] 2. Empty string filtering: Remove invalid content with empty string or only space, newline.

[0096] 3. Low confidence character filtering: Set confidence threshold to 0.5, filter characters with confidence conf < 0.5.

[0097] 4. Vertical character filtering: Calculate the width-height ratio of the text box :

[0098]

[0099] where w represents the width of the text box, represents the height of the text box. If , filter the text.

[0100] 5. Cross-frame dynamic interference character filtering: Calculate the similarity sim between the content of two consecutive text frames and text b:

[0101]

[0102] where represents the edit distance between the two text contents, represents the maximum length of the two texts. If the sim value is greater than or equal to 0.85, it is determined that the two texts are similar texts. For similar texts, calculate the difference dx of the horizontal center point

[0103]

[0104]

[0105] If the horizontal center point of the two consecutive text frames moves more than 50 pixels, i.e., dx > 50, filter out the text.

[0106] 6. The final output is the cleaned JSON format data.

[0107] This step can effectively remove non-subtitle interference such as advertisements, LOGO, and bullet screen through multi-dimensional filtering process, significantly improving the purity of the subtitle data.

[0108] In one embodiment, in the step S3, the region preliminary screening based on global position rules includes:

[0109] calculating the width-height ratio of the video frame;

[0110] if the width-height ratio is greater than a preset proportion threshold, selecting a region with a first preset proportion below the screen as the preliminary screening region; ​​

[0111] If the aspect ratio is less than or equal to the preset proportion threshold, a region of a second preset proportion below the screen is selected as the preliminary screening region, wherein the first preset proportion is less than the second preset proportion.

[0112] Step S3b, subtitle region filtering.

[0113] Based on the character position parameters in the filtered JSON data, the non-subtitle region is filtered out according to the space screening rule, and the specific steps are as follows:

[0114] 1. Calculate the text box center point feature:

[0115] Horizontal center point :

[0116] ;

[0117] Vertical center point :

[0118] ;

[0119] 2. Calculate the picture boundary parameters:

[0120] Maximum horizontal boundary :

[0121] ;

[0122] Maximum vertical boundary :

[0123] ;

[0124] wherein widthInPixel represents the width, heightInPixel represents the height, topOffsetInPixel represents the top edge distance, and leftOffsetInPixel represents the left edge distance.

[0125] 3. Subtitle region filtering:

[0126] If the aspect ratio of the video page is (widescreen video), a region of 30% below the screen is selected as the candidate subtitle region.

[0127] If the aspect ratio of the video page is (portrait video), a region of 60% below the screen is selected as the candidate subtitle region.

[0128] This step can automatically adapt to videos of different resolutions and frame ratios, avoiding the deviation of region division and reducing the interference of subtitles above the screen, thereby improving the accuracy of subtitle positioning.

[0129] Next, step S4 is performed to perform spatial density-based clustering analysis on the text after the multi-dimensional filtering process, to identify a subtitle text cluster from the preliminary screening area.

[0130] In step S4, identifying a subtitle text cluster from the preliminary screening area includes:

[0131] calculating the average horizontal center position of each cluster;

[0132] filtering, from all clusters, a cluster that simultaneously satisfies the centering condition and the largest size condition as the subtitle text cluster, based on the offset of the average horizontal center position from the picture reference position and the number of text boxes included in each cluster.

[0133] In one embodiment, all steps of step S4 are fully described: subtitle clustering.

[0134] The characters in all candidate text boxes are clustered using the DBSCAN clustering algorithm to obtain the position of the most frequent subtitle occurrence, i.e., the real subtitle area, and the characters in this area are the real subtitles. The specific steps are as follows:

[0135] 1. Construct a coordinate set and picture reference parameters:

[0136] (1) The center point coordinates of all candidate text boxes are sorted into a two-dimensional array in the format:

[0137] ;

[0138] (2) Determine the picture horizontal center. Since subtitles are usually located near the picture horizontal center, iterate through all candidate text boxes, take half of the maximum horizontal boundary as the picture horizontal center, i.e., , which is used for subsequent filtering of the centered subtitle clustering.

[0139] 2. Set the DBSCAN clustering parameters:

[0140] According to the spatial features of video subtitles (usually densely distributed and single-region clustered), two core parameters are preset:

[0141] (1) Neighborhood radius (eps): 20 pixels;

[0142] If the distance between the center points of two text boxes is ≤ 20 pixels, they are considered "neighborhood points";

[0143] (2) Minimum number of neighbors of core point ( ): 20;

[0144] A text box needs at least 20 "neighborhood points" to be determined as a "core point".

[0145] 3. Using DBSCAN algorithm to cluster the reserved text boxes, according to the position of the center point of each text box to identify the position area where the characters appear most frequently, that is, the real subtitle area. The specific method is as follows:

[0146] (1) Randomly select a point P from all unvisited candidate text boxes. (In the first round of clustering, since all are unvisited points, a point is randomly selected as the current processing point P; in the subsequent clustering, a point is randomly selected from the remaining unvisited points).

[0147] (2) Check the neighborhood of the point P, calculate the number of points contained in the neighborhood. If the number of points in the neighborhood is greater than or equal to the minimum boundary point number, the point P is taken as a core point.

[0148] (3) Expand the cluster, add all points in the neighborhood of the core point to the cluster, for each point in the neighborhood, if it is also a core point, continue to expand the neighborhood of the point, and keep expanding until no new core point can be expanded.

[0149] (4) Handle noise, if a point is not a core point and does not belong to the neighborhood of any core point, mark it as noise.

[0150] (5) Keep repeating, execute the above operation on all points of the candidate text box until all points are visited.

[0151] 4. After clustering, multiple non-noise clusters are obtained, and the best subtitle area is selected, the specific steps are as follows:

[0152] (1) Calculate the average horizontal center of all text boxes in each cluster, the formula is as follows:

[0153]

[0154] Wherein, represents the number of text boxes in the cluster, represents the horizontal center point coordinate of the mth text box in the cluster.

[0155] (2) Select the central cluster center: according to the feature that the subtitle is usually located in the horizontal center of the picture, set the center offset threshold , calculate the deviation of the horizontal center of each cluster from the horizontal center of the picture , if , it is considered that the deviation is within the acceptable range, and it is taken as a candidate subtitle cluster.

[0156] ​​(3) Select the largest cluster as the final subtitle set: for the clusters that meet the centering condition, select the cluster with the most text boxes (subtitles usually span multiple frames, and the number of text boxes is much larger than that of interference clusters) as the final subtitle area.

[0157] This step effectively avoids interference such as superscripts and background text by combining character density space clustering with position constraints, ensuring the stability and accuracy of subtitle area extraction.

[0158] After completing step S4 and obtaining the subtitle text cluster, step S5 is performed to process the recognized subtitle text cluster in time sequence and output structured subtitle data. Figure 2 The subtitle clustering results are shown. For all text boxes that appear in the candidate subtitle area after filtering, through clustering, the largest red area in the figure is finally obtained as the subtitle area, and the content of all text boxes in this area is the extracted subtitle content.

[0159] The step S5 specifically includes: sorting the subtitle text cluster obtained in step S4 by frame index to form a time sequence; removing duplicate texts in the time sequence; and outputting structured subtitle data including frame index, subtitle content and position information.

[0160] The step S5 further includes: further converting the structured subtitle data into SRT standard format.

[0161] In step S5, sorting, deduplication and structured output.

[0162] 1. Sort the subtitles by frame index to form a time sequence.

[0163] 2. Detect adjacent frame subtitles content, and remove duplicates if repeated.

[0164] 3. Output structured subtitle results, including frame index, subtitle content and position information, which can be further converted into SRT standard format.

[0165] This step ensures the integrity and continuity of the subtitle sequence, eliminates redundant information, and makes the output result directly applicable to various application scenarios such as retrieval, review, translation, etc.

[0166] The present application effectively eliminates irrelevant noise text by frame-by-frame extraction and OCR recognition of the input video, combined with text cleaning, vertical text filtering and adaptive region screening, significantly improving the purity of subtitle detection. On this basis, a de-duplication method using cross-frame similarity and displacement constraint is adopted to ensure the continuity and stability of the subtitles in time sequence and position. Further, by introducing the DBSCAN clustering method, the main subtitle cluster is screened in the joint feature space of spatial position and text similarity, which can maintain high robustness in different resolutions, frame ratios and complex scenes, and avoid the influence of advertising, corner marks and other interference information. The final output of the subtitle result has frame index, text content and position information, forming a structured result data, which not only guarantees the accuracy and stability of the subtitle extraction, but also provides a reliable data basis for subsequent automatic subtitle generation, video content retrieval and multimedia analysis, with strong universality and application value.

[0167] For the video to be content audited, the present application automatically extracts subtitles and performs text analysis for wrong word detection and sensitive content identification. Internal practice shows that this method has high accuracy and stability in large-scale video content processing, which can effectively improve the automation level and auditing efficiency of the video supervision platform, reduce the pressure of manual review, and at the same time play an important role in ensuring the content safety and compliance of video publishing.

[0168] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

Claims

1. A method for extracting video subtitles, characterized in that, include: Step S1: Frame sampling is performed on the input video data to form a video frame sequence; wherein, at least one frame in the video frame sequence contains subtitles; Step S2: Perform OCR recognition on the video frame sequence to obtain initial recognition results; Step S3: Perform multi-dimensional filtering on the initial recognition result to remove non-subtitle text interference. The multi-dimensional filtering process includes at least text content-based cleaning and region screening based on global position rules. Step S4: Perform spatial density-based clustering analysis on the text after the multi-dimensional filtering process to identify subtitle text clusters from the initial screening area. Step S5: Perform time-series processing on the identified subtitle text clusters to output structured subtitle data.

2. The video subtitle extraction method according to claim 1, characterized in that, In step S4, the subtitle text clusters identified from the initial screening area include: Calculate the average center position of each cluster; Based on the offset between the average level center position and the screen reference position, and the number of text boxes contained in each cluster, the cluster that simultaneously satisfies the centering condition and the largest size condition is selected from all clusters as the subtitle text cluster.

3. The video subtitle extraction method according to claim 1 or 2, characterized in that, Step S4 specifically includes: Step S41: Construct coordinate sets and image reference parameters for each frame sequence obtained after multi-dimensional filtering in step S3. Step S411: Find the center coordinates of all candidate text boxes. Organize into a two-dimensional array, in the following format: ； Step S412: Traverse all candidate text boxes and take the maximum horizontal boundary. Half of the horizontal center of the image ,Right now ; Step S42, set DBSCAN clustering parameters: Neighborhood radius eps: If the center points of two text boxes are separated by a preset pixel value, they are considered to be within the neighborhood. Minimum number of neighbors for core point A text box must have at least A neighborhood points to be considered a core point; where A is a preset value. Step S43: Use the DBSCAN algorithm to cluster the text boxes retained after processing in step S3: Step S431: Randomly select a point P from all unvisited candidate text boxes; Step S432: Check the ε neighborhood of point P and calculate the number of points contained in the neighborhood; if the number of points in the neighborhood is greater than or equal to the minimum number of boundary points, then point P is taken as the core point. Step S433: Add all points in the neighborhood of the core point to the cluster. For each point in the neighborhood, if it is also a core point, continue to expand the neighborhood of that point, and keep expanding until no new core points can be added. Step S434: If a point is not a core point and does not belong to the neighborhood of any core point, mark it as noise; Step S435, repeat continuously, performing the above steps S431-S434 on all points of the candidate text box until all points have been visited; Step S44: For the multiple non-noise clusters obtained after the clustering in step S43, select the optimal subtitle region: Step S441: Calculate the average center of all text boxes in each cluster. The formula is as follows: Where k represents the number of text boxes in this cluster. This represents the coordinates of the horizontal center point of the m-th text box in this cluster; Step S442, Filtering centered cluster centers: Set center offset threshold Calculate the deviation between the horizontal center of each cluster and the horizontal center of the image. ,like If the deviation is within an acceptable range, it is considered to be clustered as a candidate subtitle. Step S443: For clusters that meet the centering condition, select the cluster with the most text boxes as the final subtitle area.

4. The method according to claim 1, characterized in that, In step S3, the initial screening of regions based on global location rules includes: Calculate the aspect ratio of the video frame; If the aspect ratio is greater than a preset ratio threshold, then the area at the bottom of the screen with the first preset ratio is selected as the initial screening area. If the aspect ratio is less than or equal to the preset ratio threshold, then the area at the bottom of the screen with a second preset ratio is selected as the initial screening area, wherein the first preset ratio is less than the second preset ratio.

5. The method according to claim 1, characterized in that, In step S3, the text content-based cleaning includes at least one of the following operations: Invalid character filtering, empty character filtering, low-confidence character filtering, vertical character filtering, and cross-frame dynamic interference character filtering.

6. The method according to claim 5, characterized in that, The cross-frame dynamic interference character filtering specifically includes: Calculate the similarity of text content in two consecutive frames. If the similarity is greater than or equal to a first predetermined threshold, the text is determined to be similar. Calculate the horizontal center point movement distance of similar texts. If the movement distance is greater than a second predetermined threshold, then filter out the text.

7. The method according to claim 1, characterized in that, The initial recognition result obtained in step S2 is structured, which includes frame-level temporal information and text position information.

8. The method according to claim 1 or 7, characterized in that, Also includes: The initial recognition result obtained in step S2 and / or the structured subtitle data obtained in step S5 are organized into JSON format data frame by frame.

9. The method according to claim 1, characterized in that, Step S5 specifically includes: The subtitle text clusters obtained in step S4 are sorted by frame index to form a time series; Deduplication is performed on adjacent texts with identical content in the time series; The output includes structured subtitle data, including frame index, subtitle content, and position information.

10. The method according to claim 1 or 9, characterized in that, Step S5 further includes: The structured subtitle data is then further converted into the SRT standard format.

Images