A video frame clipping, subtitle extraction, text processing method and storage medium

Abstract

The present invention discloses a video frame clipping and intercepting subtitle extraction text processing method and storage medium. Firstly, a target video is split to obtain a video frame image, which can facilitate subsequent analysis and recognition of subtitles. In a further operation, the video frame image is preprocessed, and the image size is cropped, and the resolution is unified and denoised, so that the subsequent recognition of the subtitle image can be more accurate. The position of the subtitle in the video frame image is accurately located by setting a virtual positioning grid line, and the text appearing in the target video is prevented from being mistakenly recognized as a subtitle in subsequent recognition by positioning the subtitle. Further, the video frame image is analyzed in combination with the positioning grid line to obtain a subtitle image. Finally, the subtitle text is recognized on the subtitle image by using the OCR text recognition technology to obtain the subtitle of the target video, thereby improving the efficiency and accuracy of subtitle recognition.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a video frame clipping, intercepting, subtitle extraction, text processing method and a storage medium. Background Art

[0002] In the process of secondary creation and editing of movies or videos, it is often necessary to manually extract the subtitles, and then select new voice actors or voice actors to re-dub for secondary creation.

[0003] In the prior art, the method of extracting subtitles from video frames is to manually watch the video, play it frame by frame, and then manually record the subtitle information of the video frame; specifically, in the prior art, for extracting subtitles from digital movies or movies created in recent years, audio data is directly obtained, and then the audio data is calibrated with the subtitle text of the played video frame; especially for many film movies or early-made videos, it is very difficult to extract subtitles, and the video is split into video frame images, and then subtitles are extracted from each video frame image;

[0004] However, further research found that in the process of subtitle extraction from video frame images, there may be multiple areas with text in a video frame image. For example, there is and happens to be a book text displayed in the video. At this time, if subtitle extraction is performed directly, the text content in a page of text displayed by the current video frame image will be extracted as subtitles; this is definitely wrong, so how to accurately determine the subtitle area in the video frame image and obtain the subtitle image while avoiding the interference of other images (i.e., text-like images, such as tree branches, object frames, etc., which are close to or intersecting with the subtitle image and interfere with the screening of the subtitle image), and then accurately extract the subtitle text in the subtitle area is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a video frame clipping, subtitle extraction and text processing method and storage medium, which solves the above technical problems pointed out in the prior art.

[0006] The present invention provides a method for processing video frame clipping, intercepting subtitles and extracting text, comprising the following steps:

[0007] Acquire a target video; split the target video to obtain multiple video frame images;

[0008] Preprocessing the video frame image to obtain a first video frame image;

[0009] Setting positioning grid lines of fixed size for the first video frame image;

[0010] Analyzing the first video frame image based on the positioning grid lines to obtain a subtitle image;

[0011] The subtitles are obtained by OCR recognition based on the subtitle image.

[0012] Preferably, the positioning grid lines are a plurality of virtual grid lines parallel to the upper and lower edge lines of the first video frame image;

[0013] Preferably, the analyzing the first video frame image based on the positioning grid lines to obtain the subtitle image comprises the following steps:

[0014] Performing an image simplification processing operation on the first video frame image to obtain a second video frame image;

[0015] Performing edge detection on the second video frame image to obtain a plurality of first edge pixel points;

[0016] Acquire multiple text images based on all the first edge pixel points;

[0017] A plurality of subtitle images are obtained by screening from all the text images according to the positioning grid lines.

[0018] Preferably, the step of filtering and acquiring the subtitle image according to the positioning grid lines comprises the following steps:

[0019] Clustering the first edge pixel points of all text images in each of the second video frame images to obtain a plurality of clusters;

[0020] Calculate the screening distance from each cluster to each positioning grid line;

[0021] Obtain the enclosed text image corresponding to each positioning grid line according to each screening distance;

[0022] Obtaining the screening coordinates of the positioning grid lines corresponding to each of the enclosed text images;

[0023] By acquiring the screening coordinates of a plurality of second video frame images, a text image on each second video frame image corresponding to the screening coordinates having the same screening coordinates and having the highest appearance frequency in all the second video frame images is selected as a subtitle image.

[0024] Preferably, the acquiring of multiple text images based on all the first edge pixels comprises the following steps:

[0025] Calculating the gradient direction of each of the first edge pixel points, and obtaining the butting edge pixel points corresponding to each of the first edge pixel points according to the gradient direction;

[0026] Calculate and obtain the gradient distance between each of the first edge pixel points and the butted edge pixel points;

[0027] Acquire a plurality of second edge pixel points according to the gradient distance screening;

[0028] A plurality of text images are acquired according to the second edge pixel points.

[0029] Preferably, the acquiring of a plurality of text images according to the second edge pixel points comprises the following steps:

[0030] Acquire the second coordinates of each of the second edge pixel points; cluster the second edge pixel points according to the second coordinates to obtain a plurality of to-be-selected clusters;

[0031] Calculate and obtain the size K of each of the to-be-selected clusters;

[0032] Perform discrimination and screening according to the size quantity to obtain a target cluster corresponding to the text image;

[0033] A text image is acquired according to the target clustering cluster.

[0034] Preferably, the size is calculated as follows:

[0035]

[0036] Where W represents the cluster bounding box of the selected cluster K; W p With W q is the height and width of the cluster bounding box; p(K) is the number of rows of the second edge pixel points in the vertical direction of cluster K; q(K) is the number of rows of the second edge pixel points in the horizontal direction of cluster K; max(p(K),q(K)) is the maximum height and maximum width of the cluster bounding box; E W is the maximum threshold of direction rate; W q / q(K) is the directional rate; otherwise means if the directional rate is greater than or equal to the maximum directional rate threshold.

[0037] Preferably, the target cluster is identified and screened by:

[0038]

[0039] In the formula, E n With E t is the screening threshold; where E n <E t ; otherwise means if Size(K) is greater than or equal to E t or Size(K) is less than or equal to E n hour.

[0040] Preferably, the step of acquiring a text image according to the target cluster comprises the following steps:

[0041] Acquire a first text image according to the target cluster;

[0042] Acquire all third pixel points in the first text image, that is, third coordinates corresponding to the third pixel points; cluster the third pixel points according to the third coordinates to obtain a plurality of third clustering clusters;

[0043] Acquire text block images according to each of the sentence clusters;

[0044] Obtain average sizes a and b of each two adjacent text block images; calculate and obtain the inclusion distance d according to the average sizes; wherein a is the average height of each two adjacent text block images; and b is the average width of each two adjacent text block images;

[0045] The calculation method of the included distance d is: d = 0.5 × max (a, b);

[0046] Putting every two text block images corresponding to the inclusion distance d being less than or equal to the preset inclusion distance into the inclusion set;

[0047] The included set is output as a whole as a text image.

[0048] The present invention provides a storage medium for storing a computer-readable program, wherein the computer-readable program is used to execute the above-mentioned video frame clipping, intercepting, subtitle extraction and text processing method.

[0049] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0050] By analyzing the above-mentioned video frame clipping, subtitle extraction and text processing method and storage medium provided by the present invention, it can be known that in specific applications, the target video is first split to obtain a video frame image, which can facilitate subsequent analysis and identification of subtitles; in a further operation, the video frame image is preprocessed, and the image size is cropped, the resolution is unified, and the denoising operation is performed, so that the subsequent identification of the subtitle image can be more accurate; by setting a virtual positioning grid line, the position of the subtitle in the video frame image is accurately located, and the positioning of the subtitle is used to prevent the text appearing in the target video from being mistakenly identified as a subtitle in subsequent identification; further, the video frame image is analyzed in combination with the positioning grid line to obtain a subtitle image; finally, the subtitle text is recognized on the subtitle image using OCR text recognition technology to obtain the subtitle of the target video, thereby improving the efficiency and accuracy of subtitle recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A schematic diagram of the overall operation steps of a video frame clipping, intercepting, subtitle extraction and text processing method provided by the present invention;

[0053] Figure 2 A schematic diagram of the operation steps of analyzing and obtaining a subtitle image in a video frame clipping and subtitle extraction text processing method provided by the present invention;

[0054] Figure 3 A schematic diagram of the operation steps for obtaining multiple text images in a video frame clipping, intercepting, subtitle extraction and text processing method provided by the present invention;

[0055] Figure 4 A simulation schematic diagram of docking edge pixel points in a video frame clipping, intercepting, subtitle extraction, and text processing method provided by the present invention;

[0056] Figure 5 A schematic diagram of simulating interference elements in a method for extracting text from subtitles by clipping video frames provided by the present invention;

[0057] Figure 6 A schematic diagram of the operation steps for obtaining a text image in a video frame clipping, intercepting, subtitle and text extraction processing method provided by the present invention;

[0058] Figure 7 A simulation schematic diagram of recognition errors caused by inconsistent text stroke spacing in a video frame clipping, intercepting, and subtitle extraction text processing method provided by the present invention;

[0059] Figure 8 A schematic diagram of the operation steps of obtaining text images according to target clusters in a video frame clipping, subtitle extraction and text processing method provided by the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0062] like Figure 1 As shown, the present invention proposes a video frame clipping, intercepting, subtitle extraction and text processing method, comprising the following steps:

[0063] Step S10: obtaining a target video; splitting the target video to obtain a plurality of video frame images;

[0064] Step S20: preprocessing the video frame image to obtain a first video frame image;

[0065] The above preprocessing includes image size consistency processing, resolution consistency processing, denoising processing, etc.;

[0066] Step S30: setting positioning grid lines of a fixed size for the first video frame image;

[0067] The positioning grid lines are a plurality of virtual grid lines parallel to the upper and lower edge lines of the first video frame image;

[0068] Step S40: analyzing the first video frame image based on the positioning grid lines to obtain a subtitle image;

[0069] Step S50: Obtain subtitles through OCR recognition based on the subtitle image.

[0070] It should be noted that the above OCR recognition is the abbreviation of optical character recognition, which is a technology that converts text in an image into editable text. Through OCR technology, computers can recognize and understand the text in an image and convert it into a text format that can be edited and searched;

[0071] The embodiment of the present application first splits the target video to obtain video frame images, which can facilitate subsequent analysis and recognition of subtitles; in a further operation, the video frame images are pre-processed by cropping the image size, unifying the resolution, and performing denoising operations, so that subsequent recognition of subtitle images can be more accurate; the position of the subtitles in the video frame images is accurately located by setting virtual positioning grid lines, and the positioning of the subtitles prevents the text appearing in the target video from being mistakenly recognized as subtitles in subsequent recognition; further, the video frame images are analyzed in combination with the positioning grid lines to obtain subtitle images; finally, the subtitle text is recognized on the subtitle image using OCR text recognition technology to obtain the subtitles of the target video, thereby improving the efficiency and accuracy of subtitle recognition.

[0072] It should be noted that the above-mentioned OCR recognition is a prior art, that is to say, the embodiment of the present invention does not focus on recognizing subtitle images, because OCR recognition can be implemented using existing technical means; the embodiment of the present invention focuses on how to screen and identify which are real subtitle images and which are text images contained in the video frame images themselves (such as books in the video frame, etc.); for details, see the subsequent processing operations such as steps S41-S44.

[0073] Specifically, Figure 2 As shown, in step S40, the first video frame image is analyzed based on the positioning grid lines to obtain a subtitle image, including the following operation steps:

[0074] Step S41: performing an image simplification processing operation on the first video frame image to obtain a second video frame image;

[0075] The image simplification processing operation includes a grayscale processing operation and a binarization processing operation;

[0076] It should be noted that the above-mentioned grayscale processing can reduce the server pressure of subsequent detection calculation and analysis processing and improve the running speed. At the same time, the grayscale processing can well preserve the edge information in the image, reduce the appearance of discrete points, and facilitate subsequent edge detection;

[0077] The above binarization operation is performed after the grayscale operation. The above binarization operation can use a local binarization operation. Local binarization can more effectively process images with uneven lighting or backgrounds, as well as images with complex textures or noise. The threshold can be adjusted according to the characteristics of the local area, thereby better retaining the detail information in the image (here specifically edge information).

[0078] Step S42: performing edge detection on the second video frame image to obtain a plurality of first edge pixel points;

[0079] Step S43: acquiring a plurality of text images based on all the first edge pixel points;

[0080] Step S44: Filter and obtain multiple subtitle images from all the text images according to the positioning grid lines.

[0081] It should be noted that, in the above-mentioned embodiment of the present application, the subtitle image is obtained by screening according to the positioning grid line, including the following steps:

[0082] Step S441: clustering first edge pixel points of all text images in each of the second video frame images to obtain a plurality of clusters;

[0083] Step S442: Calculate the screening distance from each cluster to each positioning grid line;

[0084] Step S443: obtaining the surrounding text image corresponding to each positioning grid line according to each screening distance (the surrounding text image, i.e., the image formed by a plurality of text images whose screening distances of the clustering clusters of the text images are within the screening distance threshold);

[0085] Step S444: obtaining the screening coordinates of the positioning grid lines corresponding to each of the enclosing text images;

[0086] Step S445: by acquiring the filter coordinates of the plurality of second video frame images, selecting the text image on each second video frame image corresponding to the filter coordinates having the same filter coordinates and having the highest frequency of appearance in all the second video frame images as the subtitle image;

[0087] It should be noted that the above-mentioned text image refers to a collection of a subtitle image and an image of text displayed in a video frame image (i.e., an image of a book's text displayed in the current video frame image proposed in the above-mentioned background technology) and a text-like image. In addition, under normal circumstances, in a text image of a video, the position of the subtitle image is fixed. At the same time, the number of subtitle images will occupy a larger proportion in all text images (i.e., among multiple video frame images, the number of subtitle images will be the largest, while the image of a book's text displayed in a video frame image will only appear in a part of the video frame images in the entire target video, and the text-like image will be even more so, and will be less). At the same time, the position of the image of a book's text displayed in the target video and the text-like image will change in different video frame images. Therefore, in the above-mentioned embodiment of the present application, by clustering the second pixel points of the text image in each second video frame image, the clustering clusters obtained will reflect the position of each text image.

[0088] Then, based on the distances from each cluster to the positioning grid line, an enclosed text image is obtained (i.e., when multiple text images are within a certain distance from a positioning grid line, these text images together constitute an enclosed text image; for example, in the middle and lower part of the video frame image, there is a text image composed of a text segment, and the distance from this text image to a positioning grid line in the middle and lower part is within a preset distance threshold, and then this text image can be used as an enclosed text image; the enclosed text image is different from the above-mentioned text image in that the enclosed text image here is an image obtained by including multiple text images around a positioning grid line; and the above-mentioned text images are images that exist individually or as text sentences, that is, the text image content of the enclosed text image is greater than or equal to a single text image); and then the screening coordinates of each enclosed text image are obtained (the screening coordinates refer to the position of the enclosed text image in the video frame image); and then, by selecting the screening coordinates in each video frame image (here, it means comparing the screening coordinates in each video frame image) that are consistent and appear the most times as the coordinates corresponding to the subtitle image (for example, the screening coordinates appear the most in the middle and lower part of each video frame image and are consistent in position); then the text image that appears in each video frame image corresponding to the screening coordinates here is the subtitle image;

[0089] The above-mentioned embodiment of the present application first simplifies the image to improve the server processing efficiency and the display effect of the image edge; then detects through the edge detection algorithm to obtain multiple first edge pixel points; extracts multiple text images through the first edge pixel points, and then screens and determines the subtitle image in the text image by locating the position of the grid line, and filters out the text information displayed in the video frame image, so as to obtain text belonging only to the subtitles, which is convenient for users to watch.

[0090] Specifically, Figure 3 As shown, in step S43, multiple text images are obtained based on all the first edge pixels, including the following steps:

[0091] Step S431: calculating the gradient direction of each of the first edge pixel points, and acquiring the butting edge pixel points corresponding to each of the first edge pixel points according to the gradient direction;

[0092] It should be noted that, in the above-mentioned embodiment of the present application, the gradient direction O of any first edge pixel point X is extended to search for another edge pixel point Y (ie, the above-mentioned docking edge pixel point, such as Figure 4 shown);

[0093] Step S432: Calculate and obtain the gradient distance between each of the first edge pixel points and the butted edge pixel points;

[0094] It should be noted that in the video, the text (the text may be a text image played in the video, such as the text content of a book or a letter in the video frame, or it may be the text content of a billboard in the video frame, or it may be a subtitle text) is usually in a fixed font and a fixed font size. Therefore, the width of each text stroke is consistent. Therefore, the gradient distance between edge pixels is used to describe the stroke width. Furthermore, by presetting the gradient distance threshold, it is determined whether the first edge pixel is a pixel point of the text stroke, so that the edge pixel points belonging to the text (i.e., the second edge pixel points described below) can be screened out from a large number of first edge pixels.

[0095] Step S433: Filter and obtain a plurality of second edge pixel points according to the gradient distance;

[0096] It should be noted that the above-mentioned embodiment of the present application uses the gradient distance threshold to screen out two first edge pixel points corresponding to a gradient distance less than or equal to the gradient distance threshold and determines that both are second edge pixel points, thereby screening out a large number of first edge pixel points that do not belong to the text image, thereby improving the efficiency of subsequent re-screening.

[0097] Step S434: Acquire multiple text images according to the second edge pixel points (multiple text images are acquired from the second edge pixel points, for details, see the subsequent execution of steps S4341 to S4344).

[0098] It should be noted that the above-mentioned embodiment of the present application calculates the gradient direction of each first edge pixel point, and then calculates the gradient distance using the connecting edge pixel points in the gradient direction, and further uses the gradient distance to screen out the second edge pixel points that may be pixels of the text image.

[0099] During the specific implementation of the technical solution adopted by the present application, the technicians found that there may be some interference elements (such as Figure 5 As shown), that is, the second edge pixel points of the text-like image (or the image similar to the text). The width of the strokes of the text in the text-like image is close to that of the text image (such as a small branch, a frame of an object in a video frame image, etc.). The text image obtained by collecting and analyzing the second edge pixel points is not accurate. Therefore, it is necessary to eliminate these interference elements to obtain the second edge pixel points that belong only to the text image.

[0100] Specifically, Figure 6 As shown, in step S434, a plurality of text images are obtained according to the second edge pixel points, including the following operation steps:

[0101] Step S4341: obtaining the second coordinates of each of the second edge pixel points; clustering the second edge pixel points according to the second coordinates to obtain a plurality of to-be-selected clusters;

[0102] Step S4342: Calculate and obtain the size K of each of the to-be-selected clusters;

[0103] The dimensions are calculated as follows:

[0104]

[0105] Where W represents the cluster bounding box of the selected cluster K; W p With W q is the height and width of the cluster bounding box; p(K) is the number of rows of the second edge pixel points in the vertical direction of cluster K; q(K) is the number of rows of the second edge pixel points in the horizontal direction of cluster K; max(p(K),q(K)) is the maximum height and maximum width of the cluster bounding box (because the cluster bounding box is an irregular shape, only the maximum height and maximum width are used here to determine the size of the cluster); E W is the maximum threshold of direction rate; W q / q(K) is the directional rate; otherwise means if the directional rate is greater than or equal to the maximum directional rate threshold, that is, otherwise can also be understood as otherwise;

[0106] Step S4343: performing discrimination and screening according to the size to obtain a target cluster corresponding to the text image;

[0107] The target cluster is identified and screened in the following manner:

[0108]

[0109] In the formula, E n With E t Both are screening thresholds, and E n and E t are both constants; but both satisfy the following conditions: E n <E t ; otherwise means if Size(K) is greater than or equal to E t or Size(K) is less than or equal to E n hour;

[0110] It should be noted that tn is the minimum size of the candidate clusters that can be generated by the text-like image, and Ti is the maximum size of the candidate clusters that can be generated by the text image;

[0111] Step S4344: acquiring a text image according to the target cluster;

[0112] It should be noted that in the above embodiments of the present application, by clustering each second edge pixel point according to the second coordinates of each second edge pixel point, the candidate clustering clusters obtained can intuitively display the aggregation information of each image edge pixel point; furthermore, according to the height and width of the clustering cluster bounding box of each candidate clustering cluster and the number of rows of the second edge pixel points in the vertical and horizontal directions of each clustering cluster, the size quantity of each candidate clustering cluster is calculated and analyzed to reflect the size of each candidate clustering cluster; further, the target clustering cluster is screened out by using the size quantity through a preset screening threshold; however, in the screened target clustering cluster, due to the inconsistent character spacing, subsequent OCR recognition may be incorrect. Therefore, it is necessary to further analyze the target clustering cluster again to obtain the text image.

[0113] In the specific implementation process of the further technical solution, the technical personnel also found that when using the target clustering cluster to analyze and obtain the text image, due to the inconsistent spacing between the text strokes of some text images, some text images analyzed by the clustering cluster may have incorrect text recognition during the subsequent OCR recognition process. For example, the recognition of "木木昆", "木棍", and "林昆" may be incorrect due to the spacing of the character radicals (as Figure 7 shown), so it is necessary to refine each text image to obtain individual text images arranged according to their arrangement in the video frame image.

[0114] Specifically, as Figure 8 shown, in step S4344, obtaining the text image according to the target clustering cluster includes the following operation steps:

[0115] Step S43441: Obtain the first text image according to the target clustering cluster;

[0116] It should be noted that the above first text image is an image obtained by visually displaying all the pixel points surrounded by the target clustering cluster (that is, the candidate clustering cluster after removing the pixel points of the text image from the second edge pixel points obtained by the above scheme) with each second edge pixel point in the target clustering cluster as the boundary.

[0117] Step S43442: Obtain all the third pixel points in the first text image, that is, the third coordinates corresponding to the third pixel points; cluster the third pixel points according to the third coordinates to obtain a plurality of third clustering clusters;

[0118] Step S43443: Obtain the text block image according to each of the sentence clustering clusters;

[0119] In the above embodiments of the present application, an image formed by all the statement pixel points in each statement clustering cluster is used as a text block image. Among them, the text block image can be a radical (when the spacing between texts in the text image is relatively large, for example, for the character "棍" (stick), when the text spacing is relatively large, it may be recognized as "木昆", that is, the "木" radical on the left will be used as a text block image, and "昆" will also be used as a text block image).

[0120] Step S43444: Obtain the average sizes a and b of every two adjacent text block images; calculate and obtain the inclusion distance d according to the average sizes; where a is the average height of every two adjacent text block images; b is the average width of every two adjacent text block images;

[0121] The calculation method of the inclusion distance d is as follows:

[0122] d = 0.5 × max(a, b);

[0123] It should be noted that the above inclusion distance refers to the distance between two text block images;

[0124] Step S43445: Put every two text block images for which the inclusion distance d is less than or equal to the preset inclusion distance into the inclusion set;

[0125] Step S43446: Output the inclusion set as the overall text image.

[0126] It should be noted that in the above embodiments of the present application, specific text images are extracted from a relatively large image dataset, and the target clustering clusters are determined to locate and display text information; this is the preliminary stage of image processing to ensure that subsequent operations focus on valuable data, that is, the pixels within the area enclosed by the clustering clusters obtained through previous screening; further, the pixel points in the text images obtained in step S43441 are further clustered, and by analyzing the spatial positions (i.e., the third coordinates) of these pixel points, a more refined text information group, that is, the third clustering cluster, is obtained; further refining the data structure in the text images to lay a foundation for more detailed text recognition; by analyzing and processing the pixel points in each statement clustering cluster, more specific text block images are obtained; these can be texts segmented into smaller units, such as the image representations of individual characters or words; adjacent texts can be correctly separated, improving the recognition accuracy of text blocks;

[0127] By calculating the average sizes of adjacent text block images and determining the inclusion distance between them based on this, it is decided whether two text blocks are considered close and grouped into the same set, improving the accuracy of subsequent text recognition for each text; through the inclusion distance, it is decided which text block images should be considered adjacent and grouped into the same set;

[0128] Finally, the set of text blocks classified and organized through the above steps is output as a whole to obtain the final text image, which reflects the text part in the original image with a clearer structure and higher recognition accuracy.

[0129] In summary, the method for extracting text from subtitles by clipping video frames proposed in the example of the present invention can facilitate subsequent analysis and recognition of subtitles by first splitting the target video to obtain video frame images; in further operations, the video frame images are preprocessed, and the image size is cropped, the resolution is unified, and the denoising operation is performed, so that the subsequent recognition of subtitle images can be more accurate;

[0130] By setting virtual positioning grid lines, the position of subtitles in the video frame image is accurately located. The positioning of subtitles prevents the text appearing in the target video from being mistakenly recognized as subtitles in subsequent recognition. Furthermore, the video frame image is analyzed in combination with the positioning grid lines to obtain the subtitle image. Finally, the subtitle image is recognized by using OCR text recognition technology to obtain the subtitles of the target video, thereby improving the efficiency and accuracy of subtitle recognition.

[0131] Specifically, in the process of obtaining the subtitle image, the image simplification process is used to improve the server processing efficiency and the display effect of the image edge; then, a plurality of first edge pixel points are detected by an edge detection algorithm; the gradient direction of each first edge pixel point is calculated, and then the gradient distance is calculated using the docking edge pixel points in the gradient direction, and the second edge pixel points that may be the pixels of the text image are further screened out by the gradient distance; further, each second edge pixel point is clustered according to the second coordinates of each second edge pixel point, and the obtained candidate clustering cluster can intuitively display the aggregation information of each image edge pixel point; and then the size of each candidate clustering cluster is calculated and analyzed according to the height and width of the clustering cluster bounding box of each candidate clustering cluster and the number of rows of the second edge pixel points in the longitudinal and transverse directions of each clustering cluster K, reflecting the size of each candidate clustering cluster;

[0132] In addition, the size quantity is further used to filter out the target clusters through a preset screening threshold; and by extracting a specific text image from a larger image data set, the text information is located and displayed by determining the target cluster; this is the initial stage of image processing, ensuring that subsequent operations focus on valuable data, that is, the pixels within the area surrounded by the previously screened clusters; further, the pixels in the acquired text image are further clustered, and the spatial positions (that is, the third coordinates) of these pixels are analyzed to obtain a more subdivided text information group, that is, the third cluster;

[0133] Further refine the data structure in the text image to lay the foundation for more detailed text recognition; obtain more specific text block images by analyzing and processing the pixels in each sentence cluster; this can be text divided into smaller units, such as image representations of single characters or words; correctly separate adjacent text to improve the recognition accuracy of text blocks; calculate the average size of adjacent text block images and determine the inclusion distance between them accordingly, so as to decide whether to regard two text blocks as similar and include them in the same set, thereby improving the accuracy of subsequent recognition of each text; determine which text block images should be regarded as adjacent and included in the same set through the inclusion distance; finally, output the classified and sorted text block set as a whole to obtain the final text image, which reflects the text part in the original image, has a clearer structure and higher recognition accuracy.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace part or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video frame clipping and subtitle extraction text processing method, characterized in that: The steps are as follows: Acquire a target video; split the target video to obtain multiple video frame images; Preprocessing the video frame image to obtain a first video frame image; Setting positioning grid lines of fixed size for the first video frame image; Analyzing the first video frame image based on the positioning grid lines to obtain a subtitle image; Obtaining subtitles based on the subtitle image through OCR recognition; The step of analyzing the first video frame image based on the positioning grid lines to obtain a subtitle image comprises the following steps: Performing an image simplification processing operation on the first video frame image to obtain a second video frame image; Performing edge detection on the second video frame image to obtain a plurality of first edge pixel points; Acquire multiple text images based on all the first edge pixel points; Filtering and acquiring a plurality of subtitle images from all the text images according to the positioning grid lines; The method of obtaining the subtitle image by screening according to the positioning grid lines comprises the following steps: Clustering the first edge pixel points of all text images in each of the second video frame images to obtain a plurality of clusters; Calculate the screening distance from each cluster to each positioning grid line; Obtain the enclosed text image corresponding to each positioning grid line according to each screening distance; Obtaining the screening coordinates of the positioning grid lines corresponding to each of the enclosed text images; By acquiring the screening coordinates of a plurality of second video frame images, a text image on each second video frame image corresponding to the screening coordinates having the same screening coordinates and having the highest appearance frequency in all the second video frame images is selected as a subtitle image; The step of acquiring multiple text images based on all the first edge pixels includes the following steps: Calculating the gradient direction of each of the first edge pixel points, and obtaining the butting edge pixel points corresponding to each of the first edge pixel points according to the gradient direction; Calculate and obtain the gradient distance between each of the first edge pixel points and the butted edge pixel points; Acquire a plurality of second edge pixel points according to the gradient distance screening; A plurality of text images are acquired according to the second edge pixel points.

2. A video frame clipping and subtitle extraction text processing method according to claim 1, characterized in that: The positioning grid lines are a plurality of virtual grid lines parallel to the upper and lower edge lines of the first video frame image.

3. A video frame clipping and subtitle extraction text processing method according to claim 2, characterized in that: The step of acquiring a plurality of text images according to the second edge pixel points comprises the following steps: Acquire the second coordinates of each of the second edge pixel points; cluster the second edge pixel points according to the second coordinates to obtain a plurality of to-be-selected clusters; Calculate and obtain the size K of each of the to-be-selected clusters; Perform discrimination and screening according to the size quantity to obtain a target cluster corresponding to the text image; A text image is acquired according to the target clustering cluster.

4. A video frame clipping and subtitle extraction text processing method according to claim 3, characterized in that: The dimensions are calculated as follows: Where W represents the cluster bounding box of the selected cluster K; W p With W q is the height and width of the cluster bounding box; p(K) is the number of rows of the second edge pixel points in the vertical direction of cluster K; q(K) is the number of rows of the second edge pixel points in the horizontal direction of cluster K; max(p(K),q(K)) is the maximum height and maximum width of the cluster bounding box; E W is the maximum threshold of direction rate; W q / q(K) is the directional rate; otherwise means if the directional rate is greater than or equal to the maximum directional rate threshold.

5. A video frame clipping and subtitle extraction text processing method according to claim 4, characterized in that: The target cluster is identified and screened in the following manner: In the formula, E n With E t All are screening thresholds; Among them, E n <E t ; otherwise means if Size(K) is greater than or equal to E t or Size(K) is less than or equal to E n hour.

6. A method for extracting text from subtitles by clipping video frames according to claim 5, characterized in that: The step of acquiring a text image according to the target cluster comprises the following steps: Acquire a first text image according to the target cluster; Acquire all third pixel points in the first text image, that is, third coordinates corresponding to the third pixel points; cluster the third pixel points according to the third coordinates to obtain a plurality of third clustering clusters; Acquire text block images according to each of the sentence clusters; Obtain average sizes a and b of each two adjacent text block images; and calculate and obtain the inclusion distance d according to the average sizes; wherein a is the average height of each two adjacent text block images; b is the average width of every two adjacent text block images; The calculation method of the included distance d is: d = 0.5 × max ( a,b ) ; Putting every two text block images corresponding to the inclusion distance d being less than or equal to the preset inclusion distance into the inclusion set; Output the text image as a whole by including the set;.

7. A storage medium, characterized in that: The storage medium is used to store a computer-readable program, and the computer-readable program is used to execute a video frame clipping, intercepting, subtitle extraction and text processing method as described in any one of claims 1 to 6.

Images