Subtitle extraction method, device, computer equipment, readable storage medium and product

Abstract

The embodiment of the present application discloses a subtitle extraction method, device, computer equipment, readable storage medium and product, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. In the embodiment of the present application, the subtitle extraction device obtains a video to be processed; performs frame processing on the video to be processed to obtain a multi-frame image sequence; performs text detection processing on each frame of the image to identify the text detection frame in each frame of the image; performs temporal trajectory tracking on the text detection frame in each frame of the image to obtain a text detection frame tracking trajectory set of the video to be processed; determines a reference subtitle area based on the text detection frame following trajectory set of the video to be processed; and extracts the subtitles of the video to be processed based on the reference subtitle area. The accuracy of subtitle extraction is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a subtitle extraction method, device, computer equipment, readable storage medium and product. Background Art

[0002] Many existing video subtitle extraction methods do not automatically extract subtitles from input videos, but instead input images of subtitle areas in the video and extract subtitles from these subtitle images.

[0003] Many existing methods for extracting subtitles from videos use traditional computer vision methods (e.g., edge detection, image filtering). The video text detection and recognition steps use traditional OCR. Traditional OCR technology cannot handle text in Internet videos with complex backgrounds, which results in previous subtitle detection and subtitle following methods targeting simple videos, that is, there are no other texts or very few other texts in the video except for the subtitles. With the increasing number of Internet videos, there are many types of text in the video. Subtitles are only part of the text in the video, and there may be many other non-subtitle text areas in the video. For these complex videos containing a lot of text, previous methods cannot accurately extract and follow the subtitles. Summary of the invention

[0004] The embodiments of the present application provide a subtitle extraction method, apparatus, computer equipment, readable storage medium and product, which can improve the efficiency of subtitle extraction.

[0005] A subtitle extraction method, comprising:

[0006] Get the video to be processed;

[0007] Performing frame processing on the video to be processed to obtain a multi-frame image sequence;

[0008] Performing text detection processing on each frame of the image to identify a text detection frame in each frame of the image;

[0009] Performing temporal tracking of the text detection frame in each frame of the image to obtain a set of text detection frame tracking trajectories of the video to be processed;

[0010] Determine a reference subtitle area according to a set of text detection frames following trajectories of the video to be processed;

[0011] The subtitles of the video to be processed are extracted based on the reference subtitle area.

[0012] Accordingly, an embodiment of the present application provides a subtitle extraction device, comprising:

[0013] An acquisition unit, used for acquiring a video to be processed;

[0014] A frame division unit, used for performing frame division processing on the video to be processed to obtain a multi-frame image sequence;

[0015] A detection unit, used to perform text detection processing on each frame of the image to identify a text detection frame in each frame of the image;

[0016] A trajectory following unit, used to perform temporal trajectory following on the text detection frame in each frame of the picture, and obtain a set of text detection frame following trajectories of the video to be processed;

[0017] A determination unit, configured to determine a reference subtitle area according to a set of trajectories followed by a text detection frame of the video to be processed;

[0018] An extraction unit is used to extract subtitles of the video to be processed based on the reference subtitle area.

[0019] Optionally, in some embodiments, the trajectory following unit can be specifically used to perform temporal trajectory following on the text detection box in each frame of the image to obtain a following trajectory set of the text detection box; and generate a following trajectory set of the text detection box of the video to be processed based on the following trajectory set of the text detection box.

[0020] Optionally, in some embodiments, the trajectory following unit can be specifically used to obtain the text editing distance of the text detection box in each adjacent frame image, and the coordinate information of the corresponding text detection box in each frame image in the adjacent frame images; based on the coordinate information and the text editing distance, generate a following trajectory set of the text detection box.

[0021] Optionally, in some embodiments, the trajectory following unit can be specifically used to determine the area information of the text detection box in the adjacent frame picture based on the coordinate information; if it is determined according to the area information that the text detection box in the adjacent frame picture meets the first matching condition, and it is determined according to the text editing distance that the text detection box in the adjacent frame picture meets the second matching condition, then the text detection box of the subsequent frame picture in the adjacent frame pictures is added to the following trajectory set corresponding to the text detection box of the previous frame picture.

[0022] Optionally, in some embodiments, the trajectory following unit can be specifically used to obtain the intersection area and union area of ​​the text detection boxes in the adjacent frame images; calculate the intersection ratio of the text detection boxes in the adjacent frame images based on the intersection area and the union area; if the intersection ratio of the text detection box is greater than or equal to a preset intersection ratio, it is determined that the text detection box meets the first matching condition.

[0023] Optionally, in some embodiments, the track following unit may be specifically configured to determine that the text detection frame satisfies a second matching condition if the text edit distance is less than or equal to a preset edit distance.

[0024] Optionally, in some embodiments, the trajectory following unit can be specifically used to use a preset text recognition algorithm to identify text content in text detection boxes in adjacent frame images; and calculate the text editing distance between the text contents in the text detection boxes in the adjacent frame images based on the text content.

[0025] Optionally, in some embodiments, the trajectory following unit can be specifically used to initialize the following trajectory set of the text detection box according to the text detection box in the next frame picture of the adjacent frame pictures if it is determined that the text detection box in the adjacent frame pictures does not satisfy the first matching condition based on the area information, and / or it is determined that the text detection box in the adjacent frame pictures does not satisfy the second matching condition based on the text editing distance.

[0026] Optionally, in some embodiments, the trajectory following unit can be specifically used to determine a target text detection frame from the multiple text detection frames according to the union and intersection ratios corresponding to the multiple text detection frames if it is determined according to the area information that there are multiple text detection frames in the adjacent frame image that satisfy the first matching condition and the second matching condition; and add the target text detection frame to the following trajectory set corresponding to the text detection frame in the previous frame image of the adjacent frame image.

[0027] Optionally, in some embodiments, the determination unit can be specifically used to select coordinate information corresponding to a reference text detection frame from each text detection frame following trajectory in the text detection frame following trajectory set of the video to be processed in turn; determine candidate values ​​of the image area corresponding to the reference text detection frame according to a preset algorithm and the coordinate information corresponding to the reference text detection frame; and select the image area corresponding to the maximum value among the candidate values ​​as the reference subtitle area.

[0028] Optionally, in some embodiments, the extraction unit can be specifically used to obtain the coordinate information of the reference subtitle area and the coordinate information of all text detection boxes in the text detection box following trajectory set of the video to be processed; based on the coordinate information of the reference subtitle area and the coordinate information of all text detection boxes in the text detection box following trajectory set of the video to be processed, the subtitles of the video to be processed are extracted, wherein the difference between the coordinate information of the subtitle area of ​​the video to be processed and the coordinate information of the reference subtitle area is less than a preset value.

[0029] In addition, an embodiment of the present application further provides a computer device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to implement the subtitle extraction method provided in the embodiment of the present application.

[0030] In addition, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for a processor to load to execute the steps in any one of the subtitle extraction methods provided in the embodiment of the present application.

[0031] In addition, an embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, it is implemented to perform any of the subtitle extraction methods provided in the embodiments of the present application.

[0032] In the embodiment of the present application, after acquiring the video to be processed, the electronic device performs frame processing on the video to be processed to obtain a multi-frame image sequence, and then performs text detection processing on each frame of the image to identify the text detection frame in each frame of the image; then, the text detection frame in each frame of the image is tracked to obtain a set of track following the text detection frame of the video to be processed; then, a reference subtitle area is determined based on the set of track following the text detection frame of the video to be processed; and the subtitles of the video to be processed are extracted based on the reference subtitle area. In this way, all the text in the video to be processed is detected and the track of the text detection frame is followed, and the subtitle area of ​​the video to be processed is calculated based on the track following each text detection frame, so as to distinguish the subtitles from other texts in the video to be processed, and then extract the subtitles, thereby improving the accuracy of subtitle extraction in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a flowchart of a subtitle extraction method provided in an embodiment of the present application;

[0035] Figure 2 is a schematic diagram of two frames of images of a video to be processed provided by an embodiment of the present application;

[0036] Figure 3 is a coordinate diagram of a text detection frame provided in an embodiment of the present application;

[0037] Figure 4 This is a schematic diagram of a picture after two adjacent frames of pictures are merged provided in an embodiment of the present application;

[0038] Figure 5 is another schematic diagram of the process of subtitle extraction provided by an embodiment of the present application;

[0039] Figure 6 is a structural schematic diagram of a subtitle extraction device provided in an embodiment of the present application;

[0040] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0042] The embodiment of the present application provides a subtitle extraction method, device and computer-readable storage medium. The subtitle extraction device can be integrated in an electronic device, which can be a server or a terminal.

[0043] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0044] For example, see Figure 1 Taking the subtitle extraction device integrated in an electronic device as an example, when the electronic device obtains a video to be processed, it performs frame processing on the video to be processed to obtain a multi-frame image sequence, and then performs text detection processing on each frame of the image to identify the text detection frame in each frame of the image; then, the text detection frame in each frame of the image is tracked in time to obtain a text detection frame tracking trajectory set of the video to be processed; then, a reference subtitle area is determined according to the text detection frame tracking trajectory set of the video to be processed; and the subtitles of the video to be processed are extracted based on the reference subtitle area.

[0045] The text detection frame may be a circumscribed rectangular frame of the text, and the tracking trajectory set of the text detection frame includes coordinate information of multiple text detection frames.

[0046] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0047] This embodiment will be described from the perspective of a subtitle extraction device, which can be integrated into an electronic device, which can be a server or a terminal; wherein the terminal can include a mobile phone, a computer, a personal computer (PC), an intelligent voice interaction device, an intelligent home appliance, a vehicle terminal, an aircraft, a wearable device, a virtual reality device or other intelligent devices that can obtain data. The embodiment of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0048] like Figure 1 As shown, the specific process of the subtitle extraction method is as follows:

[0049] 101. Obtain the video to be processed.

[0050] For example, the electronic device obtains the video to be processed from locally stored videos, or the electronic device obtains the video to be processed from other devices, or the electronic device obtains the video to be processed from a server, or the electronic device receives the video to be processed input by a user.

[0051] 102. Perform frame processing on the video to be processed to obtain a multi-frame image sequence.

[0052] Among them, the video to be processed can be an Internet video or a TV video, etc., and the video to be processed can contain subtitle text and non-subtitle text. The so-called non-subtitle text can be a watermark, barrage, station logo or advertising slogan in the video. In order to extract the subtitles in the video to be processed, it is first necessary to divide the video to be processed into a multi-frame image sequence, so as to extract the subtitles in each frame of the image.

[0053] Optionally, performing frame processing on the video to be processed to obtain a multi-frame image sequence includes: obtaining a preset frame rate for the video to be processed, and dividing the video to be processed into a multi-frame image sequence arranged in chronological order according to the preset frame rate. The preset frame rate can be the original frame rate of the video to be processed, or it can be a sampling frame rate lower than the original frame rate. For example, if the original frame rate of the video to be processed is 24FPS, the sampling frame rate when performing frame processing on the video to be processed can be 10FPS. For example, if the video to be processed is 10s and the preset frame rate is 10FPS, the video to be processed can be divided into 100 frames of images.

[0054] 103. Perform text detection processing on each frame of the image to identify a text detection frame in each frame of the image.

[0055] There may be one or more text detection frames in each frame of the image. In order to accurately identify the text detection frame in the image, a text detection algorithm may be used to process each frame of the image, thereby extracting the text detection frame. Specifically, performing text detection processing on each frame of the image to identify the text detection frame in each frame of the image may include: obtaining a preset text detection algorithm; performing text detection processing on each frame of the image according to the text detection algorithm to extract feature information corresponding to the text in each frame of the image; and determining the text detection frame in each frame of the image according to the feature information. The preset text detection algorithm may be an end-to-end image segmentation-based algorithm Pixellink based on deep learning, and the feature information may be the position information of the text and the visual feature information of the text. The visual feature information of the text may include information with distinct features in the visual image, such as edges, corners, centers of circles or ellipses, and shape features of the image. Determining the text detection frame in each frame of the image according to the feature information may include: inputting the position information and visual feature information of the text into a recurrent neural network model to obtain the text detection frame.

[0056] Considering that the subtitles in the video to be processed are generally horizontal text, the text detection frame output by the text detection algorithm can also be a positive rectangular frame, which can be represented by the coordinates (x, y, w, h), where x and y are the horizontal and vertical coordinates of the upper left corner of the text detection frame, respectively, and w and h are the height and width of the text detection frame, respectively. For the convenience of description, the symbol Bt can be used to represent a text detection frame in the t-th frame image. For example, Bt(x1, y1, w1, h1) is a text detection frame in the t-th frame image, and Bt(x2, y2, w2, h2) is another text detection frame in the t-th frame image.

[0057] 104. Follow the trajectory of the text detection frame in each frame of the image in time to obtain a set of text detection frame following trajectories of the video to be processed.

[0058] Based on the text detection frame obtained by text detection, the track is followed, so as to connect the same text between different frames in the video to be processed to form the track of the text detection frame in time. Since the track of the text detection frame is a collection of text detection frames of multiple frames, the last text detection frame in the collection can be used to represent the track of the text detection frame. For example, the track of a text detection frame can be represented by Tt, T t = {B s ,B s+1 ,…,B t}, where Bt represents the text detection box in the t-th frame image, and similarly Bs represents the text detection box in the s-th frame. The text detection box appears for the first time in the s-th frame of the video to be processed and for the last time in the t-th frame of the video to be processed, and it needs to satisfy s<t. The text detection box following is an online algorithm, that is, the text detection box following result of the s-th frame is generated based on the text detection box following result of the s-1-th frame and the text detection result of the s-th frame. In this embodiment, it is necessary to track the text detection box in each frame of the picture in time, so as to obtain a set of tracking trajectories of all text detection boxes in the video to be processed.

[0059] Optionally, performing temporal tracking on the text detection frame in each frame of the image to obtain a set of text detection frame following trajectories of the video to be processed may include: performing temporal tracking on the text detection frame in each frame of the image to obtain a set of following trajectories of the text detection frame; and generating a set of text detection frame following trajectories of the video to be processed based on the set of following trajectories of the text detection frame.

[0060] Among them, the set of text detection boxes following the trajectory of the video to be processed is H = {T1,…,T i ,…,T n}, H contains the tracking tracks of n text detection frames, that is, the algorithm believes that there are n different texts in the video to be processed. The text detection frame in each frame is tracked in time to obtain the tracking track set of the text detection frame, which may include the following:

[0061] S1. Obtain the text edit distance of the text detection frame in each adjacent frame image, and the coordinate information of the corresponding text detection frame in each adjacent frame image.

[0062] Among them, each adjacent frame image refers to the adjacent frames starting from the first frame in the time sequence in the multi-frame image sequence arranged in time sequence after the video to be processed is framed. For example, if the video to be processed is framed to obtain a t-frame image, the text editing distance between the text detection boxes in the first and second frames is obtained, and the text editing distance between the text detection boxes in the second and third frames is obtained, and this order is followed until the text editing distance between the text detection boxes in the t-1th frame and the tth frame is obtained.

[0063] Among them, the text edit distance can measure the similarity of the text content in two text detection frames. The larger the text edit distance, the lower the similarity of the text content in the two text detection frames. The smaller the text edit distance, the higher the similarity of the text content in the two text detection frames. In other words, the two text detection frames are more likely to be the same text detection frames. There can be one or more text detection frames corresponding to each frame of the image, and accordingly, there can also be one or more coordinate information of the text detection frame. For example, the coordinate information of a text detection frame in the previous frame, i.e., the t-2th frame in the adjacent frame image can be expressed as B t-2 (x1, y1, w1, h1), the coordinate information of another text detection box can be expressed as B t-2 (x2, y2, w2, h2), the coordinate information of a text detection frame in the next frame, i.e., the t-1th frame, can be expressed as B t-1 (x1, y1, w1, h1), the coordinate information of another text detection box can be expressed as B t-1 (x2,y2,w2,h2).

[0064] Wherein, obtaining the text edit distance of the text detection frame in each adjacent frame of the picture may include:

[0065] Using a preset text recognition algorithm to recognize text content in text detection boxes in adjacent frame images;

[0066] The text edit distance of the text content in the text detection frame in the adjacent frame image is calculated according to the text content.

[0067] The preset text recognition algorithm may be an end-to-end text recognition algorithm based on deep learning, such as a convolutional recurrent neural network (CRNN) model. The text edit distance of text detection boxes in adjacent frames may be calculated to compare whether the contents of the text detection boxes in adjacent frames are the same.

[0068] S2. Generate a set of follow-up trajectories of the text detection box according to the coordinate information and the text edit distance.

[0069] Optionally, S2 may include:

[0070] (1) determining area information of a text detection frame in an adjacent frame image according to coordinate information; if it is determined according to the area information that the text detection frame in the adjacent frame image satisfies a first matching condition, and it is determined according to the text edit distance that the text detection frame in the adjacent frame image satisfies a second matching condition, then adding the text detection frame of the next frame image in the adjacent frame image to a follow-up trajectory set corresponding to the text detection frame of the previous frame image.

[0071] The area information of the text detection frames in adjacent frames includes the intersection area and the union area of ​​the corresponding areas of the text detection frames in adjacent frames. The first matching condition is that the ratio of the intersection area and the union area of ​​the corresponding areas of the text detection frames in adjacent frames is greater than or equal to the preset union-intersection ratio, and the second matching condition is that the text edit distance of the text detection frames in adjacent frames is less than or equal to the preset edit distance.

[0072] The preset cross-over ratio may be set in advance according to actual needs, for example, the preset cross-over ratio may be set to 0.8, and the preset edit distance may be set in advance according to actual needs, for example, the preset edit distance may be set to 2.

[0073] That is to say, if the ratio of the intersection area and the union area of ​​the corresponding areas of the text detection frames in adjacent frame images is greater than or equal to the preset union-intersection ratio, and the text editing distance of the text detection frame is less than or equal to the preset editing distance, then the text detection frame of the next frame in the adjacent frame images is added to the follow-up trajectory set corresponding to the text detection frame in the previous frame image.

[0074] For example, if the text detection box B in the second frame, that is, the t-1 frame, of the two adjacent frames of the video to be processed is t-1 (x1, y1, w1, h1) and the text detection box B in the previous frame, i.e. the t-2 frame t-2 (x1, y1, w1, h1) satisfies both the first and second matching conditions, then B t-1 (x1,y1,w1,h1) added to B t-2 The following trajectory set T corresponding to (x1, y1, w1, h1) t-2 = {B1, ..., B t-2 (x1,y1,w1,h1)} to get T t-1 = {B1, ..., B t-2 (x1,y1,w1,h1),B t-1 (x1,y1,w1,h1)}.

[0075] Optionally, in one embodiment, if it is determined according to the area information that the text detection frame in the adjacent frame picture satisfies the first matching condition, and it is determined according to the text edit distance that the text detection frame in the adjacent frame picture satisfies the second matching condition, then adding the text detection frame of the next frame picture in the adjacent frame picture to the following track set corresponding to the text detection frame of the previous frame picture may include:

[0076] If it is determined based on the area information that there are multiple text detection frames in the adjacent frame images that meet the first matching condition and the second matching condition, then determine the target text detection frame from the multiple text detection frames based on the union and intersection ratios corresponding to the multiple text detection frames; and add the target text detection frame to the follow-up trajectory set corresponding to the text detection frame in the previous frame image of the adjacent frame image.

[0077] For example, if the text detection box B in the t-1th frame of two adjacent frames t-1 (x1, y1, w1, h1) and the text detection box B of the t-2th frame t-2 (x1, y1, w1, h1) satisfies both the first and second matching conditions, and the text detection box B of the t-1th frame t-1 (x2, y2, w2, h2) also matches the text detection box B in the t-2th frame t-2 (x1, y1, w1, h1) satisfies both the first and second matching conditions. That is, there are two text detection frames in the t-1th frame image that match a text detection frame in the t-2th frame image. At this time, it is necessary to determine the target text detection frame from the two text detection frames in the t-1th frame image as the target text detection frame that matches the text detection frame B. t-2 (x1, y1, w1, h1) correctly matched following trajectory.

[0078] Optionally, determining a target text detection frame from multiple text detection frames may include: comparing the union-intersection ratios of the multiple text detection frames with the text detection frame in the previous frame image, and selecting a text detection frame from the multiple text detection frames that has the largest union-intersection ratio with the text detection frame in the previous frame image as the target text detection frame.

[0079] For example, if the text detection box B in the t-1th frame t-1 (x1, y1, w1, h1) and the text detection box B of the t-2th frame t-2 The intersection-and-joint ratio of (x1, y1, w1, h1) is 0.83, and the text detection box B in the t-1th frame t-1 (x2, y2, w2, h2) and the text detection box B of the t-2th frame t-2 The intersection ratio of (x1, y1, w1, h1) is 0.86, so the text detection box B is selected t-1 (x2, y2, w2, h2) is used as the target text detection box.

[0080] Correspondingly, adding the target text detection frame to the follow-up trajectory set corresponding to the text detection frame in the previous frame image of the adjacent frame image may specifically include: adding the target text detection frame B t-1 (x2,y2,w2,h2) added to text detection box B t-2 The following trajectory set T corresponding to (x1, y1, w1, h1)t-2 = {B1, ..., B t-2 (x1,y1,w1,h1)} to get T t-1 = {B1, ..., B t-2 (x1,y1,w1,h1),B t-1 (x2,y2,w2,h2)}.

[0081] Optionally, determining whether the text detection box in adjacent frame pictures meets the first matching condition based on the area may include: obtaining the intersection area and the union area of ​​the text detection boxes in adjacent frame pictures; calculating the intersection ratio of the text detection boxes in the adjacent frame pictures based on the intersection area and the union area; if the intersection ratio of the text detection box is greater than or equal to a preset intersection ratio, determining that the text detection box meets the first matching condition.

[0082] Among them, the intersection-and-intersection ratio of the text detection frames in adjacent frame images is the ratio of the intersection area of ​​the two detection frames to the union area. For example, if the intersection area of ​​two text detection frames in adjacent frame images is M, and the union area is N, then the intersection-and-intersection ratio of the two text detection frames is M / N. If there are multiple text detection frames in adjacent frame images, the intersection-and-intersection ratio of the text detection frame in the subsequent frame image and the text detection frame in the previous frame image are calculated in sequence to obtain multiple intersection-and-intersection ratios.

[0083] Optionally, determining that the text detection frame in the adjacent frame image satisfies the second matching condition according to the text edit distance may include: if the text edit distance is less than or equal to a preset edit distance, determining that the text detection frame satisfies the second matching condition.

[0084] It is understandable that in actual applications, due to reasons such as occlusion of text in the image, interference from the image background, and errors in the text recognition algorithm, the text recognition results of the text detection frame belonging to the same text and the following trajectory of the text detection frame may not be the same. Therefore, in this embodiment, we calculate the text editing distance of the text detection frames in adjacent frame images. If the editing distance is greater than or equal to the preset editing distance, it means that the text detection frames in adjacent frame images do not match and cannot be followed as the same text detection frame.

[0085] (2) If it is determined based on the area information that the text detection box in the adjacent frame picture does not satisfy the first matching condition, and / or it is determined based on the text edit distance that the text detection box in the adjacent frame picture does not satisfy the second matching condition, then the following trajectory set of the text detection box is initialized according to the text detection box in the next frame picture in the adjacent frame picture.

[0086] Specifically, if the text detection frame in the adjacent frame pictures does not meet the first matching condition and does not meet the second matching condition, the following track set of the text detection frame is initialized according to the text detection frame in the next frame picture in the adjacent frame pictures. In other words, if the intersection ratio of the text detection frames in the adjacent frame pictures is less than the preset intersection ratio and the text edit distance between the text detection frames is greater than the preset edit distance, a new text detection frame following track is created for the text detection frame in the next frame picture in the adjacent frame pictures.

[0087] Alternatively, if the text detection frame in the adjacent frame picture does not meet the first matching condition or does not meet the second matching condition, the text detection frame follow-up track set is initialized according to the text detection frame in the next frame picture in the adjacent frame picture. In other words, if the intersection ratio of the text detection frames in the adjacent frame pictures is less than the preset intersection ratio or the text edit distance between the text detection frames is greater than the preset edit distance, the text detection frame in the next frame picture in the adjacent frame pictures is initialized as a new text detection frame follow-up track.

[0088] It can be understood that if a text detection frame in the next frame of adjacent frames does not match the text detection frame in the previous frame, it means that the text detection frame in the next frame has never appeared in the previous frame, that is, it is a new text detection frame. Therefore, it is necessary to use the text detection frame in the next frame to initialize the following track to create a new text detection frame following track.

[0089] For example, reference Figure 2 , Figure 2 is a schematic diagram of two frames of images in the video to be processed during the subtitle extraction process, wherein the station logo in each frame of the video to be processed, namely Figure 2 The text detection frames of the area where "XXTV" is located in the two adjacent frames of the picture are the same, and can be used as the same text detection frame to follow the trajectory. However, the subtitles in the video to be processed may change in real time, that is, the subtitles of the next frame are different from those of the previous frame. Figure 2 The intersection-and-join ratio of the text detection frame "Looking up at the bright moon, looking down at my hometown" in the previous frame and the text detection frame "I wonder if it's frost on the ground" in the previous frame is less than the preset intersection-and-join ratio, and the text edit distance is greater than the preset text edit distance, that is, the text detection frames in the two frames do not match. At this time, the text detection frame in the subtitle area of ​​the next frame needs to create a new text detection frame following track, that is, the text detection frame "Looking up at the bright moon, looking down at my hometown" in the next frame is a new text detection frame and needs to be initialized as a new text detection frame following track.

[0090] In this way, if new subtitles appear in the video to be processed, and the text detection frame corresponding to the new subtitles must not match the text detection frame in the previous frame, the text detection frame corresponding to the subtitles is initialized to a new text detection frame following track. In other words, each subtitle will have a corresponding text detection frame following track, so that the text detection frame corresponding to each subtitle can be accurately detected, thereby improving the accuracy of extracting the text detection frame corresponding to the subtitle.

[0091] 105. Determine a reference subtitle area according to a set of text detection frames following a track of the video to be processed.

[0092] Optionally, determining the reference subtitle area based on the text detection box following trajectory set of the video to be processed may include: selecting coordinate information corresponding to a reference text detection box from each text detection box following trajectory set of the video to be processed in turn; determining candidate values ​​of the image area corresponding to the reference text detection box based on a preset algorithm and the coordinate information corresponding to the reference text detection box; and selecting the image area corresponding to the maximum value among the candidate values ​​as the reference subtitle area.

[0093] It should be noted that, assuming that the set of text detection boxes following the trajectory of the video to be processed is H, H can be expressed as

[0094] H={T1,…,T i ,…,T n}

[0095] Among them, H contains n text detection frame following tracks, that is, the subtitle extraction device believes that there are n different text areas in the video, and for each text detection frame following track in the video to be processed, a reference text detection frame B is selected for Ti in turn. t , B t ∈T i .

[0096] Reference Figure 3 , Figure 3 is a coordinate diagram of the text detection box. t Use coordinates

[0097] B t =(x t ,y t ,w t ,h t )

[0098] To represent, where x t and t is the horizontal and vertical coordinates of the upper left corner of the text detection box, w t and h t They are the width and height of the text detection box respectively.

[0099] The preset algorithm for calculating the candidate value R(x,y) of the image area corresponding to the reference text detection box is:

[0100]

[0101] That is to say, the tracking track of the text detection frame in each frame of the video to be processed is updated once, and the candidate value of the image area corresponding to the text detection frame increases by one number. It can be understood that, generally speaking, the text detection frame in the subtitle area of ​​the video to be processed changes the fastest, that is, the text detection frame in the subtitle area has the most tracking tracks, so the candidate value of the area where the text detection frame corresponding to the subtitle appears the most times is the largest.

[0102] Reference Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a picture after two adjacent frames of a video to be processed are merged. It is assumed that the video to be processed includes picture frame 10 and picture frame 20, and picture frame 10 is the first frame of the video to be processed. Since the text detection frame 22 of the station logo "XXTV" in picture frame 20 and the text detection frame 12 of the station logo "XXTV" in picture frame 10 in the two adjacent frames meet the first matching condition and the second matching condition, they belong to the same text detection frame following set, and the text detection frames corresponding to the subtitles of the two adjacent frames are not the same, that is, the text detection frame 21 in picture frame 20 does not match the text detection frame 11 in picture frame 10, that is, the text detection frame 21 is a newly appeared text detection frame, so the text detection frame 21 and the text detection frame 11 are two text detection frames. The text detection frame follows the trajectory. At this time, the text detection frame 11 in the picture frame 10 and the text detection frame 22 in the picture frame 20 corresponding to the video to be processed are calculated to be the same text following trajectory. Therefore, the R value of the picture area corresponding to the text detection frame in the video to be processed is 1, and the text detection frame 11 in the picture frame 10 and the text detection frame 21 in the picture frame 20 are two text detection frames, that is, the R value of the picture area corresponding to the text detection frame 11 is 1, and the R value of the picture area corresponding to the text detection frame 21 is also 1, that is to say, after the picture frame 10 and the picture frame 20 are merged, the R value of the overlapping part of the area corresponding to the text detection frame 11 and the text detection frame 21 in the merged picture frame 30 is 2. The candidate value of the area where the text detection frame 11 and the text detection frame 21 corresponding to the subtitles in the two frames of picture 10 and picture 20 appear repeatedly is the largest, that is, Figure 4 The candidate value of the middle gray area is the largest. This means that the area with more text following tracks in the image has a larger candidate value, and the text in the subtitle area of ​​the video to be processed has the most changes, that is, the text detection box in the subtitle area follows the most tracks, and the candidate value of the subtitle area is also the largest.

[0103] In this way, the position information of the subtitle area can be accurately distinguished according to the candidate value, thereby improving the accuracy of subtitle extraction.

[0104] Assuming that the largest candidate value R is r, calculate the binary matrix M:

[0105]

[0106] The forward rectangular frame of the largest connected area with an M value of 1 is selected as the reference subtitle area in the video to be processed.

[0107] In this way, the electronic device follows the trajectory set for each text detection frame in the processed video

[0108] 106. Extract subtitles of the video to be processed based on the reference subtitle area.

[0109] Optionally, extracting subtitles of a video to be processed based on a reference subtitle area may include: obtaining coordinate information of the reference subtitle area and coordinate information of all text detection boxes in a set of text detection boxes following trajectories of the video to be processed; extracting subtitles of the video to be processed based on the coordinate information of the reference subtitle area and coordinate information of all text detection boxes in a set of text detection boxes following trajectories of the video to be processed, wherein a difference between the coordinate information of the subtitle area of ​​the video to be processed and the coordinate information of the reference subtitle area is less than a preset value.

[0110] The reference subtitle area can be represented by coordinates (x s ,y s ,w s ,h s )express.

[0111] It is understandable that since the reference subtitle area is the area where the text detection frames corresponding to all subtitles in the video to be processed overlap the most, the coordinates of the text detection frames corresponding to the subtitles in the video to be processed are closer in ordinate and height of the text circumscribed rectangular frame, i.e., the text detection frame, than those of the reference subtitle area. Therefore, in order to accurately obtain all subtitles of the video to be processed, it is necessary to determine the text detection frames corresponding to all subtitles from all text detection frames of the video to be processed based on the reference subtitle area, i.e., select the text detection frame following tracks whose ordinate and text detection frame are equal to or have very small errors with those of the reference subtitle area and text detection frame height from all following track sets of the video to be processed, and consider them as subtitles, and extract the subtitles of the video to be processed from the text detection frames corresponding to the subtitles.

[0112] Optionally, extracting the subtitles of the video to be processed based on the coordinate information of the reference subtitle area and the coordinate information of all text detection boxes in the trajectory set followed by the text detection box of the video to be processed includes: if the difference between the vertical coordinate of the text detection box of the video to be processed and the vertical coordinate of the reference subtitle area is less than a first preset value, and the difference between the height of the text detection box of the video to be processed and the height of the reference subtitle area is less than a second preset value, then extracting the text in the text detection box as the subtitle of the video to be processed.

[0113] Among them, the first preset value and the second preset value are both relatively small, and the first preset value can be δ y , the second preset value may be δ h For example, for the text detection box following the trajectory T i , select a text detection box B t , B t =(x t ,y t ,w t ,h t ), when the text detection box B t Satisfy t -y s |<δ y and|h t -h s |<δ h When the text detection box follows the trajectory T i For subtitles.

[0114] In actual application scenarios, some Internet videos contain a lot of text content, and there are a large number of other texts that are not subtitles in the video to be processed, such as bullet comments or advertising text. In order to more accurately extract the subtitles in the video to be processed, the text detection box corresponding to the obtained subtitles can be post-processed following the trajectory. The rules followed by post-processing are, for example, (1) the position of the same subtitle in different frames of the video to be processed is the same, even if the detection box output by the text detection algorithm for the same subtitle in different frames is biased, it should be within a certain error range. (2) Select horizontal text as subtitles. In the entire video to be processed, the vertical coordinates of the horizontal text corresponding to different subtitles are the same or the error is small.

[0115] From the above, it can be seen that after the subtitle extraction device in the embodiment of the present invention obtains the video to be processed, it performs frame processing on the video to be processed to obtain a multi-frame image sequence, and then performs text detection processing on each frame of the image to identify the text detection frame in each frame of the image, and performs temporal trajectory tracking on the text detection frame in each frame of the image to obtain a text detection frame following trajectory set of the video to be processed; according to the text detection frame following trajectory set of the video to be processed, a reference subtitle area is determined; and the subtitles of the video to be processed are extracted based on the reference subtitle area. In this way, all the text in the video to be processed is detected and the trajectory of the text detection frame is followed, and the subtitle area of ​​the video to be processed is calculated according to the following trajectory of the text detection frame, so as to distinguish the subtitles in the video to be processed from other texts, and then extract the subtitles, thereby improving the accuracy of subtitle extraction in the video.

[0116] According to the method described in the above embodiment, the following is further described in detail with examples.

[0117] In this embodiment, the subtitle extraction device will be specifically integrated into an electronic device as an example for description. Figure 5 As shown, Figure 5 This is another flowchart of subtitle extraction provided by an embodiment of the present application.

[0118] A subtitle extraction method, the specific process is as follows:

[0119] 201. The electronic device obtains a video to be processed;

[0120] For example, the electronic device obtains the video to be processed input by the user, or the electronic device automatically detects the video to be processed. The video to be processed may be an Internet video or a TV video, etc. The video to be processed may contain subtitle text and non-subtitle text. The so-called non-subtitle text may be a watermark, barrage, station logo or advertising slogan in the video.

[0121] 202. The electronic device performs frame processing on the video to be processed to obtain a multi-frame image sequence;

[0122] For example, the electronic device performs frame processing on the acquired video to be processed according to the original frame rate or sampling frame rate of the video to be processed, thereby obtaining a multi-frame image sequence arranged in chronological order.

[0123] 203. The electronic device performs text detection processing on each frame of the image to identify a text detection frame in each frame of the image;

[0124] For example, the electronic device can use an end-to-end image segmentation-based algorithm based on deep learning to perform text detection processing on each frame of the image, thereby outputting a text detection box for each frame of the image. The text detection box is a positive rectangular box and can be represented by coordinates Bt (x, y, w, h), where Bt represents a text detection box in the tth frame of the image, x and y are the horizontal and vertical coordinates of the upper left corner of the text detection box, respectively, and w and h are the height and width of the text detection box, respectively.

[0125] 204. The electronic device obtains a text edit distance of a text detection frame in each adjacent frame of the picture, and coordinate information of a corresponding text detection frame in each adjacent frame of the picture;

[0126] The electronic device obtains the coordinate information of the text detection frame in each adjacent frame image. For example, the coordinate information of a text detection frame in the previous frame, i.e., the t-2th frame in the adjacent frame image can be expressed as B t-2 (x1, y1, w1, h1), the coordinate information of another text detection box can be expressed as B t-2 (x2, y2, w2, h2), the coordinate information of a text detection frame in the next frame, i.e., the t-1th frame, can be expressed as B t-1 (x1, y1, w1, h1), the coordinate information of another text detection box can be expressed as B t-1 (x2,y2,w2,h2).

[0127] The electronic device uses a text recognition network model to recognize the text content in the text detection box in each adjacent frame image, and calculates the text editing distance of the text detection boxes in the adjacent frame images based on the text content. For example, the text editing distance of the text detection boxes between the first frame and the second frame image, and the text editing distance of the text detection boxes between the second frame and the third frame image are calculated.

[0128] 205. The electronic device generates a set of text detection frame following trajectories of the video to be processed according to the coordinate information and the text editing distance.

[0129] The track following of the text detection frame refers to connecting the same text between different frames of the video to be processed in time sequence to form the track of the text detection frame in time. A text following track can be represented by T.

[0130] T={B s ,B s+1 ,…,B t}

[0131] B t represents the text detection box of the tth frame. Similarly, B sRepresents the text detection box of the sth frame. The text first appears in the sth frame of the video and last appears in the tth frame of the video. It needs to satisfy s <t。

[0132] The trajectory following of the text detection box is an online algorithm, that is, the text detection box following result of the s-1 frame and the text detection box following result of the s frame are generated. For the text detection box detected in the first frame, s = 1, and the initialized text detection box following result is T = {B1}.

[0133] The electronic device generates a set of text detection frame following trajectories of the video to be processed according to the coordinate information and the text editing distance, which can be specifically:

[0134] (1) Calculate the union-intersection ratio and text edit distance of the text detection boxes in adjacent frame images.

[0135] The intersection ratio of a text detection frame refers to the ratio of the intersection area of ​​two text detection frames to the union area. For example, the intersection area of ​​the text detection frame B1 in the first frame and the text detection frame B2 in the second frame is A, and the union area is B. Then the intersection ratio of the text detection frame B1 in the first frame and the text detection frame B2 in the second frame is A / B. Because the text detection frame following trajectory T is a set of text detection frames, the last text detection frame in the set is used to represent the following trajectory of the text detection frame.

[0136] (2) Determine whether the intersection ratio of the text detection boxes in adjacent frame images meets the first matching condition.

[0137] The first matching condition is that the intersection area and the ratio of the union area of ​​the corresponding areas of the text detection frames in the adjacent frame images are greater than or equal to the preset intersection ratio. If the intersection ratio of the text detection frames in the adjacent frame images is less than the preset intersection ratio, it is determined that the two text detection frames fail to match; if the intersection ratio of the text detection frames in the adjacent frame images is greater than or equal to the preset intersection ratio, the next step is entered to determine whether the edit distance of the two text detection frames meets the second matching condition.

[0138] (3) Determine whether the text edit distance of the text detection boxes in adjacent frame images meets the second matching condition.

[0139] The second matching condition is that the text edit distance of the text detection frames in adjacent frames is less than or equal to the preset edit distance. If the edit distance of the text detection frames in adjacent frames is greater than the preset edit distance, it is determined that the two text detection frames fail to match; if the edit distance of the text detection frames in adjacent frames is less than or equal to the preset edit distance, it is determined that the two text detection frames match successfully, that is, the two text detection frames are the same text.

[0140] (4) If the intersection-and-parallel ratio of the text detection frames in the adjacent frames meets the first matching condition, and the text edit distance meets the second matching condition, then the two text detection frames are determined to be matched successfully, and the text detection frame in the next frame is added to the set of the text detection frame tracking trajectory in the previous frame. For example, the text tracking trajectory of the previous frame is T t-1 , and the detection box that matches the text track is B t , then the new text trajectory is T t , T t-1 and T t The form is as follows:

[0141] T t-1 = {B s ,B s+1 ,…,B t-1},T t = {B s ,B s+1 ,…,B t-1 ,B t}.

[0142] (5) If there are multiple text detection frames in adjacent frame images whose union-intersection ratios satisfy the first matching condition and the text edit distances satisfy the second matching condition, the text detection frame with the largest union-intersection ratio during the matching process is selected as the correctly matched text detection frame.

[0143] (6) If the intersection ratio of the text detection boxes in the adjacent frame images does not satisfy the first matching condition, and / or the text edit distance does not satisfy the second matching condition, that is, a certain text detection box fails to match all the text detection boxes in the past frames, then it is considered that this is a newly appeared text. At this time, the text detection box can be used to initialize the following trajectory of the text detection box, which is equivalent to creating a new text detection box following trajectory.

[0144] 206. The electronic device selects coordinate information corresponding to a reference text detection frame from each text detection frame following trajectory in the text detection frame following trajectory set of the video to be processed in turn;

[0145] It should be noted that, assuming that the height and width of the image corresponding to the video to be processed are O and P respectively, an all-zero matrix R with a dimension of OxP is initialized; then the value of the matrix R is calculated based on the tracking trajectory set of the text detection frame. Assuming that the text detection frame tracking set of the video to be processed is H, H is expressed as follows:

[0146] H={T1,…,T i ,…,T n}

[0147] H contains n text detection frame following tracks, that is, the electronic device calculates that there are n different texts in the video to be processed. For each following track in the video, that is, for T i , select a reference text detection box B t , B t ∈T i , B t It is represented by four coordinates:

[0148] B t =(x t ,y t ,w t ,h t )

[0149] Among them, x t and t is the horizontal and vertical coordinates of the upper left corner of the reference text detection box, w t and h t They are the width and height of the reference text detection box respectively.

[0150] 207. The electronic device determines, according to a preset algorithm and coordinate information corresponding to the reference text detection frame, a candidate value of the image area corresponding to the reference text detection frame;

[0151] Calculate the reference text detection box B according to the preset algorithm t The candidate value of the corresponding image area is the R value, and the preset algorithm is

[0152]

[0153] 208. The electronic device selects a picture area corresponding to a maximum value among the candidate values ​​as a reference subtitle area.

[0154] Assuming that the largest R value in the R matrix is ​​r, calculate the binary matrix M,

[0155]

[0156] Select the positive rectangular box of the largest connected area with M value of 1 as the reference subtitle area in the video, using the coordinate (x s ,y s ,w s ,h s )express.

[0157] 209. The electronic device extracts subtitles of the video to be processed based on the reference subtitle area.

[0158] Based on the y coordinate of the reference subtitle area s and the height of the subtitle h s, we select the ordinate and height of the text detection frame from the set H of the text detection frame following the trajectory of the entire video to be processed, which are respectively the ordinate y of the reference subtitle area s and the height of the subtitle h s Text detection boxes that are equal or have very small errors follow the trajectory and are considered to be subtitles. For example, if the difference between the vertical coordinate of the text detection box of the video to be processed and the vertical coordinate of the reference subtitle area is less than a first preset value, and the difference between the height of the text detection box of the video to be processed and the height of the reference subtitle area is less than a second preset value, the text in the text detection box is extracted as the subtitle of the video to be processed.

[0159] From the above, it can be seen that after the subtitle extraction device in the embodiment of the present invention obtains the video to be processed, it performs frame processing on the video to be processed to obtain a multi-frame image sequence, and then performs text detection processing on each frame of the image to identify the text detection frame in each frame of the image, and performs temporal trajectory tracking on the text detection frame in each frame of the image to obtain a text detection frame following trajectory set of the video to be processed; according to the text detection frame following trajectory set of the video to be processed, a reference subtitle area is determined; and the subtitles of the video to be processed are extracted based on the reference subtitle area. In this way, all the text in the video to be processed is detected and the trajectory of the text detection frame is followed, and the subtitle area of ​​the video to be processed is calculated according to the following trajectory of the text detection frame, so as to distinguish the subtitles in the video to be processed from other texts, and then extract the subtitles, thereby improving the accuracy of subtitle extraction in the video.

[0160] In order to better implement the above method, an embodiment of the present invention also provides a data acquisition device, which can be integrated into an electronic device, such as a server or a terminal. The terminal may include a tablet computer, a smart TV, a mobile phone, a laptop computer and / or a personal computer.

[0161] For example, Figure 6 As shown, the data acquisition device may include an acquisition unit 301, a framing unit 302, a detection unit 303, a track following unit 304, a determination unit 305, and an extraction unit 306, as follows:

[0162] An acquisition unit 301 is used to acquire a video to be processed;

[0163] The video to be processed may be an Internet video or a TV video, etc. The video to be processed may contain subtitle text and non-subtitle text. The so-called non-subtitle text may be a watermark, a station logo or an advertisement in the video. In order to extract the subtitles from the video to be processed, it is first necessary to divide the video to be processed into a multi-frame image sequence, so as to extract the subtitles from each frame.

[0164] A frame division unit 302 is used to perform frame division processing on the video to be processed to obtain a multi-frame image sequence;

[0165] For example, the video to be processed is frame-processed according to a preset frame rate to obtain a multi-frame image sequence arranged in time order.

[0166] The detection unit 303 is used to perform text detection processing on each frame of the picture to identify the text detection frame in each frame of the picture;

[0167] For example, a text detection algorithm may be used to process each frame of the image, thereby extracting a text detection frame. Specifically, a preset text detection algorithm may be obtained; text detection processing may be performed on each frame of the image according to the text detection algorithm to extract feature information corresponding to the text in each frame of the image; and a text detection frame in each frame of the image may be determined according to the feature information.

[0168] A track following unit 304 is used to track the text detection frame in each frame of the picture in time to obtain a track set of the text detection frame in the video to be processed;

[0169] For example, based on the text detection frame obtained by text detection, trajectory tracking is performed to connect the same text between different frames of the video to be processed to form a temporal tracking trajectory of the text detection frame.

[0170] A determination unit 305 is used to determine a reference subtitle area according to a set of text detection frames following trajectories of the video to be processed;

[0171] For example, from each text detection frame following trajectory in the text detection frame following trajectory set of the video to be processed, select the coordinate information corresponding to a reference text detection frame; determine the candidate value of the image area corresponding to the reference text detection frame according to a preset algorithm and the coordinate information corresponding to the reference text detection frame; and select the image area corresponding to the maximum value among the candidate values ​​as the reference subtitle area.

[0172] The extraction unit 306 is configured to extract subtitles of the video to be processed based on the reference subtitle area.

[0173] For example, since the reference subtitle area is the area where the text detection frames corresponding to all subtitles in the video to be processed overlap the most, the coordinates of the text detection frames corresponding to the subtitles in the video to be processed are closer in vertical coordinates and height of the text circumscribed rectangular frame, i.e., the text detection frame, than those of the reference subtitle area. Therefore, in order to accurately obtain all subtitles of the video to be processed, it is necessary to determine the text detection frames corresponding to all subtitles from all text detection frames of the video to be processed based on the reference subtitle area, i.e., select the text detection frame following tracks whose vertical coordinates and text detection frames are equal to the vertical coordinates and text detection frame heights of the reference subtitle area or whose errors are very small from all the following track sets of the video to be processed, and consider them as subtitles, and extract the subtitles of the video to be processed from the text detection frames corresponding to the subtitles.

[0174] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0175] As can be seen from the above, in this embodiment, the acquisition unit 301 acquires the video to be processed; the frame division unit 302 performs frame division processing on the video to be processed to obtain a multi-frame image sequence; the detection unit 303 performs text detection processing on each frame of the image to identify the text detection frame in each frame of the image; the trajectory following unit 304 performs temporal trajectory tracking on the text detection frame in each frame of the image to obtain a text detection frame following trajectory set of the video to be processed; the determination unit 305 determines the reference subtitle area according to the text detection frame following trajectory set of the video to be processed; the extraction unit 306 extracts the subtitles of the video to be processed based on the reference subtitle area. In this way, all the text in the video to be processed is detected and the trajectory of the text detection frame is followed, and the subtitle area of ​​the video to be processed is calculated according to the following trajectory of the text detection frame, so as to distinguish the subtitles in the video to be processed from other texts, and then extract the subtitles, thereby improving the accuracy of subtitle extraction in the video.

[0176] The present application also provides a computer device, such as Figure 7 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0177] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 7 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0178] The processor 401 is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the computer device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0179] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0180] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0181] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0182] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0183] Acquire a video to be processed; perform frame processing on the video to be processed to obtain a multi-frame image sequence; perform text detection processing on each frame of the image to identify a text detection frame in each frame of the image; perform temporal trajectory tracking on the text detection frame in each frame of the image to obtain a text detection frame following trajectory set of the video to be processed; determine a reference subtitle area based on the text detection frame following trajectory set of the video to be processed; and extract subtitles of the video to be processed based on the reference subtitle area.

[0184] For example, a computer device divides a video to be processed into a multi-frame image sequence according to a preset frame rate, and then performs text detection processing on each frame according to a preset text detection algorithm to identify a text detection frame in each frame; the text detection frame in each frame is tracked in time to obtain a set of tracking trajectories of all text detection frames in the video to be processed, and the subtitle area of ​​the video to be processed is calculated based on the set of text detection frame tracking trajectories, so as to distinguish the subtitles from other texts in the video to be processed, and then extract the subtitles.

[0185] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.

[0186] From the above, it can be seen that after obtaining the video to be processed, the computer device of the embodiment of the present application performs frame processing on the video to be processed to obtain a multi-frame image sequence, and then performs text detection processing on each frame of the image to identify the text detection frame in each frame of the image; then, the text detection frame in each frame of the image is tracked in time to obtain a set of text detection frame tracking trajectories of the video to be processed; then, a reference subtitle area is determined based on the set of text detection frame tracking trajectories of the video to be processed; and the subtitles of the video to be processed are extracted based on the reference subtitle area. In this way, all the text in the video to be processed is detected and the trajectory of the text detection frame is followed, and the subtitle area of ​​the video to be processed is calculated based on the tracking trajectory of the text detection frame, and then the subtitles are extracted, which improves the accuracy of subtitle extraction in the video.

[0187] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0188] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the subtitle extraction methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0189] Acquire a video to be processed; perform frame processing on the video to be processed to obtain a multi-frame image sequence; perform text detection processing on each frame of the image to identify a text detection frame in each frame of the image; perform temporal trajectory tracking on the text detection frame in each frame of the image to obtain a text detection frame following trajectory set of the video to be processed; determine a reference subtitle area based on the text detection frame following trajectory set of the video to be processed; and extract subtitles of the video to be processed based on the reference subtitle area.

[0190] For example, a computer device divides a video to be processed into a multi-frame image sequence according to a preset frame rate, and then performs text detection processing on each frame according to a preset text detection algorithm to identify a text detection frame in each frame; the text detection frame in each frame is tracked in time to obtain a set of tracking trajectories of all text detection frames in the video to be processed, and the subtitle area of ​​the video to be processed is calculated based on the set of text detection frame tracking trajectories, so as to distinguish the subtitles from other texts in the video to be processed, and then extract the subtitles.

[0191] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0192] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0193] Since the instructions stored in the computer-readable storage medium can execute the steps in any subtitle extraction method provided in the embodiments of the present application, the beneficial effects that can be achieved by any subtitle extraction method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0194] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations of the above-mentioned subtitle extraction aspect.

[0195] The above is a detailed introduction to a subtitle extraction method, device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A subtitle extraction method, characterized in that: include: Get the video to be processed; Performing frame processing on the video to be processed to obtain a multi-frame image sequence; Performing text detection processing on each frame of the image to identify a text detection frame in each frame of the image; Performing temporal tracking of the text detection frame in each frame of the image to obtain a set of text detection frame tracking trajectories of the video to be processed; Determine a reference subtitle area according to the set of text detection frame following trajectories of the video to be processed, wherein the reference subtitle area is an area in which the following trajectories of the text detection frames in the video to be processed are updated the most times and the text detection frames corresponding to all subtitles in the video to be processed overlap the most; A text detection frame corresponding to the position of the reference subtitle area is determined from all the text detection frames in the text detection frame following trajectory set to extract the subtitles of the video to be processed.

2. The subtitle extraction method according to claim 1, characterized in that: The step of performing temporal tracking on the text detection frame in each frame of the picture to obtain a set of text detection frame tracking trajectories of the video to be processed includes: Performing temporal tracking of the text detection frame in each frame of the image to obtain a set of tracking trajectories of the text detection frame; A set of following trajectories of the text detection frame of the video to be processed is generated according to the set of following trajectories of the text detection frame.

3. The subtitle extraction method according to claim 2, characterized in that: The step of performing temporal tracking on the text detection frame in each frame of the picture to obtain a set of tracking trajectories of the text detection frame includes: Obtaining text edit distances of text detection frames in adjacent frames of images, and coordinate information of a corresponding text detection frame in each of the adjacent frames of images; A set of follow-up trajectories of the text detection frame is generated according to the coordinate information and the text edit distance.

4. The subtitle extraction method according to claim 3, characterized in that: The generating a set of follow-up trajectories of the text detection frame according to the coordinate information and the text edit distance comprises: Determine area information of the text detection frame in the adjacent frame image according to the coordinate information; If it is determined based on the area information that the text detection box in the adjacent frame picture meets the first matching condition, and it is determined based on the text editing distance that the text detection box in the adjacent frame picture meets the second matching condition, then the text detection box of the subsequent frame picture in the adjacent frame pictures is added to the follow-up trajectory set corresponding to the text detection box of the previous frame picture.

5. The subtitle extraction method according to claim 4, characterized in that: The area information includes an intersection area and a union area, and determining, according to the area information, that the text detection frame in the adjacent frame picture meets the first matching condition includes: Obtaining the intersection area and union area of ​​the text detection frames in the adjacent frame images; Calculate the intersection ratio of the text detection frames in the adjacent frame images according to the intersection area and the union area; If the union-intersection ratio of the text detection frame is greater than or equal to the preset union-intersection ratio, it is determined that the text detection frame meets the first matching condition.

6. The subtitle extraction method according to claim 4, characterized in that: Determining, according to the text edit distance, that the text detection frame in the adjacent frame picture meets the second matching condition includes: If the text edit distance is less than or equal to the preset edit distance, it is determined that the text detection frame satisfies the second matching condition.

7. The subtitle extraction method according to claim 3, characterized in that: The obtaining of the text edit distance of the text detection frame in the adjacent frame pictures comprises: Using a preset text recognition algorithm to recognize text content in text detection boxes in adjacent frame images; The text edit distance between the text contents in the text detection frames in the adjacent frame images is calculated according to the text contents.

8. The subtitle extraction method according to claim 4, characterized in that: The method further comprises: If it is determined based on the area information that the text detection box in the adjacent frame picture does not satisfy the first matching condition, and / or it is determined based on the text editing distance that the text detection box in the adjacent frame picture does not satisfy the second matching condition, then the following trajectory set of the text detection box is initialized according to the text detection box in the next frame picture in the adjacent frame picture.

9. The subtitle extraction method according to claim 4, characterized in that: If it is determined according to the area information that the text detection frame in the adjacent frame picture satisfies the first matching condition, and it is determined according to the text edit distance that the text detection frame in the adjacent frame picture satisfies the second matching condition, then adding the text detection frame of the next frame picture in the adjacent frame picture to the following track set corresponding to the text detection frame of the previous frame picture includes: If it is determined according to the area information that there are multiple text detection frames in the adjacent frame images that meet the first matching condition and the second matching condition, then determine a target text detection frame from the multiple text detection frames according to the union-intersection ratios corresponding to the multiple text detection frames; The target text detection frame is added to a set of follow-up trajectories corresponding to the text detection frame in a previous frame of the adjacent frame.

10. The subtitle extraction method according to claim 1, characterized in that: The determining of the reference subtitle area according to the text detection frame following trajectory set of the video to be processed comprises: Selecting coordinate information corresponding to a reference text detection frame from each text detection frame following trajectory in the text detection frame following trajectory set of the video to be processed; Determine, according to a preset algorithm and coordinate information corresponding to the reference text detection frame, a candidate value of the image area corresponding to the reference text detection frame; The picture area corresponding to the maximum value among the candidate values ​​is selected as the reference subtitle area.

11. The subtitle extraction method according to claim 10, characterized in that: The step of determining a text detection frame corresponding to the position of the reference subtitle area from all the text detection frames in the text detection frame following trajectory set to extract the subtitles of the video to be processed includes: Acquire the coordinate information of the reference subtitle area and the coordinate information of all the text detection frames in the set of the text detection frames following the trajectory of the video to be processed; Extract the subtitles of the video to be processed based on the coordinate information of the reference subtitle area and the coordinate information of all text detection boxes in the set of text detection boxes following trajectories of the video to be processed, wherein the difference between the coordinate information of the subtitle area of ​​the video to be processed and the coordinate information of the reference subtitle area is less than a preset value.

12. A subtitle extraction device, characterized in that: include: An acquisition unit, used for acquiring a video to be processed; A frame division unit, used for performing frame division processing on the video to be processed to obtain a multi-frame image sequence; A detection unit, used to perform text detection processing on each frame of the image to identify a text detection frame in each frame of the image; A trajectory following unit, used to perform temporal trajectory following on the text detection frame in each frame of the picture, and obtain a set of text detection frame following trajectories of the video to be processed; A determination unit, configured to determine a reference subtitle region according to a set of following trajectories of the text detection frames of the video to be processed, wherein the reference subtitle region is a region in which the following trajectories of the text detection frames in the video to be processed are updated the most times and in which the text detection frames corresponding to all the subtitles in the video to be processed overlap the most; The extraction unit is used to determine the text detection frame corresponding to the position of the reference subtitle area from all the text detection frames in the set of text detection frames following the trajectory, so as to extract the subtitles of the video to be processed.

13. A computer device, characterized in that: The invention comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the subtitle extraction method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is loaded by a processor to execute the subtitle extraction method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the subtitle extraction method according to any one of claims 1 to 11 is implemented.

Images