Video subtitle extraction method and apparatus

Abstract

Embodiments of the present disclosure disclose a video subtitle extraction method and device. The specific implementation of the method comprises: disassembling a target video into a plurality of image frames, and numbering each image frame in chronological order; determining an initial subtitle for each image frame respectively; dividing each initial subtitle into a plurality of subtitle groups based on the similarity between the initial subtitles of adjacent image frames; for a subtitle group containing a plurality of initial subtitles, determining a target subtitle in the subtitle group based on the number of repeated occurrences and the text length of each initial subtitle in the subtitle group, and setting the start time and end time of the target subtitle based on the numbering of the image frames corresponding to the subtitle group. The implementation can extract subtitles in a video coherently and accurately.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a video subtitle extraction method and device. BACKGROUND

[0002] In the videos obtained on the Internet at present, generally, there is no independent subtitle file, and the videos contain embedded subtitles. Therefore, when a subtitle file needs to be extracted or translated, technology or manual operation is needed to obtain the subtitles. After the subtitles are obtained, the traditional translation generally relies on manual operation. Therefore, it generally takes a long time to translate a video from appearing on the Internet into a subtitle in another language.

[0003] In the prior art, the method for extracting video subtitles by using OCR (Optical Character Recognition) is relatively mature, and generally, text recognition is performed on pictures parsed frame by frame. However, the merging and time synchronization of subtitles still face many problems, which directly affect the completeness, accuracy and visual effect of the subtitles, and the subtitles affect the final accuracy of the translation. SUMMARY

[0004] Embodiments of the present disclosure provide a video subtitle extraction method and device.

[0005] In a first aspect, embodiments of the present disclosure provide a video subtitle extraction method, comprising: disassembling a target video into a plurality of image frames, and numbering each image frame in time sequence; determining initial subtitles of each image frame respectively; dividing the initial subtitles into a plurality of subtitle groups based on the similarity between the initial subtitles of adjacent image frames; for a subtitle group containing a plurality of initial subtitles, determining a target subtitle in the subtitle group based on the number of repeated appearances and the text length of each initial subtitle in the subtitle group, and setting a start time and an end time of the target subtitle based on the numbering of the image frames corresponding to the subtitle group.

[0006] In some embodiments, the method further comprises: calculating the similarity between the target subtitles in adjacent subtitle groups; and in response to detecting that the similarity between the target subtitles is greater than a predetermined value, taking the target subtitle with the longest text length as the target subtitle of each subtitle group in the adjacent subtitle groups.

[0007] In some embodiments, the determining of the initial subtitles of each image frame respectively comprises: performing text recognition on each image frame by using an optical character recognition technology to obtain text information, text position and confidence; and filtering out invalid text information according to at least one of the text language, the text length, the confidence and the text position to obtain the initial subtitles.

[0008] In some embodiments, the method of disassembling a target video into a plurality of image frames comprises: decoding the middle part of the target video frame by frame to obtain at least one middle video frame; inputting the at least one middle video frame into a subtitle region detection model to output the subtitle region of each middle video frame; determining the subtitle region of the target video according to the subtitle region of each middle video frame; cropping the non-subtitle region of the target video according to the subtitle region of the target video to obtain a cropped video; and decoding the cropped video frame by frame to obtain a plurality of image frames.

[0009] In some embodiments, the method of dividing the initial subtitles of adjacent image frames into a plurality of subtitle groups based on the similarity between the initial subtitles comprises: setting the initial subtitle of the first image frame as a comparison subtitle, setting the initial subtitle of the second image frame as a current subtitle, creating a subtitle group, and adding the comparison subtitle to the subtitle group; performing the following grouping steps: in response to the difference between the number of the image frame where the current subtitle is located and the number of the image frame where the comparison subtitle is located being within a predetermined range, calculating the similarity between the current subtitle and the comparison subtitle; in response to the similarity being greater than a predetermined threshold, adding the current subtitle to the subtitle group where the comparison subtitle is located, resetting the number of trials to zero, and updating the current subtitle based on the initial subtitle of the next image frame of the image frame where the current subtitle is located, and continuing to perform the grouping steps; in response to the similarity being less than or equal to the predetermined threshold and the number of trials not reaching a predetermined number, accumulating the number of trials, and updating the current subtitle based on the initial subtitle of the next image frame of the image frame where the current subtitle is located, and continuing to perform the grouping steps; in response to the similarity being less than or equal to the predetermined threshold and the number of trials reaching the predetermined number, creating a new subtitle group, taking the current subtitle when the number of trials is 1 as a new comparison subtitle and adding it to the new subtitle group, taking the initial subtitle of the next image frame of the image frame where the new comparison subtitle is located as a new current subtitle, and continuing to perform the grouping steps.

[0010] In some embodiments, updating the current subtitle based on the initial subtitle of the next image frame of the image frame where the current subtitle is located and continuing to perform the grouping steps comprises: in response to the length of the current subtitle being greater than the length of the comparison subtitle, updating the comparison subtitle based on the current subtitle, and taking the initial subtitle of the next image frame of the image frame where the current subtitle is located as a new current subtitle, and continuing to perform the grouping steps.

[0011] In some embodiments, the method further comprises: in response to the difference between the number of the image frame where the current subtitle is located and the number of the image frame where the comparison subtitle is located not being within the predetermined range, detecting whether the number of lines of the current subtitle is greater than 1 and equal to the number of lines of the comparison subtitle; and in response to detecting that the number of lines of the current subtitle is greater than 1 and equal to the number of lines of the comparison subtitle, adding the current subtitle to the subtitle group where the comparison subtitle is located.

[0012] In some embodiments, determining the target subtitle in the subtitle group based on the number of times each initial subtitle appears and the text length of each initial subtitle in the subtitle group includes: selecting a predetermined number of initial subtitles in descending order of the number of times the initial subtitles appear in the subtitle group to obtain a candidate subtitle set; and selecting the candidate subtitle with the longest text length from the candidate subtitle set as the target subtitle in the subtitle group.

[0013] In some embodiments, the method further includes: obtaining the translation of the target subtitle in each subtitle group; and adding the translation to the target video according to the start time, end time, and text position of the target subtitle in each subtitle group.

[0014] Secondly, embodiments of this disclosure provide a video subtitle extraction apparatus, comprising: a splitting unit configured to split a target video into multiple image frames and number each image frame in chronological order; an extraction unit configured to determine the initial subtitle for each image frame; a grouping unit configured to divide each initial subtitle into multiple subtitle groups based on the similarity between the initial subtitles of adjacent image frames; and a determining unit configured to, for a subtitle group containing multiple initial subtitles, determine the target subtitle in the subtitle group based on the number of times each initial subtitle appears in the subtitle group and the text length, and set the start time and end time of the target subtitle based on the number of the image frame corresponding to the subtitle group.

[0015] In some embodiments, the determining unit is further configured to: calculate the similarity between target subtitles in adjacent subtitle groups; and in response to detecting that the similarity between target subtitles is greater than a predetermined value, select the target subtitle with the longest text length as the target subtitle of each subtitle group in the adjacent subtitle groups.

[0016] In some embodiments, the extraction unit is further configured to: perform text recognition on each image frame using optical character recognition technology to obtain text information, text position, and confidence level; and filter out invalid text information based on at least one of the language of the text information, text length of the text information, confidence level, and text position to obtain initial subtitles.

[0017] In some embodiments, the splitting unit is further configured to: decode the middle portion of the target video frame by frame to obtain at least one intermediate video frame; input the at least one intermediate video frame into a subtitle region detection model and output the subtitle region of each intermediate video frame; determine the subtitle region of the target video based on the subtitle region of each intermediate video frame; crop the non-subtitle region of the target video based on the subtitle region of the target video to obtain a cropped video; and decode the cropped video frame by frame to obtain multiple image frames.

[0018] In some embodiments, the grouping unit is further configured to: set the initial subtitle of the first image frame as the comparison subtitle, set the initial subtitle of the second image frame as the current subtitle, create a subtitle group, and add the comparison subtitle into the subtitle group; perform the following grouping steps: in response to a difference between a number of an image frame where the current subtitle is located and a number of an image frame where the comparison subtitle is located being within a predetermined range, calculate a similarity between the current subtitle and the comparison subtitle; in response to the similarity being greater than a predetermined threshold, add the current subtitle into the subtitle group where the comparison subtitle is located, reset the number of trials to zero, and update the current subtitle based on an initial subtitle of a next image frame of the image frame where the current subtitle is located, and continue to perform the grouping steps; in response to the similarity being less than or equal to the predetermined threshold and the number of trials not reaching a predetermined number of times, accumulate the number of trials, and update the current subtitle based on an initial subtitle of a next image frame of the image frame where the current subtitle is located, and continue to perform the grouping steps; in response to the similarity being less than or equal to the predetermined threshold and the number of trials reaching the predetermined number of times, create a new subtitle group, add the current subtitle when the number of trials is one as a new comparison subtitle into the new subtitle group, and add an initial subtitle of a next image frame of the image frame where the new comparison subtitle is located as a new current subtitle, and continue to perform the grouping steps.

[0019] In some embodiments, the grouping unit is further configured to: in response to a length of the current subtitle being greater than a length of the comparison subtitle, update the comparison subtitle based on the current subtitle, and continue to perform the grouping steps with an initial subtitle of a next image frame of the image frame where the current subtitle is located as a new current subtitle.

[0020] In some embodiments, the grouping unit is further configured to: in response to the difference between the number of the image frame where the current subtitle is located and the number of the image frame where the comparison subtitle is located not being within the predetermined range, detect whether a number of lines of the current subtitle is greater than one and equal to a number of lines of the comparison subtitle; and in response to detecting that the number of lines of the current subtitle is greater than one and equal to the number of lines of the comparison subtitle, add the current subtitle into the subtitle group where the comparison subtitle is located.

[0021] In some embodiments, the determining unit is further configured to: select a predetermined number of initial subtitles in the subtitle group in a descending order of a number of repeated occurrences of the initial subtitles, to obtain a candidate subtitle set; and select a candidate subtitle with a longest text length from the candidate subtitle set as a target subtitle in the subtitle group.

[0022] In some embodiments, the apparatus further comprises an editing unit configured to: obtain a translation of the target subtitle in each subtitle group; and add the translation into the target video according to a start time, an end time, and a text position of the target subtitle in each subtitle group.

[0023] In a third aspect, embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having stored thereon one or more computer programs that, when executed by the one or more processors, cause the one or more processors to implement the method of any one of the first aspect.

[0024] In a fourth aspect, embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method of any one of the first aspect.

[0025] In a fifth aspect, embodiments of the present disclosure provide a computer program product comprising a computer program that, when executed by a processor, implements the method of any one of the first aspect.

[0026] The video subtitle extraction method and device provided by the embodiments of the present disclosure optimizes the subtitle merging and time synchronization strategy on the basis of the existing OCR recognition result, provides an efficient, accurate and intelligent subtitle generation method, can significantly improve the recognition accuracy, time synchronization, merging strategy rationality and processing efficiency of the subtitle, and has high application value and market prospect.

[0027] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0028] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings:

[0029] Figure 1 is an exemplary system architecture diagram to which one embodiment of the present disclosure can be applied;

[0030] Figure 2 is a flowchart of one embodiment of the video subtitle extraction method according to the present disclosure;

[0031] Figure 3 is a schematic diagram of one application scenario of the video subtitle extraction method according to the present disclosure;

[0032] Figure 4 is a flowchart of another embodiment of the video subtitle extraction method according to the present disclosure;

[0033] Figure 5 is a structural schematic diagram of one embodiment of the video subtitle extraction device according to the present disclosure;

[0034] Figure 6 is a structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the sake of description, only the parts related to the application are shown in the drawings.

[0036] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the drawings and embodiments.

[0037] Figure 1 An exemplary system architecture 100 is shown, which can apply embodiments of the video subtitle extraction method or video subtitle extraction device of the present disclosure.

[0038] As shown in Figure 1 , the system architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0039] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as video players, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0040] The terminal devices 101, 102, and 103 can be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with a display screen and supporting video playing, including but not limited to a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a desktop computer, and the like. When the terminal devices 101, 102, and 103 are software, they can be installed in the above-listed electronic devices. They can be implemented as multiple software or software modules (for example, to provide distributed services) or as a single software or software module. No specific limitation is made herein.

[0041] The server 105 can be a server providing various services, for example, a background video playing server supporting a video displayed on the terminal devices 101, 102, and 103. The background video playing server can extract an embedded subtitle from the video, translate the subtitle according to a user-required language, and finally add the translated subtitle to the video. The extracted subtitle can also be used for other purposes, for example, content compliance detection, and the like.

[0042] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules to provide distributed services) or as a single software or software module. No specific limitation is made herein. The server can also be a server of a distributed system or a server combined with a blockchain. The server can also be a cloud server or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0043] It should be noted that the video subtitle extraction method provided by the embodiments of the present disclosure is generally executed by the server 105, and accordingly, the video subtitle extraction apparatus is generally arranged in the server 105. The video subtitle extraction method can also be executed by the terminal device, and accordingly, the video subtitle extraction apparatus is arranged in the terminal device.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system 100 is merely illustrative. According to implementation needs, there can be any number of terminal devices, networks, and servers.

[0045] With reference to the foregoing description, the video subtitle extraction method provided by the embodiments of the present disclosure can be implemented by using a computer program product.Figure 2 The diagram illustrates a flow 200 of an embodiment of a video subtitle extraction method according to the present disclosure. The video subtitle extraction method includes the following steps:

[0046] Step 201: Decompose the target video into multiple image frames and number each image frame in chronological order.

[0047] In this embodiment, the execution entity of the video subtitle extraction method (e.g.) Figure 1 The server shown can acquire the target video from the terminal device via a wired or wireless connection. The executing entity can use a decoder such as ffmpeg to break down the target video into multiple image frames. Then, each image frame is numbered sequentially by time. The numbers can be consecutive natural numbers from smallest to largest.

[0048] Step 202: Determine the initial captions for each image frame.

[0049] In this embodiment, text can be recognized from each image frame using OCR. If text is recognized, an initial caption is obtained. If no text is recognized in an image frame, that image frame can be filtered out. Each initial caption is associated with the number of the image frame it belongs to; that is, the image frame number can be used as an index for the initial caption.

[0050] Step 203: Based on the similarity between the initial captions of adjacent image frames, divide each initial caption into multiple caption groups.

[0051] In this embodiment, the similarity between the initial captions of adjacent image frames can be calculated by methods such as editing distance and string matching degree. The similarity between the initial captions of adjacent image frames can be calculated pairwise, and then adjacent image frames with a similarity greater than a predetermined threshold are grouped together. Optionally, a clustering algorithm can be used to cluster and group the initial captions of multiple image frames.

[0052] Step 204: For a subtitle group containing multiple initial subtitles, determine the target subtitle in the subtitle group based on the number of times each initial subtitle appears and the text length, and set the start and end times of the target subtitle based on the image frame number corresponding to the subtitle group.

[0053] In this embodiment, a single target subtitle is shared within the same subtitle group. Therefore, the best subtitle is selected from this subtitle group as the target subtitle. The initial subtitle that appears most frequently and has the longest text length in the subtitle group can be used as the target subtitle. Alternatively, weights can be set for the frequency of repetition and text length, and the score of the initial subtitle can be calculated based on the weighted sum of these two factors. The initial subtitle with the highest score is then used as the target subtitle.

[0054] The number of the first image frame in each subtitle group is taken as the starting time of the target subtitle of the subtitle group, and the number of the last image frame in the subtitle group is taken as the ending time of the target subtitle of the subtitle group.

[0055] In the prior art, since the display of subtitles in a video is dynamic, the same subtitle content can be detected in different frames, resulting in repeated storage. If the text recognized by OCR is simply spliced frame by frame, the subtitles can be repeated, misplaced or lost. For example, a complete sentence of subtitles can be split into multiple frames, but due to discontinuous time axis information, direct splicing can cause semantic errors or content loss. The method provided in the above embodiments of the present disclosure can make the splitting and merging of subtitles more accurate.

[0056] In the prior art, the subtitle text recognized by OCR usually does not have accurate time information, and the current mainstream method is to calculate the time range of the appearance of subtitles based on the frame rate. However, since the switching of subtitles is not completed instantaneously, but has a certain fade-in, fade-out or overlap, it is difficult to accurately locate the start and end time of the subtitles. For example, a piece of subtitle can continue to appear within 100 frames, but the same text can not be completely recognized in each frame, resulting in deviation in the calculation of the start and end time of the subtitles. The method provided in the above embodiments of the present disclosure can accurately match the display time of the subtitles.

[0057] In the prior art, in some cases, a complete sentence of subtitles can be misrecognized by OCR as two independent sentences, and in other cases, two different sentences of subtitles can be incorrectly merged due to close time, affecting the correct presentation of the subtitles. For example, the dialogue of a movie can span multiple shots, but the subtitles generated cannot correctly distinguish these scenes, resulting in the mixing of subtitles of different speakers, affecting readability. The method provided in the above embodiments of the present disclosure can standardize the splitting strategy of subtitles.

[0058] In summary, the method for extracting subtitles provided in the present application can overcome the limitations of the prior art in deduplication, time matching and logical splitting, thereby obtaining coherent and accurate subtitle text.

[0059] In some optional implementation manners of the present embodiment, the method further includes: calculating the similarity between the target subtitles in adjacent subtitle groups; and in response to detecting that the similarity between the target subtitles is greater than a predetermined value, taking the target subtitle with the longest text length as the target subtitle of each subtitle group in the adjacent subtitle groups.

[0060] The subtitles can be further combined, and adjacent subtitles are combined. Here, the subtitles located in the adjacent subtitle groups in the order of the video playing sequence are the target subtitles. The similarity can also be calculated according to the edit distance, the string matching degree, and the like. If the similarity is high, the target subtitle with the longest text length is taken as the target subtitle of each subtitle group in the adjacent subtitle groups.

[0061] In some optional implementations of the embodiment, the initial subtitles of the image frames are determined respectively, including: performing character recognition on the image frames by an optical character recognition technology to obtain character information, character position, and confidence; and filtering out invalid character information according to at least one of the character language, the text length, the confidence, and the character position to obtain the initial subtitles.

[0062] The optical character recognition technology is image recognition based on a neural network. In addition to detecting a text box to obtain the character position, the confidence of the detection result can also be obtained.

[0063] The invalid subtitles can be filtered out according to the language of the character information. For example, if all the videos to be recognized are in English at present, if non-English subtitles appear, the subtitles can be determined as invalid subtitles.

[0064] The invalid subtitles can be filtered out according to the text length of the character information. For example, if the text length <= 1 (in the unicode mode, one Chinese character is 3 characters), the subtitles can be determined as invalid subtitles.

[0065] The character information with a confidence lower than a predetermined confidence threshold can be determined as invalid subtitles. For example, if the confidence >= 0.89, the subtitles are determined as valid subtitles, and if the confidence >= 0.87 and (the number of lowercase letters > 5 or the number of spaces >= 1 or the number of punctuation symbols >= 1), the subtitles are also determined as valid subtitles. In addition to the above, the character information is determined as invalid subtitles.

[0066] The invalid subtitles can be filtered out according to the character position. For example, if the subtitles are close to symmetry (absolute value < 20) relative to the horizontal coordinate of the picture center point, the subtitles are determined as valid subtitles, otherwise, the subtitles are filtered (which can effectively filter out the end credits, the non-valid subtitles at the beginning of the video, and the like).

[0067] In some optional implementations of the embodiment, the target video is disassembled into a plurality of image frames, including: decoding the middle part of the target video frame by frame to obtain at least one middle video frame; inputting the at least one middle video frame into a subtitle area detection model to output the subtitle area of each middle video frame; determining the subtitle area of the target video according to the subtitle area of each middle video frame; cutting off the non-subtitle area of the target video according to the subtitle area of the target video to obtain a cut video; and decoding the cut video frame by frame to obtain a plurality of image frames.

[0068] Instead of detecting the subtitles from the complete video frames, the position of the subtitles is detected from the middle section of the video, and the video is cropped, and only the cropped video is decoded, which can reduce the decoding time and the time of recognizing the subtitles by OCR.

[0069] Because the opening and closing credits of a video may not be subtitles even if there are texts, they may be the credits of the staff. Therefore, the position of the subtitles is determined from the middle section of the video, which can avoid misjudgment.

[0070] The subtitle region detection model can be a neural network for pixel point classification, that is, a binary classifier. The probability that each pixel point in each image frame belongs to the subtitle region can be obtained from the output of the binary classifier. The text position information of the target video is obtained based on the probability that each pixel point in each image frame belongs to the subtitle region; and the subtitle region of the target video is determined based on the text position information.

[0071] In some optional implementations of the embodiment, based on the similarity between the initial subtitles of adjacent image frames, and the initial subtitles are divided into a plurality of subtitle groups, including: setting the initial subtitles of the first image frame as comparison subtitles, setting the initial subtitles of the second image frame as current subtitles, creating a subtitle group, and adding the comparison subtitles to the subtitle group; performing the following grouping steps: in response to the difference between the number of the image frame where the current subtitles are located and the number of the image frame where the comparison subtitles are located being within a predetermined range, calculating the similarity between the current subtitles and the comparison subtitles; in response to the similarity being greater than a predetermined threshold, adding the current subtitles to the subtitle group where the comparison subtitles are located, resetting the number of trials to zero, and updating the current subtitles based on the initial subtitles of the next image frame of the image frame where the current subtitles are located, and continuing to perform the grouping steps; in response to the similarity being less than or equal to the predetermined threshold and the number of trials not reaching a predetermined number, accumulating the number of trials, and updating the current subtitles based on the initial subtitles of the next image frame of the image frame where the current subtitles are located, and continuing to perform the grouping steps; in response to the similarity being less than or equal to the predetermined threshold and the number of trials reaching the predetermined number, creating a new subtitle group, taking the current subtitles when the number of trials is 1 as new comparison subtitles and adding them to the new subtitle group, taking the initial subtitles of the next image frame of the image frame where the new comparison subtitles are located as new current subtitles, and continuing to perform the grouping steps.

[0072] The valid initial subtitles are traversed and grouped according to the similarity. Similar initial subtitles are merged into target subtitles. In order to avoid jitter, after detecting dissimilar initial subtitles, several trials are performed. If the trial is successful, even if the initial subtitles of the previous several frames are dissimilar, they will be merged into a subtitle group and use the same target subtitles.

[0073] In order to prevent the subtitle group from being too large, the difference between the numbers of the compared initial subtitles is controlled by fps (frame per second).

[0074] If the difference between the numbers is within fps / 2, directly merge the similar subtitles. If the difference between the numbers exceeds fps / 2, determine whether to merge according to the number of lines of the subtitles.

[0075] In some optional implementations of the embodiment, the current subtitle is updated based on the initial subtitle of the next image frame of the image frame in which the current subtitle is located, and the grouping step is continued to be performed, including: in response to the length of the current subtitle being greater than the length of the comparison subtitle, updating the comparison subtitle based on the current subtitle, and taking the initial subtitle of the next image frame of the image frame in which the current subtitle is located as a new current subtitle, and continuing to perform the grouping step.

[0076] The comparison subtitle is updated according to the length of the subtitle, and comparison with only adjacent subtitles is avoided. Thus, the accuracy of subtitle extraction is improved.

[0077] In some optional implementations of the embodiment, the method further includes: in response to the difference between the number of the image frame in which the current subtitle is located and the number of the image frame in which the comparison subtitle is located not being within the predetermined range, detecting whether the number of lines of the current subtitle is greater than 1 and equal to the number of lines of the comparison subtitle; and in response to detecting that the number of lines of the current subtitle is greater than 1 and equal to the number of lines of the comparison subtitle, adding the current subtitle to the subtitle group in which the comparison subtitle is located.

[0078] If similar subtitle content is displayed in lines, the similar subtitle content is divided into different subtitle groups.

[0079] In some optional implementations of the embodiment, the target subtitle in the subtitle group is determined based on the number of repeated occurrences and the text length of each initial subtitle in the subtitle group, including: selecting a predetermined number of initial subtitles in the order of the number of repeated occurrences of the initial subtitles in the subtitle group from large to small to obtain a candidate subtitle set; and selecting a candidate subtitle with the longest text length from the candidate subtitle set as the target subtitle in the subtitle group.

[0080] The initial subtitles are sorted in the order of the number of repeated occurrences from large to small, and then the subtitle with the longest text length is selected, which can improve the accuracy of the subtitles.

[0081] Continuing to refer to Figure 3 , Figure 3 is a schematic diagram of an application scenario of the video subtitle extraction method according to the embodiment. In the application scenario of Figure 3 , the process of extracting subtitles is as follows:

[0082] Step 1. Obtain a video, decode 1-3 minutes of the middle part of the video into frame-by-frame pictures, use OCR technology to recognize the pictures, and then determine the most likely video area of the pictures in the minutes according to the OCR, and appropriately enlarge the area, for example, to 1 / 5 of the bottom;

[0083] Step 2. Use ffmpeg to decode the video into frame-by-frame images with numbers (e.g., crop the bottom 1 / 5).

[0084] Step 3. Use OCR to recognize the captured image and determine if the subtitles are usable, dividing them into valid subtitles and filtered subtitles. For example, using English subtitles:

[0085] a. Language judgment: For example, if the video being identified is in English, and the subtitles are not in English, then the subtitles can be filtered.

[0086] b. Length determination: If the text length is <= 1 (in Unicode mode, one Chinese character is 3 characters), then the subtitle will be filtered.

[0087] c. Confidence score judgment: If the confidence score is >= 0.89, it is judged as a valid subtitle; if the confidence score is >= 0.87 and (lowercase letters > 5 or spaces >= 1 or punctuation marks >= 1);

[0088] d. Coordinate judgment: If the subtitle is nearly symmetrical with respect to the horizontal coordinate of the image center point (absolute value < 20), it is judged as a valid subtitle; otherwise, it is a filtered subtitle (which can effectively filter out the scrolling cast list at the end of the credits, invalid subtitles at the beginning of the credits, etc.).

[0089] Step 4. Traverse the valid subtitles:

[0090] a. If this is the first iteration, then beg=0, create a new temporary subtitle group (group), and set the comparison subtitle (firstword) of the temporary subtitle group;

[0091] b. If it is within fps / 2 frames, determine whether the edit distance between the current subtitle currentword and firstword is within N times (N=3, text length<=3, N=1) or the same substring is 75%. If they are similar, add currentword to the group. If the length of currentword is greater than the length of firstword, update the value of firstword to currentword. Otherwise, try forward M times (M=5) and perform the above similarity judgment process.

[0092] c. If the number of subtitles exceeds fps / 2 frames, but after splitting with a separator, the number of array elements is greater than 1 (i.e., the number of subtitle lines is greater than 1), and the number of lines of the two subtitles is exactly equal, then they are considered to be subtitles of the same group.

[0093] d.If the edit distance is more than N times (and the operation processes of b, c have been done), it is considered that the subtitles should be different, then the lucky word strategy is used at this time (first, sort according to the frequency from large to small, take the top 10; second, sort according to the length from large to small, if the length is the same, then sort according to the frequency from large to small, take the first one), get the most suitable subtitle txt in the group, add the starting frame number beg, the ending frame number end, the most suitable subtitle txt of the group subtitle group to the srtfile subtitle group, clear the subtitle group group, assign the current frame number to beg and end, and assign the current subtitle currentword to firstword.

[0094] Step 5. Loop the above steps to finally obtain the first edition of the subtitle srtfile;

[0095] Step 6. To prevent subtitles such as x, y, and xy that should be merged together from appearing, perform a final traversal of the srtfile. Traverse the srtfile to obtain beg, end, txt, and compare the current subtitle and the next subtitle by edit distance (N = 3, same as the above strategy) or the same substring 75%. If similar, update the current subtitle text to the longer text and expand the current subtitle time range to the end time of the adjacent subtitle; otherwise, keep it unchanged. Finally, obtain the final subtitle and output.

[0096] Further reference Figure 4 , which shows the flow 400 of another embodiment of the video subtitle extraction method. The flow 400 of the video subtitle extraction method includes the following steps:

[0097] Step 401, disassemble the target video into multiple image frames, and number each image frame in chronological order.

[0098] Step 402, determine the initial subtitles of each image frame respectively.

[0099] Step 403, divide the initial subtitles into multiple subtitle groups based on the similarity between the initial subtitles of adjacent image frames.

[0100] Step 404, for a subtitle group containing multiple initial subtitles, determine the target subtitle in the subtitle group based on the number of repeated appearances and the text length of each initial subtitle in the subtitle group, and set the starting time and ending time of the target subtitle based on the numbering of the image frames corresponding to the subtitle group.

[0101] Steps 401-404 are basically the same as steps 201-204, so they will not be described again.

[0102] Step 405, obtain the translation of the target subtitle in each subtitle group.

[0103] In the embodiment, the translation of the target subtitle in each subtitle group can be obtained by manual translation or AI translation.

[0104] In step 406, the translation is added to the target video according to the start time, end time and text position of the target subtitle in each subtitle group.

[0105] In the embodiment, the position coordinates of the text obtained by step 402 can be used to calculate the accurate position of the coordinates in the target video. After translation, the corresponding subtitle and time (start time and end time) are obtained. Through the subtitle, time and coordinates, the translated subtitle can be accurately displayed below the embedded subtitle in the front-end player, so as to perfectly display the subtitle. Alternatively, if bilingual subtitles are not needed, the original subtitles in the video can be erased and replaced with target subtitles.

[0106] Further referring to Figure 5 , as an implementation of the method shown in the above figures, the disclosure provides an embodiment of a video subtitle extraction device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied in various electronic devices.

[0107] As shown in Figure 5 , the video subtitle extraction device 500 of the embodiment includes a splitting unit 501, an extraction unit 502, a grouping unit 503 and a determination unit 504. The splitting unit 501 is configured to split the target video into multiple image frames and number each image frame in time sequence; the extraction unit 502 is configured to determine the initial subtitles of each image frame respectively; the grouping unit 503 is configured to divide the initial subtitles into multiple subtitle groups based on the similarity between the initial subtitles of adjacent image frames; and the determination unit 504 is configured to determine the target subtitle in each subtitle group containing multiple initial subtitles based on the number of repeated appearances and the text length of each initial subtitle in the subtitle group, and set the start time and end time of the target subtitle based on the number of the image frame corresponding to the subtitle group.

[0108] In the embodiment, the specific processing of the splitting unit 501, the extraction unit 502, the grouping unit 503 and the determination unit 504 of the video subtitle extraction device 500 can refer to steps 201, 202, 203 and 204 in the corresponding embodiment. Figure 2

[0109] In some optional implementations of the embodiment, the determination unit 504 is further configured to: calculate the similarity between the target subtitles in adjacent subtitle groups; and in response to detecting that the similarity between the target subtitles is greater than a predetermined value, determine the target subtitle with the longest text length as the target subtitle of each subtitle group in the adjacent subtitle groups.​

[0110] In some optional implementation of the embodiment, the extraction unit 502 is further configured to perform text recognition on each image frame by an optical character recognition technology to obtain text information, text position and confidence; and filter out invalid text information according to at least one of text language, text length, confidence and text position to obtain initial subtitles.

[0111] In some optional implementation of the embodiment, the splitting unit 501 is further configured to decode the intermediate part of the target video frame by frame to obtain at least one intermediate video frame; input the at least one intermediate video frame into a subtitle region detection model to output a subtitle region of each intermediate video frame; determine the subtitle region of the target video according to the subtitle region of each intermediate video frame; crop the non-subtitle region of the target video according to the subtitle region of the target video to obtain a cropped video; and decode the cropped video frame by frame to obtain a plurality of image frames.

[0112] In some optional implementation of the embodiment, the grouping unit 503 is further configured to set the initial subtitles of the first image frame as comparison subtitles, set the initial subtitles of the second image frame as current subtitles, create a subtitle group, and add the comparison subtitles into the subtitle group; and perform the following grouping steps: in response to a difference between a number of an image frame where the current subtitles are located and a number of an image frame where the comparison subtitles are located being within a predetermined range, calculate a similarity between the current subtitles and the comparison subtitles; in response to the similarity being greater than a predetermined threshold, add the current subtitles into a subtitle group where the comparison subtitles are located, reset the number of trials to zero, and update the current subtitles based on the initial subtitles of a next image frame of the image frame where the current subtitles are located, and continue to perform the grouping steps; in response to the similarity being less than or equal to the predetermined threshold and the number of trials not reaching a predetermined number, accumulate the number of trials, update the current subtitles based on the initial subtitles of a next image frame of the image frame where the current subtitles are located, and continue to perform the grouping steps; in response to the similarity being less than or equal to the predetermined threshold and the number of trials reaching the predetermined number, create a new subtitle group, add the current subtitles when the number of trials is 1 as new comparison subtitles into the new subtitle group, and add the initial subtitles of a next image frame of the image frame where the new comparison subtitles are located as new current subtitles, and continue to perform the grouping steps.

[0113] In some optional implementation of the embodiment, the grouping unit 503 is further configured to, in response to a length of the current subtitles being greater than a length of the comparison subtitles, update the comparison subtitles based on the current subtitles, and continue to perform the grouping steps by taking the initial subtitles of a next image frame of the image frame where the current subtitles are located as new current subtitles.

[0114] In some optional implementations of the embodiment, the grouping unit 503 is further configured to: in response to the difference between the number of the image frame where the current subtitle is located and the number of the image frame where the compared subtitle is located not being within the predetermined range, detecting whether the number of lines of the current subtitle is greater than 1 and equal to the number of lines of the compared subtitle; and in response to detecting that the number of lines of the current subtitle is greater than 1 and equal to the number of lines of the compared subtitle, adding the current subtitle in the subtitle group where the compared subtitle is located.

[0115] In some optional implementations of the embodiment, the determining unit 504 is further configured to: select a predetermined number of initial subtitles in the order of the number of repeated occurrences of the initial subtitles in the subtitle group from large to small, to obtain a candidate subtitle set; and select a candidate subtitle with the longest text length from the candidate subtitle set as the target subtitle in the subtitle group.

[0116] In some optional implementations of the embodiment, the apparatus further includes an editing unit (not shown in the figure) configured to: obtain a translation of the target subtitle in each subtitle group; and add the translation into the target video according to the start time, end time and text position of the target subtitle in each subtitle group.

[0117] It should be noted that in the technical solutions of the present disclosure, the collection, collection, update, analysis, processing, use, transmission, storage, etc. of user personal information involved in the technical solutions comply with relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken to prevent illegal access to user personal information data and to maintain user personal information security and network security.

[0118] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.

[0119] An electronic device includes one or more processors; a storage device having stored thereon one or more computer programs which, when executed by the one or more processors, cause the one or more processors to implement the method described in flow 200 or 400.

[0120] A computer readable medium having stored thereon a computer program, wherein the computer program is executed by a processor to implement the method described in flow 200 or 400.

[0121] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0123] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the road zone planning method. For example, in some embodiments, the road zone planning method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the road zone planning method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the road zone planning method by any other suitable means, such as by means of firmware.

[0125] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0126] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / operations specified in the flowchart and / or block diagram block or blocks. The program code can be retrieved from storage and / or memory and / or can be executed by a processor or controller, either alone or in combination with another programming language. The program code can be executed as a stand-alone program or in a networked environment using multiple processors or controller.

[0127] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0129] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] The computer system can include clients and servers. This relationship can be. The servers are generally remote from the users and can be accessed via the Internet using a communication network. The relationship can be a client-server relationship over a communications network, and as such, the servers can be accessed by the clients using computer programs. The servers can be servers of a distributed system, or servers combined with a blockchain. The servers can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0131] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the steps described above. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0132] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for extracting video subtitles, comprising: The target video is broken down into multiple image frames, and each image frame is numbered in chronological order. Determine the initial captions for each image frame; Based on the similarity between the initial captions of adjacent image frames, each initial caption is divided into multiple caption groups; For a subtitle group containing multiple initial subtitles, the target subtitle in the subtitle group is determined based on the number of times each initial subtitle appears and the text length. The start time and end time of the target subtitle are set based on the image frame number corresponding to the subtitle group. The weights of the number of times the subtitle appears and the text length are set respectively. The score of the initial subtitle is calculated based on the weighted sum of the number of times the subtitle appears and the text length. The initial subtitle with the highest score is used as the target subtitle. The method of dividing initial subtitles into multiple subtitle groups based on the similarity between initial subtitles in adjacent image frames includes: Set the initial caption of the first image frame as the comparison caption, set the initial caption of the second image frame as the current caption, create a caption group, and add the comparison caption to the caption group; The following grouping steps are performed: In response to the difference between the number of the image frame where the current subtitle is located and the number of the image frame where the comparison subtitle is located being within a predetermined range, the similarity between the current subtitle and the comparison subtitle is calculated; in response to the similarity being greater than a predetermined threshold, the current subtitle is added to the subtitle group where the comparison subtitle is located, the number of attempts is reset to zero, and the current subtitle is updated based on the initial subtitle of the next image frame where the current subtitle is located, and the grouping steps are continued. In response to the similarity being less than or equal to the predetermined threshold and the number of attempts not reaching the predetermined number, the number of attempts is incremented, and the current subtitle is updated based on the initial subtitle of the next image frame in the image frame where the current subtitle is located, and the grouping step is continued. In response to the similarity being less than or equal to the predetermined threshold and the number of attempts reaching the predetermined number, a new subtitle group is created, the current subtitle when the number of attempts is 1 is used as the new comparison subtitle and added to the new subtitle group, the initial subtitle of the next image frame of the image frame where the new comparison subtitle is located is used as the new current subtitle, and the grouping step is continued. The difference in the initial subtitle numbers is controlled by the frames per second (fps). If the difference in numbers is within fps / 2, similar subtitles are directly merged; otherwise, the number of lines in the subtitle determines whether to merge. The subtitles are updated and compared according to their length to avoid comparing only with adjacent subtitles.

2. The method according to claim 1, wherein, The method further includes: Calculate the similarity between target subtitles in adjacent subtitle groups; In response to the detection that the similarity between the target subtitles is greater than a predetermined value, the target subtitle with the longest text length is selected as the target subtitle for each subtitle group in the adjacent subtitle group.

3. The method according to claim 1, wherein, The process of determining the initial captions for each image frame includes: By using optical character recognition technology, text is recognized in each image frame to obtain text information, text location, and confidence level; Based on at least one of the language of the text information, the text length of the text information, the confidence level, and the text position, invalid text information is filtered out to obtain the initial subtitle.

4. The method according to claim 1, wherein, The step of decomposing the target video into multiple image frames includes: Decode the middle portion of the target video frame by frame to obtain at least one intermediate video frame; The at least one intermediate video frame is input into the subtitle region detection model, and the subtitle region of each intermediate video frame is output. The subtitle region of the target video is determined based on the subtitle region of each intermediate video frame; Based on the subtitle area of ​​the target video, the non-subtitle area of ​​the target video is cropped to obtain the cropped video; The cropped video is decoded frame by frame to obtain multiple image frames.

5. The method according to claim 1, wherein, The step of updating the current subtitle based on the initial subtitle of the next image frame in the image frame where the current subtitle is located, and continuing to execute the grouping step, includes: In response to the fact that the length of the current caption is greater than the length of the comparison caption, the comparison caption is updated based on the current caption, and the initial caption of the next image frame in the image frame where the current caption is located is taken as the new current caption, and the grouping step is continued.

6. The method according to claim 5, wherein, The method further includes: In response to the fact that the difference between the number of the image frame where the current subtitle is located and the number of the image frame where the comparison subtitle is located is not within a predetermined range, it is detected whether the number of lines of the current subtitle is greater than 1 and equal to the number of lines of the comparison subtitle; In response to detecting that the line number of the current subtitle is greater than 1 and equal to the line number of the compared subtitle, the current subtitle is added to the subtitle group where the compared subtitle is located.

7. The method according to claim 1, wherein, The process of determining the target subtitles in the subtitle group based on the frequency of repetition and text length of each initial subtitle in the subtitle group includes: A predetermined number of initial subtitles are selected in descending order of the frequency of repetition of the initial subtitles in the subtitle group to obtain a candidate subtitle set; Select the candidate subtitle with the longest text length from the candidate subtitle set as the target subtitle in the subtitle group.

8. The method according to claim 3, wherein, The method further includes: Obtain the translation of the target subtitle in each subtitle group; The translation is added to the target video based on the start time, end time, and text position of the target subtitle in each subtitle group.

9. A video subtitle extraction device, comprising: The splitting unit is configured to break down the target video into multiple image frames and number each image frame in chronological order. The extraction unit is configured to determine the initial captions for each image frame; Grouping units are configured to divide each initial caption into multiple caption groups based on the similarity between initial captions in adjacent image frames; The determining unit is configured to, for a subtitle group containing multiple initial subtitles, determine the target subtitle in the subtitle group based on the number of times each initial subtitle appears and the text length of each initial subtitle in the subtitle group, and set the start time and end time of the target subtitle based on the image frame number corresponding to the subtitle group. The weights of the number of times the initial subtitle appears and the text length are set respectively, and the score of the initial subtitle is calculated based on the weighted sum of the number of times the initial subtitle appears and the text length. The initial subtitle with the highest score is taken as the target subtitle. The grouping unit is further configured as follows: Set the initial caption of the first image frame as the comparison caption, set the initial caption of the second image frame as the current caption, create a caption group, and add the comparison caption to the caption group; The following grouping steps are performed: In response to the difference between the number of the image frame where the current subtitle is located and the number of the image frame where the comparison subtitle is located being within a predetermined range, the similarity between the current subtitle and the comparison subtitle is calculated; in response to the similarity being greater than a predetermined threshold, the current subtitle is added to the subtitle group where the comparison subtitle is located, the number of attempts is reset to zero, and the current subtitle is updated based on the initial subtitle of the next image frame where the current subtitle is located, and the grouping steps are continued. In response to the similarity being less than or equal to the predetermined threshold and the number of attempts not reaching the predetermined number, the number of attempts is incremented, and the current subtitle is updated based on the initial subtitle of the next image frame in the image frame where the current subtitle is located, and the grouping step is continued. In response to the similarity being less than or equal to the predetermined threshold and the number of attempts reaching the predetermined number, a new subtitle group is created, the current subtitle when the number of attempts is 1 is used as the new comparison subtitle and added to the new subtitle group, the initial subtitle of the next image frame of the image frame where the new comparison subtitle is located is used as the new current subtitle, and the grouping step is continued. The difference in the initial subtitle numbers is controlled by the frames per second (fps). If the difference in numbers is within fps / 2, similar subtitles are directly merged; otherwise, the number of lines in the subtitle determines whether to merge. The subtitles are updated and compared according to their length to avoid comparing only with adjacent subtitles.

10. An electronic device, comprising: One or more processors; Storage device, on which one or more computer programs are stored, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images