Subtitle literal display method and device, electronic equipment and readable storage medium

Abstract

The application discloses a subtitle word-by-word display method and device, electronic equipment and a readable storage medium. The subtitle word-by-word display method comprises the following steps: extracting each image frame from a target video, detecting a subtitle area in each image frame by using a text detection model to obtain the coordinates of each subtitle area, wherein the text detection model is obtained by training pictures with completed subtitle area annotation; extracting text in each image frame according to the coordinates of each subtitle area to obtain a string queue composed of each string containing a timestamp; comparing each string in the string queue with a corresponding previous string in sequence to determine a subtitle word-by-word recognition result, and performing word-by-word display of the subtitle according to the subtitle word-by-word recognition result and the corresponding timestamp. The application solves the technical problem of low accuracy of word-by-word display of the subtitle.

Description

Technical Field

[0001] This application relates to the field of display technology, and in particular to a method, apparatus, electronic device, and readable storage medium for displaying subtitles character by character. Background Technology

[0002] As people's entertainment options become increasingly diverse, more and more people choose to sing in their spare time to relax and unwind. Lyrics in music videos (MVs) are a crucial way for singers to control their pace and rhythm. Currently, the word-by-word text displayed in MV lyrics is primarily obtained through OCR (optical character recognition) of the color of the lyrics subtitles in the target MV video. This color-based method only retains the color features of the original lyrics subtitles, while the text features are simplified. Therefore, it extracts too few text features from the lyrics subtitles in the target MV video, resulting in a mismatch with the corresponding time and consequently, low accuracy in generating word-by-word text. Summary of the Invention

[0003] The main objective of this application is to provide a method, electronic device, and computer-readable storage medium for displaying subtitles word by word, aiming to solve the technical problem of low accuracy in displaying subtitles word by word.

[0004] To achieve the above objectives, this application provides a method for displaying subtitles character by character, the method comprising:

[0005] Each image frame is extracted from the target video, and the subtitle region in each image frame is detected by a text detection model to obtain the coordinates of each subtitle region. The text detection model is trained on images with subtitle regions already annotated.

[0006] Based on the coordinates of each subtitle area, the text in each image frame is extracted to obtain a string queue consisting of strings containing timestamps;

[0007] Each string in the string queue is compared with the corresponding preceding string to determine the subtitle character recognition result, and the subtitle is displayed character by character based on the subtitle character recognition result and the corresponding timestamp.

[0008] Optionally, each image frame is extracted from the target video, and the subtitle region in each image frame is detected by a text detection model to obtain the coordinates of each subtitle region. The text detection model is trained based on images with completed subtitle region annotation.

[0009] Based on the coordinates of each subtitle area, the text in each image frame is extracted to obtain a string queue consisting of strings containing timestamps;

[0010] Each string in the string queue is compared with the corresponding preceding string to determine the subtitle character recognition result, and the subtitle is displayed character by character based on the subtitle character recognition result and the corresponding timestamp.

[0011] Optionally, the step of sequentially comparing each string in the string queue with its corresponding preceding string to determine the subtitle character-by-character recognition result, and then displaying the subtitle character-by-character based on the subtitle character-by-character recognition result and the corresponding timestamp, includes:

[0012] Each string in the string queue is compared with the corresponding preceding string in turn. If there is a difference, the string and the corresponding timestamp are added to the recognition queue.

[0013] By comparing each string in the recognition queue with the corresponding previous string, the character-by-character recognition result of the subtitle is determined, and the subtitle is displayed character by character based on the character-by-character recognition result and the corresponding timestamp.

[0014] Optionally, the step of sequentially comparing each string in the string queue with its corresponding preceding string, and adding the string and its corresponding timestamp to the recognition queue if a difference exists, includes:

[0015] Arrange the strings in the string queue according to their corresponding timestamps;

[0016] Each string is compared with the preceding string in turn. If the string is different from the preceding string, the string is added to the recognition queue.

[0017] If the string is the same as the previous string, then the string is discarded.

[0018] Optionally, the strings in the recognition queue are arranged in order of their corresponding timestamps. The step of comparing each string in the recognition queue with the corresponding preceding string to determine the subtitle character-by-character recognition result, and displaying the subtitle character-by-character based on the subtitle character-by-character recognition result and the corresponding timestamp, includes:

[0019] Determine whether the preceding string is a subset of the string;

[0020] If so, it is determined that the string and the preceding string belong to the same subtitle, and the difference between the string and the preceding string is used as the subtitle character recognition result corresponding to the string;

[0021] If not, it is determined that the string and the preceding string do not belong to the same subtitle, and the string is used as the subtitle character recognition result corresponding to the string;

[0022] Based on the timestamps corresponding to each string and the character-by-character recognition results of each subtitle, the character-by-character recognition results of each subtitle are displayed.

[0023] Optionally, the step of extracting each image frame from the target video, detecting the subtitle region in each image frame using a text detection model, and obtaining the coordinates of each subtitle region includes:

[0024] One frame is extracted from the target video at preset time intervals to obtain each image frame;

[0025] The text detection model is used to detect text in the preset regions of each image frame to obtain the coordinates of each subtitle region.

[0026] Optionally, the step of extracting each image frame from the target video, detecting the subtitle region in each image frame using a text detection model, and obtaining the coordinates of each subtitle region includes:

[0027] One frame is extracted from the target video at preset time intervals to obtain each image frame;

[0028] The text detection model is used to detect text in the preset regions of each image frame to obtain the coordinates of each subtitle region.

[0029] This application also provides a character-by-character subtitle display device, which is applied to a character-by-character subtitle display equipment, and the character-by-character subtitle display device includes:

[0030] The region detection module is used to extract each image frame from the target video, detect the subtitle region in each image frame through a text detection model, and obtain the coordinates of each subtitle region. The text detection model is trained on images with completed subtitle region annotation.

[0031] The text extraction module is used to extract the text in each image frame according to the coordinates of each subtitle area, and obtain a string queue consisting of strings containing timestamps;

[0032] The identification and determination module is used to sequentially compare each string in the string queue with the corresponding preceding string to determine the subtitle character recognition result, and to display the subtitle character by character according to the subtitle character recognition result and the corresponding timestamp.

[0033] Optionally, the identification and determination module is further configured to:

[0034] Each string in the string queue is compared sequentially with the preceding string.

[0035] If the string is longer than the preceding string, the difference between the string and the preceding string is used as the character-by-character recognition result of the subtitle corresponding to the string;

[0036] If the string is shorter than the preceding string, then the string is used as the character-by-character recognition result of the subtitle corresponding to the string;

[0037] If the string is the same as the previous string, then the string is discarded;

[0038] Based on the timestamps corresponding to each string and the character-by-character recognition results of each subtitle, the character-by-character recognition results of each subtitle are displayed.

[0039] Optionally, the identification and determination module is further configured to:

[0040] Each string in the string queue is compared with the corresponding preceding string in turn. If there is a difference, the string and the corresponding timestamp are added to the recognition queue.

[0041] By comparing each string in the recognition queue with the corresponding previous string, the character-by-character recognition result of the subtitle is determined, and the subtitle is displayed character by character based on the character-by-character recognition result and the corresponding timestamp.

[0042] Optionally, the identification and determination module is further configured to:

[0043] Arrange the strings in the string queue according to their corresponding timestamps;

[0044] Each string is compared with the preceding string in turn. If the string is different from the preceding string, the string is added to the recognition queue.

[0045] If the string is the same as the previous string, then the string is discarded.

[0046] Optionally, the identification and determination module is further configured to:

[0047] Determine whether the preceding string is a subset of the string;

[0048] If so, it is determined that the string and the preceding string belong to the same subtitle, and the difference between the string and the preceding string is used as the subtitle character recognition result corresponding to the string;

[0049] If not, it is determined that the string and the preceding string do not belong to the same subtitle, and the string is used as the subtitle character recognition result corresponding to the string;

[0050] Based on the timestamps corresponding to each string and the character-by-character recognition results of each subtitle, the character-by-character recognition results of each subtitle are displayed.

[0051] Optionally, the region detection module is further configured to:

[0052] One frame is extracted from the target video at preset time intervals to obtain each image frame;

[0053] The text detection model is used to detect text in the preset regions of each image frame to obtain the coordinates of each subtitle region.

[0054] Optionally, the text extraction module is further used for:

[0055] Based on the coordinates of the subtitle area corresponding to each image frame, OCR text recognition is performed on each image frame to obtain the corresponding strings;

[0056] Based on the timestamp contained in the image frame corresponding to each string, the strings are arranged to obtain the string queue.

[0057] This application also provides an electronic device, which is a physical device, comprising: a memory, a processor, and a program for the subtitle display method stored in the memory and executable on the processor. When the program for the subtitle display method is executed by the processor, it can implement the steps of the subtitle display method as described above.

[0058] This application also provides a computer-readable storage medium storing a program for implementing a method for displaying subtitles word by word. When the program for displaying subtitles word by word is executed by a processor, it implements the steps of the method for displaying subtitles word by word as described above.

[0059] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for displaying subtitles word by word.

[0060] This application provides a method, apparatus, electronic device, and readable storage medium for displaying subtitles character by character. First, image frames are extracted from a target video. Then, a text detection model is used to detect subtitle regions in each image frame, obtaining the coordinates of each subtitle region. The text detection model is trained on images with pre-annotated subtitle regions. Next, text is extracted from each image frame based on the subtitle region coordinates, resulting in a string queue consisting of strings including timestamps. Finally, each string in the string queue is compared sequentially with its preceding string to determine the subtitle character-by-character recognition result. The results are then compared with the corresponding... The text is displayed word by word using the corresponding timestamp. In this application, a text detection model is trained on images with labeled text regions. The text detection model extracts the coordinates of the text regions in each image frame and extracts text from each text region coordinate, preserving the text features of the original image frame. This overcomes the technical defect of simplified text features generated by color-based OCR recognition. Then, the text recognition result is determined based on the timestamp corresponding to each string and the comparison result of each string in the string queue with the previous string. This achieves time matching consistency with each image frame and improves the accuracy of text display. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart illustrating the first embodiment of the text-by-text display method of this application;

[0064] Figure 2 This is a schematic diagram of the coordinate system for lyric subtitle calibration in the first embodiment of the subtitle display method of this application;

[0065] Figure 3 This is a flowchart illustrating steps S31 to S35 in the first embodiment of the subtitle display method of this application.

[0066] Figure 4 This is a schematic diagram of the components of the character-by-character subtitle display device of this application;

[0067] Figure 5This is a schematic diagram of the device structure of the hardware operating environment involved in the subtitle display method in the embodiments of this application.

[0068] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0069] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] Example 1

[0071] Currently, the word-by-word text displayed in music video lyrics is mainly obtained by OCR recognition of the color of the lyrics subtitles in the target music video. This color recognition-based method only retains the color features of the original lyrics subtitles, while the features of the text portion are greatly simplified. Therefore, the text features extracted from the lyrics subtitles in the target music video are insufficient and do not match the corresponding time, resulting in low accuracy of the generated word-by-word text display.

[0072] This application provides a method for displaying subtitles character by character. In the first embodiment of this application's method for displaying subtitles character by character, refer to... Figure 1 The method for displaying subtitles word by word includes:

[0073] Step S10: Extract each image frame from the target video, detect the subtitle region in each image frame using a text detection model, and obtain the coordinates of each subtitle region. The text detection model is trained based on images with completed subtitle region annotations.

[0074] Step S20: Extract the text from each image frame according to the coordinates of each subtitle area to obtain a string queue consisting of strings containing timestamps;

[0075] Step S30: Compare each string in the string queue with the corresponding preceding string in turn to determine the subtitle character recognition result, and display the subtitle character by character according to the subtitle character recognition result and the corresponding timestamp.

[0076] In this embodiment, it should be noted that the text detection model can be the YOLO model (You Only Look Once, object detection model). The YOLO model achieves end-to-end object detection through a series of convolutional operations. The YOLO model divides the image into an S x S grid, with each grid responsible for detecting objects falling within it. Finally, it outputs the bounding box of the contained object, its location information, and the confidence scores for all categories. (Refer to...) Figure 2 The coordinates of the target subtitle's bounding box were detected from the MV subtitle image frame. Figure 2 (The box in the image); the string is the string composed of each lyric text; the method used in the process of extracting the text from each image frame can be OCR recognition.

[0077] As an example, steps S10 to S30 include: extracting each image frame and its corresponding timestamp from the target video according to a preset time interval; detecting the subtitle region in each image frame using a trained target detection model to obtain the coordinates of each subtitle region, wherein the text detection model is trained on images whose subtitle region coordinates have been manually marked; performing OCR text recognition on each image frame according to the subtitle region coordinates to extract the string corresponding to each image frame, and adding each string and its corresponding timestamp to a string queue; comparing each string in the string queue according to the timestamp order, and if the string is inconsistent with the previous string, adding the string and its corresponding timestamp to a recognition queue; comparing each timestamp in the recognition queue according to the timestamp order, and if the string is longer than the previous string, using the comparison result of the string and the previous string as the subtitle character recognition result corresponding to the string; if the string is shorter than the previous string, using the string as the subtitle character recognition result corresponding to the string; and playing and displaying each subtitle character recognition result according to its corresponding timestamp.

[0078] Among them, reference Figure 3 The step of sequentially comparing each string in the string queue with the corresponding preceding string to determine the subtitle character-by-character recognition result, and displaying the subtitle character-by-character based on the subtitle character-by-character recognition result and the corresponding timestamp, includes:

[0079] Step S31: Compare each string in the string queue with the corresponding preceding string in turn;

[0080] Step S32: If the string is longer than the previous string, the difference between the string and the previous string is used as the character-by-character recognition result of the subtitle corresponding to the string.

[0081] Step S33: If the string is shorter than the previous string, then the string is used as the character-by-character recognition result of the subtitle corresponding to the string;

[0082] Step S34: If the string is the same as the previous string, then the string is discarded;

[0083] Step S35: Display the character-by-character recognition results of each subtitle based on the timestamp corresponding to each string and the character-by-character recognition results of each subtitle.

[0084] In this embodiment of the application, it should be noted that the subtitle character recognition result includes character information and a corresponding timestamp, which is used for subtitle character display and playback; the difference between the string and the previous string is the updated content of the string compared to the previous string. For example, if the difference between the string "Beijing welcomes you" and the previous string "Beijing welcomes you" is "you", then "you" is used as the subtitle character recognition result.

[0085] As an example, steps S31 to S35 include: comparing each string with its corresponding preceding string sequentially based on the timestamp corresponding to each string; if the string is longer than the preceding string, it is determined that the string and the preceding string belong to the same subtitle, and the string has updated content compared to the preceding string, and the difference between the string and the preceding string is used as the subtitle character recognition result corresponding to the string; if the string is shorter than the preceding string, it is determined that the string and the preceding string do not belong to the same subtitle, and the string is a new subtitle, and the string is used as the subtitle character recognition result corresponding to the string; if the string is the same as the preceding string, it is determined that the string and the preceding string belong to the same subtitle, and the string has no updated content compared to the preceding string, and the string is discarded; and displaying the subtitle character recognition results sequentially based on the timestamp corresponding to each string.

[0086] As another example, the step of sequentially comparing each string in the string queue with its corresponding preceding string to determine the subtitle character-by-character recognition result, and then displaying the subtitle character-by-character based on the subtitle character-by-character recognition result and the corresponding timestamp, includes:

[0087] Step A10: Compare each string in the string queue with the corresponding preceding string in turn. If there is a difference, add the string and the corresponding timestamp to the recognition queue.

[0088] Step A20: Compare each string in the recognition queue with the corresponding previous string to determine the subtitle character recognition result, and display the subtitle character by character according to the subtitle character recognition result and the corresponding timestamp.

[0089] As an example, steps A10 to A20 include: comparing each string in the string queue with its corresponding preceding string according to its corresponding timestamp; when a difference is detected between the string and the preceding string, adding the string and its corresponding timestamp to the recognition queue; if no difference is detected between the string and the preceding string, discarding the string; comparing each string with its corresponding preceding string in the recognition queue to determine the length relationship between the two strings; generating a character-by-character subtitle recognition result for the string based on the length relationship; and displaying the subtitle character by character based on the character-by-character recognition result of the string according to its corresponding timestamp.

[0090] The step of sequentially comparing each string in the string queue with its corresponding preceding string, and adding the string and its corresponding timestamp to the recognition queue if a difference exists, includes:

[0091] Step A11: Arrange the strings in the string queue according to their corresponding timestamps;

[0092] Step A12: Compare each string with the corresponding preceding string in turn. If the string is different from the preceding string, add the string to the recognition queue.

[0093] Step A13: If the string is the same as the previous string, then discard the string.

[0094] As an example, steps A11 to A13 include: sorting the strings in the string queue according to the chronological order of their corresponding timestamps; comparing each string in the string queue with its preceding string according to the timestamp order; if the string is not completely identical to the preceding string, it is determined that the string has updated content compared to the preceding string, and the string is added to the recognition queue; if the string is identical to the preceding string, it is determined that the string has no updated content compared to the preceding string, and the string is discarded.

[0095] As an example, in the word-by-word display of lyrics in a music video, the string queue is S, and each string result is i. i and i-1 are compared. If the recognition result of i is inconsistent with i-1, it is considered that the song moves to the next character, and the timestamp of the current i and the recognized character are recorded. For example, if the string corresponding to frame i is "Beijing Welcomes You" and the string corresponding to i-1 is "Beijing Welcomes", then the timestamps of "Beijing Welcomes You" and "Beijing Welcomes You" are saved and added to the recognition queue T.

[0096] The steps of comparing each string in the recognition queue with the corresponding previous string to determine the character-by-character subtitle recognition result, and displaying the subtitle character by character based on the character-by-character recognition result and the corresponding timestamp, include:

[0097] Step A21: Determine whether the preceding string is a subset of the string;

[0098] Step A22: If yes, then determine that the string and the previous string belong to the same subtitle, and take the difference between the string and the previous string as the subtitle character recognition result corresponding to the string;

[0099] Step A23: If not, determine that the string and the previous string do not belong to the same subtitle, and use the string as the subtitle character recognition result corresponding to the string;

[0100] Step A24: Display the character-by-character recognition results of each subtitle based on the timestamp corresponding to each string and the character-by-character recognition results of each subtitle.

[0101] As an example, steps A21 to A24 include: determining in the recognition queue whether the preceding string of two adjacent strings is a subset of the following string; if the preceding string is a subset of the following string, then determining that the following string and the preceding string belong to the same subtitle, and using the difference between the following string and the preceding string as the subtitle character recognition result corresponding to the following string, wherein the difference is used to represent the updated content of the following string compared to the preceding string; if the preceding string is not a subset of the following string, then determining that the following string and the preceding string do not belong to the same subtitle, and using the following string as the subtitle character recognition result corresponding to the string; and displaying each subtitle character recognition result character by character according to the timestamp corresponding to each string.

[0102] As an example, in the word-by-word display of lyrics in an MV video, traverse the recognition queue T, take out the recognition result t, and compare it with t-1. If the recognition result of t is longer than that of t-1, use t minus (t-1) to obtain the recognition difference as the result of the word-by-word lyrics, and record the timestamp of t. For example, if the difference between "Beijing Welcomes You" and "Beijing Welcomes" is "You", then "You" is taken as the word-by-word recognition result of the subtitle; if the recognition result of t is shorter than that of t-1, it is determined that a new line of lyrics appears in the video playback, then the recognition result of t is taken as the result of the word-by-word lyrics. For example, if "Beijing Welcomes You" and "Wei" are t-1 and t, then "Wei" is taken as the word-by-word recognition result of the subtitle, and timestamps are recorded for all difference results and saved as the word-by-word recognition results of the subtitles.

[0103] The step of extracting each image frame from the target video, detecting the subtitle region in each of the image frames through a text detection model, and obtaining the coordinates of each subtitle region comprises:

[0104] Step S11: extract one frame of image from the target video at preset time intervals to obtain each of the image frames;

[0105] Step S12: perform text detection on a preset region of each of the image frames through the text detection model to obtain the coordinates of each of the subtitle regions.

[0106] In this embodiment, it should be noted that the target video may be an MV lyric video, and the text detection model may be a YOLO object detection model, which is used to detect the subtitle region in the image frame, and is trained from images extracted from each MV video whose subtitle regions have been manually labeled.

[0107] As a preference, the preset time interval is 0.5s, and the preset region is the lower one-third area of the video, that is, the subtitle region. Since there is no subtitle display in the upper two-thirds area of the MV video, in order to avoid redundant recognition, the RGB values in the upper two-thirds area of the MV video are set to 0.

[0108] As an example, steps S11 to S12 include: sequentially extracting image frames from the target video file according to the preset time interval to obtain each image frame corresponding to the target video; performing subtitle region detection on the preset region of each of the image frames through the YOLO object detection model trained from images extracted from each MV video with completed subtitle region labeling, to obtain the corresponding coordinates of each subtitle region.

[0109] The step of extracting the text in each image frame according to the coordinates of each subtitle region to obtain a string queue composed of strings each containing a timestamp comprises:

[0110] Step S21: Based on the coordinates of the subtitle area corresponding to each image frame, perform OCR text recognition on each image frame to obtain the corresponding strings;

[0111] Step S22: Arrange the strings according to the timestamps contained in the image frames corresponding to each string to obtain the string queue.

[0112] As an example, steps S21 to S22 include: dividing each subtitle region containing the target subtitle according to the subtitle region coordinates corresponding to each image frame, and performing OCR text recognition on each subtitle region to obtain corresponding strings, wherein each string contains characters in the subtitle; confirming the timestamp corresponding to each string according to the image frame corresponding to each string and the timestamp of each image frame; and adding each string to the string queue according to the order of the timestamps corresponding to each string to obtain the string queue.

[0113] This application provides a method for displaying subtitles character by character. First, image frames are extracted from a target video. Then, a text detection model is used to detect subtitle regions in each image frame, obtaining the coordinates of each subtitle region. The text detection model is trained on images with pre-annotated subtitle regions. Next, text is extracted from each image frame based on the subtitle region coordinates, resulting in a string queue consisting of strings containing timestamps. Finally, each string in the string queue is compared sequentially with its preceding string to determine the subtitle character recognition result. The subtitles are then displayed character by character based on the recognition result and the corresponding timestamp. This application uses a text detection model trained on images with pre-annotated subtitle regions. The text detection model extracts the coordinates of subtitle regions in each image frame and extracts text from these coordinates, preserving the original text features of the image frames. This overcomes the technical shortcomings of simplified text features generated by color-based OCR recognition. Finally, the subtitle character recognition result is determined based on the timestamps corresponding to each string and the comparison results between each string in the string queue and its preceding string, achieving time matching consistency with each image frame and improving the accuracy of the subtitle character display.

[0114] Example 2

[0115] This application also provides a character-by-character subtitle display device, which is applied to a character-by-character subtitle display equipment, see reference. Figure 4 The subtitle display device includes:

[0116] The region detection module is used to extract each image frame from the target video, detect the subtitle region in each image frame through a text detection model, and obtain the coordinates of each subtitle region. The text detection model is trained on images with completed subtitle region annotation.

[0117] The text extraction module is used to extract the text in each image frame according to the coordinates of each subtitle area, and obtain a string queue consisting of strings containing timestamps;

[0118] The identification and determination module is used to sequentially compare each string in the string queue with the corresponding preceding string to determine the subtitle character recognition result, and to display the subtitle character by character according to the subtitle character recognition result and the corresponding timestamp.

[0119] Optionally, the identification and determination module is further configured to:

[0120] Each string in the string queue is compared sequentially with the preceding string.

[0121] If the string is longer than the preceding string, the difference between the string and the preceding string is used as the character-by-character recognition result of the subtitle corresponding to the string;

[0122] If the string is shorter than the preceding string, then the string is used as the character-by-character recognition result of the subtitle corresponding to the string;

[0123] If the string is the same as the previous string, then the string is discarded;

[0124] Based on the timestamps corresponding to each string and the character-by-character recognition results of each subtitle, the character-by-character recognition results of each subtitle are displayed.

[0125] Optionally, the identification and determination module is further configured to:

[0126] Each string in the string queue is compared with the corresponding preceding string in turn. If there is a difference, the string and the corresponding timestamp are added to the recognition queue.

[0127] By comparing each string in the recognition queue with the corresponding previous string, the character-by-character recognition result of the subtitle is determined, and the subtitle is displayed character by character based on the character-by-character recognition result and the corresponding timestamp.

[0128] Optionally, the identification and determination module is further configured to:

[0129] Arrange the strings in the string queue according to their corresponding timestamps;

[0130] Each string is compared with the preceding string in turn. If the string is different from the preceding string, the string is added to the recognition queue.

[0131] If the string is the same as the previous string, then the string is discarded.

[0132] Optionally, the identification and determination module is further configured to:

[0133] Determine whether the preceding string is a subset of the string;

[0134] If so, it is determined that the string and the preceding string belong to the same subtitle, and the difference between the string and the preceding string is used as the subtitle character recognition result corresponding to the string;

[0135] If not, it is determined that the string and the preceding string do not belong to the same subtitle, and the string is used as the subtitle character recognition result corresponding to the string;

[0136] Based on the timestamps corresponding to each string and the character-by-character recognition results of each subtitle, the character-by-character recognition results of each subtitle are displayed.

[0137] Optionally, the region detection module is further configured to:

[0138] One frame is extracted from the target video at preset time intervals to obtain each image frame;

[0139] The text detection model is used to detect text in the preset regions of each image frame to obtain the coordinates of each subtitle region.

[0140] Optionally, the text extraction module is further used for:

[0141] Based on the coordinates of the subtitle area corresponding to each image frame, OCR text recognition is performed on each image frame to obtain the corresponding strings;

[0142] Based on the timestamp contained in the image frame corresponding to each string, the strings are arranged to obtain the string queue.

[0143] The subtitle display device provided in this application adopts the subtitle display method in the above embodiments, solving the technical problem of low accuracy in subtitle display. Compared with the prior art, the beneficial effects of the subtitle display device provided in this application are the same as those of the subtitle display method provided in the above embodiments, and other technical features in this subtitle display device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0144] Example 3

[0145] This application provides an electronic device, which includes: at least one processor; and a memory communicatively linked to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the subtitle display method described in Embodiment 1 above.

[0146] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0147] like Figure 5 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also linked to the bus.

[0148] Typically, the following systems can be linked to the I / O interface: input devices such as touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices such as liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices such as magnetic tapes, hard disks, etc.; and communication devices. Communication devices allow electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0149] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined above in the methods of embodiments of this disclosure.

[0150] The electronic device provided in this application employs the subtitle display method described in the above embodiments, thus solving the technical problem of low accuracy in displaying subtitles word by word. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the subtitle display method provided in Embodiment 1 above, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0151] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0153] Example 4

[0154] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the method of displaying subtitles word by word in the first embodiment described above.

[0155] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical link having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0156] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0157] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: extract image frames from a target video; detect subtitle regions in each image frame using a text detection model to obtain the coordinates of each subtitle region, wherein the text detection model is trained on images with completed subtitle region annotations; extract text from each image frame based on the coordinates of each subtitle region to obtain a string queue consisting of strings containing timestamps; sequentially compare each string in the string queue with its corresponding preceding string to determine the subtitle character-by-character recognition result; and display the subtitle character-by-character based on the subtitle character-by-character recognition result and the corresponding timestamp.

[0158] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be linked to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be linked to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0161] The computer-readable storage medium provided in this application stores computer-readable program instructions for executing the above-described subtitle display method, thus solving the technical problem of low accuracy in subtitle display. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the subtitle display method provided in the above-described embodiments, and will not be repeated here.

[0162] Example 5

[0163] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for displaying subtitles word by word.

[0164] The computer program product provided in this application solves the technical problem of low accuracy in displaying subtitles word by word. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the subtitle display method provided in the above embodiments, and will not be repeated here.

[0165] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A method for displaying subtitles character by character, characterized in that, The method for displaying subtitles character by character includes: Each image frame is extracted from the target video, and the subtitle region in each image frame is detected by a text detection model to obtain the coordinates of each subtitle region. The text detection model is trained on images with subtitle regions already annotated. Based on the coordinates of each subtitle area, the text in each image frame is extracted to obtain a string queue consisting of strings containing timestamps; Each string in the string queue is compared with the corresponding preceding string to determine the subtitle character recognition result, and the subtitle is displayed character by character based on the subtitle character recognition result and the corresponding timestamp; The steps of sequentially comparing each string in the string queue with the corresponding preceding string to determine the subtitle character recognition result, and displaying the subtitle character by character based on the subtitle character recognition result and the corresponding timestamp, include: Each string in the string queue is compared sequentially with the preceding string. If the string is longer than the preceding string, the difference between the string and the preceding string is used as the character-by-character recognition result of the subtitle corresponding to the string; If the string is shorter than the preceding string, then the string is used as the character-by-character recognition result of the subtitle corresponding to the string; If the string is the same as the previous string, then the string is discarded; Based on the timestamps corresponding to each string and the character-by-character recognition results of each subtitle, the character-by-character recognition results of each subtitle are displayed.

2. The method for displaying subtitles character by character as described in claim 1, characterized in that, The steps of sequentially comparing each string in the string queue with the corresponding preceding string to determine the subtitle character recognition result, and displaying the subtitle character by character based on the subtitle character recognition result and the corresponding timestamp, include: Each string in the string queue is compared with the corresponding preceding string in turn. If there is a difference, the string and the corresponding timestamp are added to the recognition queue. By comparing each string in the recognition queue with the corresponding previous string, the character-by-character recognition result of the subtitle is determined, and the subtitle is displayed character by character based on the character-by-character recognition result and the corresponding timestamp.

3. The method for displaying subtitles character by character as described in claim 2, characterized in that, The step of sequentially comparing each string in the string queue with its corresponding preceding string, and adding the string and its corresponding timestamp to the recognition queue if a difference exists, includes: Arrange the strings in the string queue according to their corresponding timestamps; Each string is compared with the preceding string in turn. If the string is different from the preceding string, the string is added to the recognition queue. If the string is the same as the previous string, then the string is discarded.

4. The method for displaying subtitles character by character as described in claim 2, characterized in that, The strings in the recognition queue are arranged in order according to their corresponding timestamps. The steps of comparing each string in the recognition queue with the corresponding previous string to determine the subtitle character-by-character recognition result, and displaying the subtitle character-by-character based on the subtitle character-by-character recognition result and the corresponding timestamp, include: Determine whether the preceding string is a subset of the string; If so, it is determined that the string and the preceding string belong to the same subtitle, and the difference between the string and the preceding string is used as the subtitle character recognition result corresponding to the string; If not, it is determined that the string and the preceding string do not belong to the same subtitle, and the string is used as the subtitle character recognition result corresponding to the string; Based on the timestamps corresponding to each string and the character-by-character recognition results of each subtitle, the character-by-character recognition results of each subtitle are displayed.

5. The method for displaying subtitles character by character as described in claim 1, characterized in that, The steps of extracting each image frame from the target video and detecting the subtitle region in each image frame using a text detection model to obtain the coordinates of each subtitle region include: One frame is extracted from the target video at preset time intervals to obtain each image frame; The text detection model is used to detect text in the preset regions of each image frame to obtain the coordinates of each subtitle region.

6. The method for displaying subtitles character by character as described in claim 1, characterized in that, The step of extracting text from each image frame based on the coordinates of each subtitle region to obtain a string queue consisting of strings including timestamps includes: Based on the coordinates of the subtitle area corresponding to each image frame, OCR text recognition is performed on each image frame to obtain the corresponding strings; Based on the timestamp contained in the image frame corresponding to each string, the strings are arranged to obtain the string queue.

7. A character-by-character subtitle display device, characterized in that, The subtitle display device includes: The region detection module is used to extract each image frame from the target video, detect the subtitle region in each image frame through a text detection model, and obtain the coordinates of each subtitle region. The text detection model is trained on images with completed subtitle region annotation. The text extraction module is used to extract the text in each image frame according to the coordinates of each subtitle area, and obtain a string queue consisting of strings containing timestamps; The identification and determination module is used to sequentially compare each string in the string queue with the corresponding preceding string to determine the subtitle character recognition result, and to display the subtitle character by character according to the subtitle character recognition result and the corresponding timestamp; The identification and determination module is further configured to sequentially compare each string in the string queue with its corresponding preceding string; if the string is longer than the preceding string, the difference between the string and the preceding string is used as the character-by-character recognition result of the subtitle corresponding to the string; if the string is shorter than the preceding string, the string is used as the character-by-character recognition result of the subtitle corresponding to the string; if the string is the same as the preceding string, the string is discarded; and display each character-by-character recognition result of the subtitle according to the timestamp corresponding to each string and the character-by-character recognition result of each subtitle.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory that is communicatively linked to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the steps of the subtitle display method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program for implementing a method for displaying subtitles word by word, the program for implementing the method for displaying subtitles word by word being executed by a processor to implement the steps of the method for displaying subtitles word by word as described in any one of claims 1 to 6.

Images