Video subtitle extraction method and device, electronic equipment and storage medium

Abstract

The application provides a video subtitle extraction method and device, electronic equipment and storage medium, belonging to the technical field of computer vision, the method comprises the following steps: determining a jump frame image based on the frame difference image of the front and rear frames of a target video, performing text recognition, and obtaining a text recognition result; performing frame state recognition on each frame to determine the starting frame and the ending frame of each subtitle; taking the starting frame of the current subtitle as a current frame image and taking the starting frame as a delayed picture frame image; performing weighted calculation on the rear frame image and the delayed picture frame image, updating the delayed picture frame image using the calculation result; taking the rear frame image as a new current frame image, returning to the step of performing weighted calculation on the rear frame image and the delayed picture frame image until the current subtitle is processed; performing text recognition on the delayed picture frame image, and determining a subtitle recognition result based on the text recognition result. The application can effectively adapt to the recognition of progressive display type subtitles and improve the subtitle recognition efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, electronic device, and storage medium for extracting video subtitles. Background Technology

[0002] With the rapid advancement of communication technology and the widespread adoption of smart devices in the information age, video, a media format rich in information and diverse in form, has permeated every corner of daily life. The recent surge in AI-generated content has also brought video-based content creation applications into the public eye. Subtitles, as crucial supplementary information, play a vital role in both video content understanding and post-production processing.

[0003] Subtitles are mainly categorized by their storage format into embedded and external types: embedded subtitles are displayed directly in the frame and are not stored independently; external subtitles are stored independently of the video data and subtitle data, and are rendered and displayed in real-time by the video playback device. Typically, subtitles embedded in the video frame require additional algorithms for extraction. Subtitle display methods can be categorized into one-time appearance and gradual display. One-time appearance displays the complete subtitle in the first frame of its duration, while gradual display displays it gradually from beginning to end, and may incorporate fade-in / fade-out transition effects between different subtitle segments.

[0004] Currently, video subtitle extraction usually involves directly applying OCR (Optical Character Recognition) technology to the original image. This method is easily affected by transition animation effects, and progressively displayed subtitles are prone to incomplete or misrecognized subtitles.

[0005] Therefore, for progressive display subtitles, how to accurately extract the subtitles has become an urgent technical problem to be solved. Summary of the Invention

[0006] This invention provides a video subtitle extraction method, apparatus, electronic device, and storage medium to address the shortcomings of incomplete extraction of progressively displayed subtitles in the prior art.

[0007] This invention provides a method for extracting video subtitles, comprising the following steps: For the target video, based on the frame difference image between consecutive frames, after determining the jump frame image, text recognition is performed on the jump frame image to obtain the text recognition result; Frame state recognition is performed on each frame in the target video to determine the start and end frames of each subtitle in the target video; The starting frame of the current subtitle is used as the current frame image, and the starting frame is used as the delayed frame image; A weighted calculation is performed on the subsequent frame image and the delayed frame image, and the calculation result is used to update the delayed frame image. The subsequent frame image is the next frame image adjacent to the current frame image. The next frame image is used as the new current frame image. The process returns to the step of performing a weighted calculation on the next frame image and the delayed frame image, and updating the delayed frame image with the calculation result, until the next frame image is the end frame image of the current subtitle. Perform text recognition on the delayed image frame to obtain the delayed image recognition result; Based on the text recognition result corresponding to the current subtitle and the delayed image recognition result, the subtitle recognition result of the current subtitle is determined.

[0008] According to a video subtitle extraction method provided by the present invention, after determining the jump frame image based on the frame difference image between consecutive frames, text recognition is performed on the jump frame image to obtain the text recognition result, including: For the current frame image, the region of interest is determined based on the frame category of the previous frame image. The previous frame image is the frame image adjacent to the current frame image, and the frame category includes caption frames and non-caption frames. Calculate the frame difference image between the current frame image and the previous frame image in the region of interest; If the proportion of the horizontal bounding box area of ​​the connected components in the frame difference image is greater than a preset threshold, then the current frame image is determined to be a jump frame image. Text recognition is performed on the region of interest of the current frame image to obtain the text recognition result of the current frame image. The proportion of the horizontal bounding box area of ​​the connected components is the ratio of the area of ​​the horizontal bounding box of the connected components of the changed pixel region of the frame difference image to the area of ​​the frame difference image.

[0009] According to a video subtitle extraction method provided by the present invention, determining the region of interest of the current frame image based on the frame category of the previous frame image includes: If the frame category of the previous frame image is a non-subtitle frame, then the region of interest is determined to be the entire image; If the frame category of the previous frame image is a subtitle frame, then the region of interest is determined to be a preset region in the image.

[0010] According to a video subtitle extraction method provided by the present invention, if the area ratio of the horizontal bounding box of the connected components in the frame difference image is greater than a preset threshold, the current frame image is determined to be a jump frame image, and text recognition is performed on the region of interest of the current frame image, including: When the frame category of the previous frame image is a non-subtitle frame, if the area ratio of the horizontal bounding box of the connected component in the frame difference image is greater than a first preset threshold, then the current frame image is determined to be a jump frame image, and text recognition is performed on the region of interest of the current frame image. When the frame category of the previous frame image is a subtitle frame, if the area ratio of the horizontal bounding box of the connected component in the frame difference image is greater than the second preset threshold, then the current frame image is determined to be a jump frame image, and text recognition is performed on the region of interest of the current frame image. Wherein, the first preset threshold is less than the second preset threshold.

[0011] According to a video subtitle extraction method provided by the present invention, the step of performing frame state recognition on each frame of the target video to determine the start frame and end frame of each subtitle in the target video includes: When the frame category of the current frame image is a non-subtitle frame, if the frame category of the previous frame image is a non-subtitle frame, then the current frame image is determined to be a blank frame. When the frame category of the current frame image is a non-subtitle frame, if the frame category of the previous frame image is a subtitle frame, then the current frame image is determined to be the end frame of a subtitle, and the time point corresponding to the current frame image is stored in the time list. When the frame category of the current frame image is a subtitle frame, if the frame category of the previous frame image is a non-subtitle frame, then the current frame image is determined to be the starting frame of a subtitle, the time point corresponding to the current frame image is stored in the time list, and the text recognition result of the current frame image is stored in the subtitle list corresponding to the subtitle. When the frame category of the current frame image is a subtitle frame, if the frame category of the previous frame image is a subtitle frame, then the similarity between the text recognition result of the current frame image and the text recognition result in the subtitle list is calculated. If the similarity is greater than the similarity threshold, then the current frame image is determined to be the middle frame of the subtitle, and the text recognition result of the current frame image is stored in the subtitle list corresponding to the subtitle. If the similarity is less than or equal to the similarity threshold, then the current frame image is determined to be the end frame of the subtitle, and the current frame image is the start frame of the next subtitle. The time point corresponding to the current frame image is stored in the time list.

[0012] According to a video subtitle extraction method provided by the present invention, determining the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result includes: The delayed image recognition result is added to the subtitle list of the current subtitle; Based on preset weights, the frequency of each element in the subtitle list is counted; Using a preset error correction model, the element with the highest frequency is corrected to obtain the subtitle recognition result of the current subtitle.

[0013] The present invention also provides a video subtitle extraction device, comprising the following modules: The frame difference calculation module is used to determine the jump frame image based on the frame difference image between consecutive frames of the target video, and then perform text recognition on the jump frame image to obtain the text recognition result. The frame state recognition module is used to perform frame state recognition on each frame in the target video and determine the start frame and end frame of each subtitle in the target video. An initialization module is used to take the starting frame of the current subtitle as the current frame image and the starting frame as the delayed frame image; The weighted calculation module is used to perform weighted calculation on the subsequent frame image and the delayed frame image, and update the delayed frame image using the calculation result. The subsequent frame image is the next frame image adjacent to the current frame image. The iterative update module is used to take the next frame image as the new current frame image and return to the step of performing a weighted calculation on the next frame image and the delayed frame image, and using the calculation result to update the delayed frame image, until the next frame image is the end frame image of the current subtitle; The text recognition module is used to perform text recognition on the delayed image frame to obtain the delayed image recognition result; The subtitle integration module is used to determine the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video subtitle extraction method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video subtitle extraction method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video subtitle extraction method as described above.

[0017] The video subtitle extraction method, apparatus, electronic device, and storage medium provided by this invention, for a target video, after determining the jump frame image based on the frame difference image of the preceding and following frames, performs text recognition on the jump frame image to obtain a text recognition result; performs frame state recognition on each frame in the target video to determine the start frame and end frame of each subtitle in the target video; takes the start frame of the current subtitle as the current frame image and takes the start frame as the delayed frame image; performs a weighted calculation on the following frame image and the delayed frame image, and uses the calculation result to update the delayed frame image, where the following frame image is the next frame image adjacent to the current frame image; takes the following frame image as the new current frame image, and returns to the step of performing a weighted calculation on the following frame image and the delayed frame image, and using the calculation result to update the delayed frame image, until the following frame image is the end frame image of the current subtitle; performs text recognition on the delayed frame image to obtain a delayed image recognition result; and determines the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result. This scheme performs weighted calculations on all frame images during the duration of the same subtitle. By setting the weights of subsequent frame images and delayed frame images, it can be applied to both one-time subtitles and progressive subtitles. For example, for one-time subtitles, the weights of subsequent frame images and delayed frame images can be set to equal weights. For progressive subtitles, the weight of subsequent frame images can be set to a larger weight, and the weight of delayed frame images can be set to a smaller weight, thus prioritizing the retention of later frame images and effectively adapting to the recognition of progressive subtitles. In addition, by performing weighted calculations on all frame images during the duration of the subtitle, the error disturbance caused by the unsatisfactory recognition results of individual subtitles in the frame image sequence can be reduced, ensuring the accuracy of subtitle recognition. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the video subtitle extraction method provided by the present invention.

[0020] Figure 2 This is the second flowchart illustrating the video subtitle extraction method provided by this invention.

[0021] Figure 3 This is a flowchart illustrating the frame state recognition method provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the video subtitle extraction device provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] It should be noted that in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0026] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] The following is combined Figures 1-5 This invention describes the video subtitle extraction method, apparatus, electronic device, and storage medium provided in embodiments of the present invention.

[0028] Figure 1 This is one of the flowcharts illustrating the video subtitle extraction method provided by the present invention, such as... Figure 1 As shown, the method includes the following: S110, For the target video, after determining the jump frame image based on the frame difference image between the previous and next frames, perform text recognition on the jump frame image to obtain the text recognition result; S120, perform frame state recognition on each frame in the target video to determine the start frame and end frame of each subtitle in the target video; S130, take the starting frame of the current subtitle as the current frame image, and take the starting frame as the delayed frame image; S140, perform a weighted calculation on the subsequent frame image and the delayed frame image, and update the delayed frame image using the calculation result. The subsequent frame image is the next frame image adjacent to the current frame image. S150, take the next frame image as the new current frame image, return to the step of performing a weighted calculation on the next frame image and the delayed frame image, and use the calculation result to update the delayed frame image, until the next frame image is the end frame image of the current subtitle; S160, perform text recognition on the delayed image frame to obtain the delayed image recognition result; S170, based on the text recognition result corresponding to the current subtitle and the delayed image recognition result, determine the subtitle recognition result of the current subtitle.

[0029] It should be noted that the execution subject of the task construction method provided in the embodiments of the present invention can be a server or computer device, such as a mobile phone, tablet computer, laptop computer, handheld computer, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0030] In S110, the target video is the video for which subtitle extraction is required. A jump frame image is a frame image that changes from the previous frame image; that is, a jump frame image is a frame image that is dissimilar to the previous frame image. For example, if the similarity between two adjacent frames is low, the next frame image is determined to be a jump frame image; otherwise, it is determined not to be a jump frame image. Another example is if the percentage of changed pixels in the frame difference image between two frames reaches a certain proportion; otherwise, it is determined not to be a jump frame image.

[0031] In S110, after identifying the jump frame images of the target video, text recognition is performed on all jump frame images to obtain the text recognition results corresponding to each jump frame image.

[0032] In S120, frame state recognition is performed on each frame in the target video, that is, the state of each frame image is identified. The frame state can include start frame, end frame, middle frame, blank frame, etc.

[0033] In S140, the next frame image is the next frame image adjacent to the current frame image in the video segment (containing multiple consecutive frame images) corresponding to the current subtitle. It should be understood that "adjacent" here can mean that two frames are immediately preceding and following frames in the video segment; that is, if the current frame image is the i-th frame image in the video segment, then the next frame image is the (i+1)-th frame image in the video segment. When the video is processed at a fixed frequency, "adjacent" here refers to the number of frames corresponding to the frequency of frame extraction in the video segment; that is, if the current frame image is the i-th frame image in the video segment, then the next frame image is the (i+k)-th frame image in the video segment, where k is determined based on the frame extraction frequency. It is understood that frame-by-frame processing can ensure the accuracy of subtitle recognition; therefore, it is preferable to use frame-by-frame processing.

[0034] In S140, the weights of the subsequent frame image and the delayed frame image `img_delay` can be set according to actual usage requirements. For example, for one-time subtitles, the subsequent frame image and the delayed frame image can be set to equal weights. For progressive subtitles, the weight of the subsequent frame image can be set higher, and the weight of the delayed frame image can be set lower, thus prioritizing the retention of later frames, which can effectively adapt to the recognition of progressive subtitles. It should be understood that for one-time subtitles, the subtitle remains unchanged within the same video segment; therefore, setting the weight of the subsequent frame image higher and the weight of the delayed frame image lower also applies to one-time subtitles.

[0035] In one embodiment, the weighted calculation of the subsequent frame image and the delayed frame image is performed based on the following formula: ; in, This represents the value of each pixel in the subsequent frame image. The value of each pixel in the delayed frame image. This is the memory coefficient. The larger the value, the greater the weight of the subsequent frame image and the smaller the memory effect of the preceding frame.

[0036] Here, by appropriately adjusting the memory coefficient, the delayed memory of the frame sequence within the duration of a subtitle can be controlled, the priority of the latest frame can be increased, and the interference of intermediate transition frames can be masked by iteratively averaging the frame sequence.

[0037] The video subtitle extraction method provided in this embodiment of the invention, for a target video, after determining the jump frame image based on the frame difference image of the preceding and following frames, performs text recognition on the jump frame image to obtain a text recognition result; performs frame state recognition on each frame in the target video to determine the start frame and end frame of each subtitle in the target video; takes the start frame of the current subtitle as the current frame image and takes the start frame as the delayed frame image; performs a weighted calculation on the following frame image and the delayed frame image, and uses the calculation result to update the delayed frame image, where the following frame image is the next frame image adjacent to the current frame image; takes the following frame image as the new current frame image, and returns to the step of performing a weighted calculation on the following frame image and the delayed frame image, and using the calculation result to update the delayed frame image, until the following frame image is the end frame image of the current subtitle; performs text recognition on the delayed frame image to obtain a delayed image recognition result; and determines the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result. This scheme performs weighted calculations on all frame images during the duration of the same subtitle. By setting the weights of subsequent frame images and delayed frame images, it can be applied to both one-time subtitles and progressive subtitles. For example, for one-time subtitles, the weights of subsequent frame images and delayed frame images can be set to equal weights. For progressive subtitles, the weight of subsequent frame images can be set to a larger weight, and the weight of delayed frame images can be set to a smaller weight, thus prioritizing the retention of later frame images and effectively adapting to the recognition of progressive subtitles. In addition, by performing weighted calculations on all frame images during the duration of the subtitle, the error disturbance caused by the unsatisfactory recognition results of individual subtitles in the frame image sequence can be reduced, ensuring the accuracy of subtitle recognition.

[0038] To improve the efficiency of video subtitle extraction, in an optional embodiment, after determining the jump frame image based on the frame difference image between consecutive frames, text recognition is performed on the jump frame image to obtain the text recognition result, including: S210, for the current frame image, determine the region of interest based on the frame category of the previous frame image, wherein the previous frame image is the frame image adjacent to the current frame image, and the frame category includes caption frames and non-caption frames; S220, Calculate the frame difference image between the current frame image and the previous frame image in the region of interest; S230, if the proportion of the horizontal bounding box area of ​​the connected components in the frame difference image is greater than a preset threshold, then the current frame image is determined to be a jump frame image, and text recognition is performed on the region of interest of the current frame image to obtain the text recognition result of the current frame image. The proportion of the horizontal bounding box area of ​​the connected components is the ratio of the area of ​​the horizontal bounding box of the connected components of the changed pixel region of the frame difference image to the area of ​​the frame difference image.

[0039] In S210, different regions of interest (ROIs) are extracted for subtitle frames and non-subtitle frames. Non-subtitle frames are image frames determined not to contain subtitles, and their ROIs are set to be larger so that the presence of text content in the entire screen needs to be detected in subsequent processing. Subtitle frames are image frames determined to contain subtitles, and their ROIs are set to be smaller so that only the smaller ROIs need to be processed in subsequent processing, reducing unnecessary text detection and improving the efficiency of text detection and recognition. For example, considering that common screen subtitles do not appear in the center of the screen to obscure the main content, the ROI of the subtitle frame can be set to the perimeter of the screen.

[0040] It should be understood that "adjacent" here can refer to two consecutive frames within a video segment; that is, if the current frame is the i-th frame in the video segment, then the preceding frame is the (i-1)-th frame. When the video is processed by frame extraction at a fixed frequency, "adjacent" here refers to the number of frames between the two frames corresponding to the extraction frequency within the video segment; that is, if the current frame is the i-th frame in the video segment, then the following frame is the ik-th frame, where k is determined based on the extraction frequency. Preferably, frame-by-frame processing is used.

[0041] In S220, the frame difference image is an image generated by comparing the region of interest of the current frame image with the region of interest of the previous frame image.

[0042] Specifically, the current frame image and the previous frame image are converted into grayscale images; the difference in grayscale pixels between the current frame image and the previous frame image in the region of interest is calculated to obtain a difference image; a thresholding process is applied to the difference image, setting the parts with pixel values ​​below the threshold (i.e., the parts of the current frame image that are similar to the previous frame image) to black (pixel values ​​set to 0), and the parts with pixel values ​​above the threshold (i.e., the parts of the current frame image that are different from the previous frame image) to white (pixel values ​​set to 255), thus generating a binary image, namely the frame difference image.

[0043] In S230, the horizontal bounding box of a connected region refers to the smallest rectangular box calculated for a connected region (i.e., a set of pixels with the same attributes that are adjacent to each other) in a binary image, which completely covers the connected region horizontally. Here, the horizontal bounding box of the connected region of the changing pixel region is the horizontal bounding box of the connected region with a pixel value of 255. If the area of ​​the horizontal bounding box of the connected region of the changing pixel region is too small, it means that the image content of the current frame image and the previous frame image in the region of interest has not changed much, and there is no need to perform text recognition on the region of interest of the current frame image. The text recognition result of the previous frame image can be directly used as the text recognition result of the current frame image. When the area of ​​the horizontal bounding box of the connected region of the changing pixel region is greater than a preset threshold, it means that the image content of the current frame image and the previous frame image in the region of interest has changed, and text recognition on the region of interest of the current frame image is required.

[0044] Frame-by-frame processing avoids skipping keyword subtitle frames due to excessively large manually set frame intervals, resulting in more accurate subtitle timing. However, it also suffers from low processing efficiency. The video subtitle extraction method provided in this invention extracts different regions of interest for subtitle frames and non-subtitle frames, reducing the amount of screen content that needs to be detected and improving processing efficiency. By calculating the frame difference image between consecutive frames, it identifies images with significant differences for detection, quickly skipping non-subtitle frames and subtitle frames where the subtitle area content does not change much, reducing unnecessary text detection and OCR, further improving subtitle extraction efficiency.

[0045] Further optionally, determining the region of interest (ROI) of the current frame image based on the frame category of the previous frame image includes: If the frame category of the previous frame image is a non-subtitle frame, then the region of interest is determined to be the entire image; If the frame category of the previous frame image is a subtitle frame, then the region of interest is determined to be a preset region in the image.

[0046] Specifically, if the frame type of the previous frame image is a non-subtitle frame, the region of interest is determined to be the entire image, and the difference between the previous and next frames of the entire image is calculated; if the frame type of the previous frame image is a subtitle frame, the region of interest is determined to be a preset area in the image. For example, considering that subtitles are usually located in the lower part of the screen, when the previous frame image is a subtitle frame, the region of interest is the lower 1 / 4 of the screen.

[0047] The specific range of the preset area can be set according to actual usage needs, and there are no restrictions here.

[0048] The video subtitle extraction method provided in this invention sets the region of interest to the entire image when the previous frame is not a subtitle frame. This allows the entire image to be calculated when calculating the frame difference image in the future, ensuring accurate identification of whether there is a difference between the current frame image and the previous frame image. When the previous frame is a subtitle frame, the region of interest is set to a preset area in the image. This allows only the subtitle portion to be calculated when calculating the frame difference image in the future, effectively reducing the amount of computation.

[0049] Further optionally, if the proportion of the horizontal bounding box area of ​​the connected components in the frame difference image is greater than a preset threshold, then the current frame image is determined to be a transition frame image, and text recognition is performed on the region of interest of the current frame image, including: When the frame category of the previous frame image is a non-subtitle frame, if the area ratio of the horizontal bounding box of the connected component in the frame difference image is greater than a first preset threshold, then the current frame image is determined to be a jump frame image, and text recognition is performed on the region of interest of the current frame image. When the frame category of the previous frame image is a subtitle frame, if the area ratio of the horizontal bounding box of the connected component in the frame difference image is greater than the second preset threshold, then the current frame image is determined to be a jump frame image, and text recognition is performed on the region of interest of the current frame image. Wherein, the first preset threshold is less than the second preset threshold.

[0050] The video subtitle extraction method provided in this embodiment of the invention has the following characteristics: When the current frame image is a non-subtitle frame, the region of interest is the entire image, and the subtitle region only occupies a small portion of the region of interest. Therefore, a smaller first preset threshold is set to avoid skipping images with differences between consecutive frames. When the current frame image is a subtitle frame, the region of interest is a preset region, and the subtitle region occupies most of the region of interest. Therefore, a larger second preset threshold is set to ensure that subtitle frames with little change in subtitle content are quickly skipped, reducing unnecessary text detection.

[0051] Based on any of the above embodiments, considering the instability of OCR results when performing OCR recognition across the entire video frame, which may produce meaningless garbled results, a subtitle filtering algorithm is used to eliminate interference from non-subtitle areas before the text recognition results are output. Specifically: For the bounding box parameters in the text detection results ( (), where x is the x-coordinate of the top-left vertex, y is the y-coordinate of the top-left vertex, w is the width of the bounding box, h is the height of the bounding box, and the width and height of the region of interest in the current frame difference image are... If the region of interest is the entire image, filter out text boxes located near the center of the image, i.e., exclude them. For text detection boxes outside the specified range, the parameter values ​​are typically set to... , If the region of interest in the current frame difference image is the preset region (subtitle region), filter out the text boxes outside the center region, i.e., retain... The text detection box within the range typically has the following parameter values: , , , ; Based on the results of text recognition, excessively short garbled strings are excluded, as are Chinese character strings with a length of less than 2 characters, English character strings with a length of less than 3 characters, and mixed Chinese and English character strings with a length of less than 5 characters, in order to avoid interference from non-text areas being recognized as subtitles.

[0052] Finally, the text recognition results after filtering out interference are sorted according to the position of the text detection bounding box and the orientation of the text. For example, horizontal subtitles follow the order from top to bottom and from left to right within each line, while vertical text follows the order from right to left and from top to bottom within each column.

[0053] Based on any of the above embodiments, the step of performing frame state recognition on each frame of the target video to determine the start and end frames of each subtitle in the target video includes: When the frame category of the current frame image is a non-subtitle frame, if the frame category of the previous frame image is a non-subtitle frame, then the current frame image is determined to be a blank frame. When the frame category of the current frame image is a non-subtitle frame, if the frame category of the previous frame image is a subtitle frame, then the current frame image is determined to be the end frame of a subtitle, and the time point corresponding to the current frame image is stored in the time list. When the frame category of the current frame image is a subtitle frame, if the frame category of the previous frame image is a non-subtitle frame, then the current frame image is determined to be the starting frame of a subtitle, the time point corresponding to the current frame image is stored in the time list, and the text recognition result of the current frame image is stored in the subtitle list corresponding to the subtitle. When the frame category of the current frame image is a subtitle frame, if the frame category of the previous frame image is a subtitle frame, then the similarity between the text recognition result of the current frame image and the text recognition result in the subtitle list is calculated. If the similarity is greater than the similarity threshold, then the current frame image is determined to be the middle frame of the subtitle, and the text recognition result of the current frame image is stored in the subtitle list corresponding to the subtitle. If the similarity is less than or equal to the similarity threshold, then the current frame image is determined to be the end frame of the subtitle, and the current frame image is the start frame of the next subtitle. The time point corresponding to the current frame image is stored in the time list.

[0054] The subtitle list is denoted as opt_txts: ["txt1", "txt2", ...], where "txt1", "txt2", ... are the recognition results in the middle of the subtitle; the time list is denoted as clip_time: [[t1, t2], [t3, t4], ...], where each element [ti, ti+1] is the start and end time of a subtitle.

[0055] Figure 3 This is a flowchart illustrating the frame state recognition method provided by the present invention, as shown below. Figure 3 As shown, the specific steps for frame state recognition are as follows: (a) Determine whether there is a reliable text recognition result after the current frame image is filtered by the subtitle filtering algorithm. If there is no reliable text recognition result, proceed to step (b). If there is a reliable text recognition result, proceed to step (c). (b) Determine whether the previous frame image is a subtitle frame. If the previous frame image is not a subtitle frame, the current frame image is a blank frame, and continue to detect the next frame image. If the previous frame image is a subtitle frame, the current frame image is the end frame of the subtitle. Store the time point corresponding to the current frame image in clip_time, and the opt_txts recognition of the subtitle is completed. (c) Determine whether the previous frame image is a subtitle frame. If the previous frame image is a non-subtitle frame (blank frame), then the current frame image is the starting frame of the new subtitle. The time point corresponding to the current frame image is used as the starting point of the new subtitle and recorded in clip_time. At the same time, the text recognition result of the current frame image is stored in opt_txts of the new subtitle. If the previous frame image is a subtitle frame, then the text recognition result of the current frame image and the temporary result in opt_txts are used to calculate the subtitle similarity, and then proceed to step (d). (d) If the text recognition result of the current frame image is similar to the previous text recognition result in opt_txts, then the current frame image is the middle frame of the subtitle, and the text recognition result of the current frame image is stored in opt_txts; if the text recognition result of the current frame image is not similar to the previous text recognition result in opt_txts, then the current frame image is the end frame of the previous subtitle and also the start frame of the next new subtitle, with no blank frame transition in between. The time node corresponding to the current frame image is stored in clip_time as the end time point of the previous subtitle, and the time node corresponding to the current frame image is stored in clip_time as the start time point of the next new subtitle.

[0056] The video subtitle extraction method provided in this embodiment of the invention determines the state of the current frame image based on the text recognition results and the state and text recognition results of the previous frame, initially divides the start and end positions of each subtitle, and provides data basis for the subsequent integration of subtitle results.

[0057] Due to factors such as the accuracy of the text recognition module, the subtitle display method, camera transitions and special effects, and interference from the background, subtitle text recognition results may exhibit problems such as missed detections, individual character errors, disordered sentence component order, and incomplete progressive subtitles. The common method of identifying identical characters as the same subtitle can lead to the splitting of a single subtitle into multiple subtitles in practical use due to these errors. To address these issues, this invention provides a subtitle similarity calculation method to avoid misjudging the same subtitle due to text recognition errors. Specifically, the similarity between two strings (text recognition results) is calculated based on the following formula: ; in, The text recognition result (string) of the current frame image. This is the text recognition result (string) of the previous frame image. The score is given for the longest common subsequence. To edit the distance score, This is a coefficient (which can be set according to actual usage needs, and is usually set to 0.5).

[0058] The calculation formula is: ; in, The length of the string is given. The Longest Common Subsequence (LCS) is defined as follows: if a sequence S is a subsequence of two or more known sequences and is the longest of all these subsequences, then S is called the Longest Common Subsequence of these known sequences. Because LCS does not require the subsequences to be consecutive, it can handle situations where two frames have roughly the same recognition results but some characters are missed or incorrectly detected in between.

[0059] The calculation formula is: ; Among them, edit distance This tool is used to calculate the minimum number of operations required to change two strings (A and B) from A to B, where each insertion, replacement, and deletion counts as one operation. The smaller the edit distance, the greater the similarity between the two strings. The score is determined by a formula: the smaller the difference between the two strings, the higher the score.

[0060] Finally, after calculating the similarity between the text recognition result of the current frame image and the previous text recognition result in opt_txts, and setting a threshold, the results are compared. (This can be adjusted according to the actual situation). If the value is greater than this threshold, it is considered similar; otherwise, it is not similar.

[0061] When determining the current frame as the end frame, ideally, the recognition results for all frames within the duration of a subtitle should be accurate and consistent. However, in reality, some deviations may occur. Common text error types include: homophones, confused words, reversed order, and grammatical errors. In the subtitle recognition task of this invention, the more common errors are: similar-looking characters, incomplete display leading to missed recognition of individual characters, and extra characters caused by punctuation or background interference. Therefore, it is necessary to correct the text recognition results.

[0062] To correct errors in the text recognition results and improve the accuracy of video subtitle extraction, in an optional embodiment, determining the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result includes: The delayed image recognition result is added to the subtitle list of the current subtitle; Based on preset weights, the frequency of each element in the subtitle list is counted; Using a preset error correction model, the element with the highest frequency is corrected to obtain the subtitle recognition result of the current subtitle.

[0063] Specifically, the text recognition results delay_txts of the delayed frame image img_delay are appended to the current subtitle list opt_txts, and the weights are adjusted appropriately according to the video type (for example, in the case of progressive display videos, where the string length in opt_txts gradually increases, the weight of delay_txts can be increased). The string objects in opt_txts are deduplicated and their frequency is recorded. The string with the highest frequency is taken as the input of the preset error correction model to obtain the subtitle recognition result of the current subtitle.

[0064] Optionally, the WBS (Word Beam Search) error correction algorithm can be used for English subtitles to constrain words to be in a regular dictionary and avoid individual characters being misplaced.

[0065] Optionally, for Chinese subtitles, a Soft-Masked BERT error correction model is used. This model divides text correction into an error detection network and a correction network. The detection network is a Bi-GRU network that predicts the probability of an error occurring at each character position. This probability is then used to embed a soft mask of the character at that position. The correction network uses BERT for error correction. This method, during end-to-end joint training, forces the model to learn the correct context for error correction with the help of the detection network.

[0066] After obtaining the corrected subtitle recognition result, the string of the subtitle recognition result is aligned and integrated with the time point corresponding to the subtitle in clip_time. As the processing loops to the last frame, the subtitle extraction result of the entire video is also completely produced.

[0067] The video subtitle extraction method provided in this invention combines the intermediate results of the frame sequence within the same subtitle duration with the recognition results of delayed frames for error correction, avoiding the disturbance deviation caused by the error of a single frame, and making the subtitle text extraction results more accurate and reliable.

[0068] Figure 2 This is the second flowchart illustrating the video subtitle extraction method provided by this invention, as shown below. Figure 2 As shown, for ease of understanding, the following is combined with... Figure 2 The preferred video subtitle extraction method of the present invention will be described.

[0069] For the video to be identified, it is processed frame by frame. For the case where the previous frame is a subtitle frame and a non-subtitle frame, different regions of interest are extracted respectively, and the frame difference image between the current frame image and the subtitle frame is calculated in the region of interest. When the area of ​​the horizontal bounding box of the connected components in the frame difference image is greater than a preset threshold, text detection and recognition are performed on the current frame image; otherwise, frame state recognition is performed directly on the current frame image. During text detection and recognition, the interference of non-subtitle area text is eliminated through a subtitle filtering algorithm. Frame state recognition mainly determines the state of the current frame based on the text recognition results and the state and text recognition results of the previous frames, and initially divides the start and end positions of each subtitle. After determining the starting frame of each subtitle using a frame state recognition method, a delayed frame object `img_delay` is initialized. The starting frame `img_ori` is directly copied completely as `img_delay`. For all subsequent frames `img` that are determined to be the same subtitle, their images are iteratively updated with weighted averages compared to the previous delayed frames. By appropriately adjusting the memory coefficient, the delay memory of the frame sequence within the duration of a subtitle is controlled, increasing the priority of the latest frame. Simultaneously, the interference of intermediate transition frames is masked by iteratively averaging the images of the frame sequence. Once the subtitle is determined to be the end frame, text detection and recognition are performed on the delayed frame `img_delay`. The text recognition result `delay_txts` is then appended to the subtitle list `opt_txts` for subtitle parsing and error correction, and the `img_delay` image object is cleared. When the next subtitle starts, the delayed frame `img_delay` is reinitialized.

[0070] Finally, the corrected string is aligned and integrated with the time list clip_time. As the processing loops to the last frame, the subtitle extraction results for the entire video are also produced.

[0071] The video subtitle extraction device provided in the embodiments of the present invention will be described below. The video subtitle extraction device described below can be referred to in correspondence with the video subtitle extraction method described above.

[0072] Figure 4 This is a schematic diagram of the video subtitle extraction device provided by the present invention, as shown below. Figure 4 As shown, the video subtitle extraction device may include, but is not limited to; The frame difference calculation module 410 is used to determine the jump frame image based on the frame difference image of the previous and next frames for the target video, and then perform text recognition on the jump frame image to obtain the text recognition result. The frame state recognition module 420 is used to perform frame state recognition on each frame in the target video and determine the start frame and end frame of each subtitle in the target video. The initialization module 430 is used to take the starting frame of the current subtitle as the current frame image and the starting frame as the delayed frame image; The weighted calculation module 440 is used to perform weighted calculation on the subsequent frame image and the delayed frame image, and update the delayed frame image using the calculation result. The subsequent frame image is the next frame image adjacent to the current frame image. The iterative update module 450 is used to take the next frame image as the new current frame image and return to the step of performing a weighted calculation on the next frame image and the delayed frame image, and using the calculation result to update the delayed frame image, until the next frame image is the end frame image of the current subtitle. The text recognition module 460 is used to perform text recognition on the delayed image frame to obtain the delayed image recognition result; The subtitle integration module 470 is used to determine the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result.

[0073] It should be noted that the video subtitle extraction device provided in this embodiment of the invention can execute the video subtitle extraction method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0074] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can call logical instructions in the memory 530 to execute a video subtitle extraction method. This method includes: for a target video, determining a jump frame image based on the frame difference image between consecutive frames, performing text recognition on the jump frame image to obtain a text recognition result; performing frame state recognition on each frame in the target video to determine the start frame and end frame of each subtitle in the target video; using the start frame of the current subtitle as the current frame image and the start frame as a delayed frame image; performing a weighted calculation on the subsequent frame image and the delayed frame image, updating the delayed frame image using the calculation result, where the subsequent frame image is the next frame image adjacent to the current frame image; using the subsequent frame image as the new current frame image, returning to the step of performing a weighted calculation on the subsequent frame image and the delayed frame image, updating the delayed frame image using the calculation result, until the subsequent frame image is the end frame image of the current subtitle; performing text recognition on the delayed frame image to obtain a delayed recognition result; and determining the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed recognition result.

[0075] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the video subtitle extraction method provided by the above methods. The method includes: for a target video, determining a jump frame image based on the frame difference image of consecutive frames, performing text recognition on the jump frame image to obtain a text recognition result; performing frame state recognition on each frame in the target video to determine the start frame and end frame of each subtitle in the target video; taking the start frame of the current subtitle as the current frame image, and taking the start frame as the delay image. The process involves: 1. Performing a weighted calculation on the following frame image and the delayed frame image, and updating the delayed frame image using the calculation result. The following frame image is the next frame image adjacent to the current frame image. 2. Using the following frame image as the new current frame image, returning to the step of performing a weighted calculation on the following frame image and the delayed frame image, and updating the delayed frame image using the calculation result, until the following frame image becomes the end frame image of the current subtitle. 3. Performing text recognition on the delayed frame image to obtain a delayed image recognition result. 4. Determining the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result.

[0077] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the video subtitle extraction method provided by the above methods. This method includes: for a target video, determining a jump frame image based on the frame difference image between consecutive frames, performing text recognition on the jump frame image to obtain a text recognition result; performing frame state recognition on each frame in the target video to determine the start frame and end frame of each subtitle in the target video; using the start frame of the current subtitle as the current frame image, and using the start frame as the delayed frame image; comparing the subsequent frame image with the delayed... The delayed frame image is weighted and calculated, and the result is used to update the delayed frame image. The next frame image is the next frame image adjacent to the current frame image. The next frame image is used as the new current frame image, and the process returns to the step of weighting the next frame image and the delayed frame image and updating the delayed frame image with the result, until the next frame image is the end frame image of the current subtitle. Text recognition is performed on the delayed frame image to obtain the delayed image recognition result. Based on the text recognition result corresponding to the current subtitle and the delayed image recognition result, the subtitle recognition result of the current subtitle is determined.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0079] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting video subtitles, characterized in that, Applied to progressive captions, including: For the target video, based on the frame difference image between consecutive frames, after determining the jump frame image, text recognition is performed on the jump frame image to obtain the text recognition result; Frame state recognition is performed on each frame in the target video to determine the start and end frames of each subtitle in the target video; The starting frame of the current subtitle is used as the current frame image, and the starting frame is used as the delayed frame image; The weighted calculation is performed on the subsequent frame image and the delayed frame image, and the calculation result is used to update the delayed frame image. The subsequent frame image is the next frame image adjacent to the current frame image, and the weight of the subsequent frame image is greater than the weight of the delayed frame image. The next frame image is used as the new current frame image. The process returns to the step of performing a weighted calculation on the next frame image and the delayed frame image, and updating the delayed frame image with the calculation result, until the next frame image is the end frame image of the current subtitle. Perform text recognition on the delayed image frame to obtain the delayed image recognition result; Based on the text recognition result corresponding to the current subtitle and the delayed image recognition result, the subtitle recognition result of the current subtitle is determined.

2. The video subtitle extraction method according to claim 1, characterized in that, After determining the jump frame image based on the frame difference image between consecutive frames, text recognition is performed on the jump frame image to obtain the text recognition results, including: For the current frame image, the region of interest is determined based on the frame category of the previous frame image. The previous frame image is the frame image adjacent to the current frame image, and the frame category includes caption frames and non-caption frames. Calculate the frame difference image between the current frame image and the previous frame image in the region of interest; If the proportion of the horizontal bounding box area of ​​the connected components in the frame difference image is greater than a preset threshold, then the current frame image is determined to be a jump frame image. Text recognition is performed on the region of interest of the current frame image to obtain the text recognition result of the current frame image. The proportion of the horizontal bounding box area of ​​the connected components is the ratio of the area of ​​the horizontal bounding box of the connected components of the changed pixel region of the frame difference image to the area of ​​the frame difference image.

3. The video subtitle extraction method according to claim 2, characterized in that, Determining the region of interest (ROI) of the current frame image based on the frame category of the previous frame image includes: If the frame category of the previous frame image is a non-subtitle frame, then the region of interest is determined to be the entire image; If the frame category of the previous frame image is a subtitle frame, then the region of interest is determined to be a preset region in the image.

4. The video subtitle extraction method according to claim 3, characterized in that, If the area ratio of the horizontal bounding box of the connected components in the frame difference image is greater than a preset threshold, then the current frame image is determined to be a transition frame image, and text recognition is performed on the region of interest of the current frame image, including: When the frame category of the previous frame image is a non-subtitle frame, if the area ratio of the horizontal bounding box of the connected component in the frame difference image is greater than a first preset threshold, then the current frame image is determined to be a jump frame image, and text recognition is performed on the region of interest of the current frame image. When the frame category of the previous frame image is a subtitle frame, if the area ratio of the horizontal bounding box of the connected component in the frame difference image is greater than the second preset threshold, then the current frame is determined to be a jump frame image, and text recognition is performed on the region of interest of the current frame image. Wherein, the first preset threshold is less than the second preset threshold.

5. The video subtitle extraction method according to any one of claims 2-4, characterized in that, The step of performing frame state recognition on each frame of the target video to determine the start and end frames of each subtitle in the target video includes: When the frame category of the current frame image is a non-subtitle frame, if the frame category of the previous frame image is a non-subtitle frame, then the current frame image is determined to be a blank frame. When the frame category of the current frame image is a non-subtitle frame, if the frame category of the previous frame image is a subtitle frame, then the current frame image is determined to be the end frame of a subtitle, and the time point corresponding to the current frame image is stored in the time list. When the frame category of the current frame image is a subtitle frame, if the frame category of the previous frame image is a non-subtitle frame, then the current frame image is determined to be the starting frame of a subtitle, the time point corresponding to the current frame image is stored in the time list, and the text recognition result of the current frame image is stored in the subtitle list corresponding to the subtitle. When the frame category of the current frame image is a subtitle frame, if the frame category of the previous frame image is a subtitle frame, then the similarity between the text recognition result of the current frame image and the text recognition result in the subtitle list is calculated. If the similarity is greater than the similarity threshold, then the current frame image is determined to be the middle frame of the subtitle, and the text recognition result of the current frame image is stored in the subtitle list corresponding to the subtitle. If the similarity is less than or equal to the similarity threshold, then the current frame image is determined to be the end frame of the subtitle, and the current frame image is the start frame of the next subtitle. The time point corresponding to the current frame image is stored in the time list.

6. The video subtitle extraction method according to claim 5, characterized in that, The step of determining the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result includes: The delayed image recognition result is added to the subtitle list of the current subtitle; Based on preset weights, the frequency of each element in the subtitle list is counted; Using a preset error correction model, the element with the highest frequency is corrected to obtain the subtitle recognition result of the current subtitle.

7. A video subtitle extraction device, characterized in that, Applied to progressive captions, including: The frame difference calculation module is used to determine the jump frame image based on the frame difference image between consecutive frames of the target video, and then perform text recognition on the jump frame image to obtain the text recognition result. The frame state recognition module is used to perform frame state recognition on each frame in the target video and determine the start frame and end frame of each subtitle in the target video. An initialization module is used to take the starting frame of the current subtitle as the current frame image and the starting frame as the delayed frame image; The weighted calculation module is used to perform weighted calculation on the subsequent frame image and the delayed frame image, and update the delayed frame image using the calculation result. The subsequent frame image is the next frame image adjacent to the current frame image, and the weight of the subsequent frame image is greater than the weight of the delayed frame image. The iterative update module is used to take the next frame image as the new current frame image and return to the step of performing a weighted calculation on the next frame image and the delayed frame image, and using the calculation result to update the delayed frame image, until the next frame image is the end frame image of the current subtitle; The text recognition module is used to perform text recognition on the delayed image frame to obtain the delayed image recognition result; The subtitle integration module is used to determine the subtitle recognition result of the current subtitle based on the text recognition result corresponding to the current subtitle and the delayed image recognition result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video subtitle extraction method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video subtitle extraction method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video subtitle extraction method as described in any one of claims 1 to 6.

Images