Hard coded subtitle extraction method, apparatus and device, and medium

Abstract

The invention relates to the field of subtitle extraction, in particular to a hard coded subtitle extraction method and device, equipment and a medium, and the method comprises the steps: obtaining a video frame sequence; subtitle clipping is carried out on the video frame sequence, a subtitle area can be accurately focused, and a subtitle clipping video frame sequence only containing subtitle related content is obtained; grouping a plurality of caption video frames in the caption clipping video frame sequence, and grouping the continuous caption video frames which possibly belong to the same caption content into one group to form a plurality of caption groups; the subtitle video frames in the subtitle group are subjected to text recognition, so that subtitles in an image form can be converted into sub-text contents in a text form, and the processing efficiency and accuracy are improved; the subtitle file is determined according to the sub-text content of each subtitle group, the hard coded subtitles are completely extracted from the video, and efficient and accurate subtitle extraction is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of subtitle extraction, and in particular, to a hard-coded subtitle extraction method, device, equipment and medium. BACKGROUND

[0002] Currently, many video subtitles are directly hard-coded in the video screen, without a separate editable subtitle file. Hard-coded subtitles refer to directly burning the subtitle text in the video screen, becoming part of the video image. Hard-coded subtitles cannot be closed and switched, and are always displayed during video playback. In addition, since the subtitles and the screen are always combined together, it is difficult to extract or modify them later.

[0003] The technical solution of the hard-coded subtitle extraction method generally uses the following methods:

[0004] First, the user inputs the hard-coded subtitle area of the video through interaction, or the AI model intelligently identifies and determines the hard-coded subtitle area position. After determining the hard-coded subtitle area, some solutions extract the local screen of the video frame by frame through the subtitle area, and then perform text detection and OCR recognition in the local screen to recognize the subtitle text. After recognizing the subtitle text of each frame, the adjacent frame subtitle texts are merged through a subtitle merging strategy to form a subtitle group. After all the subtitle merging is completed, a subtitle file with a time axis is finally formed.

[0005] Second, the traditional visual threshold method is used to determine whether the local screen has a large change, and the subtitle switching time is located through this method. After determining the subtitle switching time, the subtitles are grouped, and then the local screen of the video frame is randomly selected in the time range of each subtitle group for text detection and OCR recognition to recognize the subtitle text. Then, the appropriate subtitle text is selected as the subtitle text of the subtitle group, and finally a subtitle file with a time axis is formed.

[0006] However, the above methods are prone to text missing detection, and the accuracy of subtitle extraction is low. SUMMARY

[0007] The present application aims to provide a hard-coded subtitle extraction method, device, equipment and medium, which can improve the accuracy of subtitle extraction.

[0008] In a first aspect, a method for extracting hard-coded subtitles is provided, including: obtaining a video frame sequence; performing subtitle cropping on the video frame sequence to obtain a subtitle-cropped video frame sequence, the subtitle-cropped video frame sequence including a plurality of subtitle video frames, the subtitle video frames including at least a subtitle region; grouping the plurality of subtitle video frames in the subtitle-cropped video frame sequence to obtain a plurality of subtitle groups, the subtitle groups including a plurality of continuous subtitle video frames; performing text recognition on the subtitle video frames in the subtitle groups to obtain subtext content; and determining a subtitle file according to the subtext content of each subtitle group.

[0009] In a preferred example, the application can be further configured to: perform subtitle cropping on the video frame sequence to obtain a subtitle-cropped video frame sequence, including: sampling the video frame sequence to obtain a video sample frame sequence; performing subtitle region positioning on each video sample frame in the video sample frame sequence using a subtitle region positioning model to obtain initial subtitle bounding box coordinate positions corresponding to each video sample frame; determining subtitle bounding box coordinate positions according to the initial subtitle bounding box coordinate positions corresponding to each video sample frame in the video sample frame sequence; and performing subtitle cropping on the video frame sequence based on the subtitle bounding box coordinate positions to obtain the subtitle-cropped video frame sequence.

[0010] In a preferred example, the application can be further configured to: group the plurality of subtitle video frames in the subtitle-cropped video frame sequence to obtain a plurality of subtitle groups, including: determining subtitle switching frames in the plurality of subtitle video frames in the subtitle-cropped video frame sequence using a subtitle transition detection model; and grouping the plurality of subtitle video frames according to the subtitle switching frames to determine the plurality of subtitle groups.

[0011] In a preferred example, the application can be further configured to: group the plurality of subtitle video frames according to the subtitle switching frames to determine the plurality of subtitle groups, including: determining frame indexes of the subtitle switching frames; determining a starting frame index and an ending frame index corresponding to each subtitle group according to the frame indexes of the subtitle switching frames; and determining the plurality of subtitle groups according to the starting frame index and the ending frame index corresponding to each subtitle group.

[0012] In a preferred example, the application can be further configured to: the subtitle transition detection model comprises a visual feature encoder, a pyramid temporal convolution module, a subtitle region attention module, and a prediction module; accordingly, determining the subtitle switching frame in the plurality of subtitle video frames in the subtitle cropped video frame sequence by using the subtitle transition detection model comprises: encoding the video frame sequence based on the visual feature encoder to obtain encoded features; inputting the encoded features into the pyramid temporal convolution module to obtain multi-scale temporal features; inputting the multi-scale temporal features and the video frame sequence into the subtitle region attention module to obtain weighted aggregated temporal features; inputting the weighted aggregated temporal features into the prediction module to obtain the respective subtitle transition probability of each of the plurality of subtitle video frames; and determining the subtitle switching frame from the plurality of subtitle video frames based on the respective subtitle transition probability of each of the plurality of subtitle video frames.

[0013] In a preferred example, the application can be further configured to: performing text recognition on the subtitle video frames in the subtitle group to obtain subtext content, comprising: sampling the subtitle video frames in the subtitle group to obtain a plurality of subtitle sampling frames; sequentially performing character recognition on the plurality of subtitle sampling frames to obtain the respective initial subtext content of each of the plurality of subtitle sampling frames; and determining the corresponding subtext content of each of the subtitle groups according to the respective initial subtext content of each of the plurality of subtitle sampling frames and the corresponding time axis information.

[0014] In a preferred example, the application can be further configured to: determining the subtitle file according to the subtext content of each of the subtitle groups, further comprising: proofreading the subtitle file based on a large language model to obtain a standard subtitle file.

[0015] In a second aspect, a hard-coded subtitle extraction device is provided, comprising: an acquisition module configured to acquire a video frame sequence; a cropping module configured to crop the video frame sequence to obtain a subtitle cropped video frame sequence, the subtitle cropped video frame sequence comprising a plurality of subtitle video frames, the subtitle video frames comprising at least a subtitle region; a grouping module configured to group the plurality of subtitle video frames in the subtitle cropped video frame sequence to obtain a plurality of subtitle groups, the subtitle groups comprising a plurality of continuous subtitle video frames; an identification module configured to perform text recognition on the subtitle video frames in the subtitle groups to obtain subtext content; and a determination module configured to determine a subtitle file according to the subtext content of each of the subtitle groups.

[0016] In a third aspect, an electronic device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the hard-coded subtitle extraction method of any one of the first aspect.

[0017] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores at least one program code, which is loaded and executed by a processor to implement the hard-coded subtitle extraction method according to any one of the first aspect.

[0018] In a fifth aspect, a computer program product is provided, and the computer program product includes a computer program or instructions, which, when executed by a processor, implement the hard-coded subtitle extraction method according to any one of the first aspect.

[0019] To sum up, the hard-coded subtitle extraction method provided by the present application has the following beneficial technical effects: a video frame sequence is obtained; the video frame sequence is subjected to subtitle cropping, which can accurately focus on a subtitle area, so that a subtitle cropped video frame sequence containing only subtitle related content is obtained; a plurality of subtitle video frames in the subtitle cropped video frame sequence are grouped, and subtitle video frames that are continuous and possibly belong to the same subtitle content are grouped into a group, so that a plurality of subtitle groups are formed; text recognition is performed on the subtitle video frames in the subtitle groups, which can convert the image form of the subtitles into the subtext content in the form of text, thereby improving the processing efficiency and accuracy; and a subtitle file is determined according to the subtext content of each subtitle group, so that the hard-coded subtitles are completely extracted from the video, and efficient and accurate subtitle extraction is achieved.

[0020] In addition, the present application also provides a hard-coded subtitle extraction device, equipment and medium, which all have the above beneficial technical effects. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0022] Figure 1 is a hard-coded subtitle extraction method flowchart provided by an embodiment of the present application;

[0023] Figure 2 is a data processing flowchart of a subtitle transition detection network model provided by an embodiment of the present application;

[0024] Figure 3 is another hard-coded subtitle extraction method flowchart provided by an embodiment of the present application;

[0025] Figure 4 is a structural diagram of a hard-coded subtitle extraction device provided by an embodiment of the present application;

[0026] Figure 5FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] The specific embodiments are only an explanation of the present application, and are not a limitation of the present application. Those skilled in the art can make modifications to the embodiments without creative contribution after reading the present specification, and the modifications are protected by the patent law as long as they are within the scope of the present application.

[0028] It should be noted that, in the optional embodiments of the present application, the data related to the object information and the like needs to be obtained with the permission or consent of the object when the embodiments of the present application are applied to specific products or technologies, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the country and region. That is, if the embodiments of the present application involve data related to the object, the data needs to be obtained with the permission and consent of the object, the permission and consent of the relevant department, and in compliance with the relevant laws, regulations and standards of the country and region. If the embodiments involve personal information, the consent of the individual needs to be obtained for the acquisition of all personal information, and the separate consent of the information subject needs to be obtained for sensitive information, and the embodiments also need to be implemented with the permission and consent of the object.

[0029] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0030] In addition, the term "and / or" in the present document is only to describe the association relationship of the associated objects, and can represent the existence of three relationships, for example, A and / or B can represent the existence of A alone, the existence of A and B together, and the existence of B alone. In addition, the character " / " in the present document generally represents an "or" relationship between the associated objects unless otherwise specified.

[0031] The present related technology has the following disadvantages:

[0032] The scheme of text detection and OCR recognition frame by frame needs to detect and recognize the subtitle text in the subtitle area frame by frame, which has huge calculation amount and slow speed. This way does not take advantage of the time domain characteristics of subtitles that do not change in continuous multiple frames, and does repeated calculation on frames without subtitles and repeated subtitle frames without distinction, resulting in a sharp increase in computing cost and time cost. Secondly, the same subtitle is identified multiple times in multiple frames, and if the text is slightly different or the text is missed or misidentified, the text of the same subtitle group cannot be correctly merged, resulting in the problem of repeated subtitle groups.

[0033] The scheme of locating the subtitle switching position by the traditional visual threshold method only uses traditional visual information (such as frame difference, edge intensity, histogram change, threshold binarization, connected domain area, etc.) to judge whether the picture has changed significantly to locate the subtitle switching position. This scheme is easily disturbed by video backgrounds. When the lens is switched, the picture moves quickly, the brightness changes suddenly, and special effects are superimposed, the overall difference of the picture is large but the subtitle does not change, this method is easy to misidentify as subtitle switching, resulting in problems such as subtitle timeline jitter and subtitle grouping error. Secondly, when the video background is basically consistent, the subtitle text is different, but the number of subtitles is consistent or the shape of the subtitles is similar, it is difficult to accurately determine that the subtitles have changed. The most important problem is that this scheme highly depends on manual visual threshold setting. This threshold is generally fixed. After calculating the inter-frame visual information value, threshold judgment is needed. If it exceeds the threshold, it is considered that the subtitle switching has occurred. However, the fixed threshold cannot adapt to different types of videos, and the accuracy is not high, which can easily cause the problems of subtitle missing and missing identification.

[0034] In view of the problems of the existing open source or commercial level hard coded subtitle extraction methods, the embodiment of the present application designs a super-fast hard coded subtitle extraction method based on subtitle transition AI detection. This method realizes a full AI link hard coded subtitle extraction scheme. First, the position of the hard coded subtitle in the video is intelligently located by a subtitle area positioning AI model. After determining the subtitle area, the local picture of the subtitle area is cropped in the video. Then, the subtitle switching frame position is intelligently detected by a subtitle transition AI model. According to the subtitle switching detection result, the time axis range of each subtitle group is divided. Finally, the video frames in each subtitle group time axis range are randomly sampled for text detection and OCR recognition, and the optimal subtitle text is selected by an intelligent merging strategy. After all the subtitle groups are recognized, the subtitle text is corrected based on a large language model, and finally a subtitle file is output. From the hard coded subtitle video, the subtitle time and subtitle content are accurately extracted and converted into an editable subtitle file with a time axis.

[0035] The embodiment of the present application provides a hard coded subtitle extraction method, such as Figure 1As shown, the hard-coded subtitle extraction method provided in the embodiments of the present application can be executed by an electronic device, which can be a server, a server cluster composed of multiple physical servers, a distributed system, or a cloud server providing cloud computing services. The terminal device can be a smartphone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The terminal device and the electronic device can be directly or indirectly connected through wired or wireless communication, and the embodiments of the present application do not limit this. The hard-coded subtitle extraction method includes:

[0036] S101, obtaining a video frame sequence.

[0037] A video to be processed is obtained, and the video to be processed is decoded to obtain a video frame sequence. The video frame sequence includes a plurality of video frames decoded in time sequence.

[0038] S102, performing subtitle cropping on the video frame sequence to obtain a subtitle-cropped video frame sequence.

[0039] The subtitle-cropped video frame sequence includes a plurality of subtitle video frames, and each subtitle video frame includes at least a subtitle region. The subtitle-cropped video frame sequence is a video frame sequence including subtitles obtained by cropping subtitles from the video frame sequence.

[0040] S103, grouping the plurality of subtitle video frames in the subtitle-cropped video frame sequence to obtain a plurality of subtitle groups.

[0041] Each subtitle group includes a plurality of continuous subtitle video frames, and the subtitle content in the subtitle video frames is theoretically the same.

[0042] In some embodiments, whether to be grouped together can be determined according to the similarity of adjacent subtitle video frames and a preset similarity threshold, so as to achieve the purpose of grouping. In other embodiments, a model can be used for grouping, and the specific manner is not limited by the embodiments of the present application.

[0043] S104, performing text recognition on the subtitle video frames in the subtitle group to obtain a subtext content.

[0044] Each subtitle group can obtain a subtext content (specific content of the subtitle). In some embodiments, any subtitle video frame in the subtitle group can be selected for recognition to obtain the subtext content corresponding to the subtitle group. In other embodiments, a plurality of subtitle video frames in the subtitle group are selected for recognition to obtain the subtext content corresponding to the subtitle group, and then the optimal subtext content is selected therefrom. The specific manner is not limited by the embodiments of the present application.

[0045] S105, determining a subtitle file according to the subtext content of each subtitle group.

[0046] In the embodiment of the present application, the standard subtitle text content with time axis information is generated combined with the subtext content of each subtitle group, and a subtitle file is obtained.

[0047] It can be seen that in the embodiment of the present application, the video frame sequence is obtained, the video frame sequence is subjected to subtitle cropping, the subtitle area can be accurately focused, and the subtitle cropped video frame sequence containing only the subtitle related content is obtained, the multiple subtitle video frames in the subtitle cropped video frame sequence are grouped, the subtitle video frames that are continuous and possibly belong to the same subtitle content are grouped into a group, and multiple subtitle groups are formed, the text recognition is performed on the subtitle video frames in the subtitle group, the image form subtitle can be converted into the subtext content in the form of text, and the processing efficiency and accuracy are improved, the subtitle file is determined according to the subtext content of each subtitle group, the hard-coded subtitle is completely extracted from the video, and efficient and accurate subtitle extraction is realized.

[0048] In one possible implementation of the embodiment of the present application, the video frame sequence is subjected to subtitle cropping to obtain the subtitle cropped video frame sequence, which includes: sampling the video frame sequence to obtain a video sampling frame sequence; using a subtitle region positioning model to position the subtitle region of each video sampling frame in the video sampling frame sequence to obtain the initial subtitle bounding box coordinate position corresponding to each video sampling frame; determining the subtitle bounding box coordinate position according to the initial subtitle bounding box coordinate position corresponding to each video sampling frame in the video sampling frame sequence; and based on the subtitle bounding box coordinate position, performing subtitle cropping on the video frame sequence to obtain the subtitle cropped video frame sequence.

[0049] In one possible implementation, two frames per second are sampled from the video frame sequence to form the video sampling frame sequence. The subtitle region positioning model is used to infer and predict each video sampling frame in the input video sampling frame sequence to obtain the initial subtitle bounding box coordinate position corresponding to each video sampling frame.

[0050] The subtitle region positioning model can be a lightweight target detection model, which can detect whether there is a subtitle in each video sampling frame, and if there is a subtitle, the subtitle bounding box coordinate position in the video sampling frame, i.e., the initial subtitle bounding box coordinate position, is located. Then, the initial subtitle bounding box coordinate position of each video sampling frame is integrated, and the maximum bounding box coordinate of the hard-coded subtitle in the video frame, i.e., the subtitle bounding box coordinate position, is finally determined.

[0051] After obtaining the subtitle bounding box coordinate position of the hard-coded subtitle in the video frame, the video frame sequence is cropped frame by frame through the subtitle bounding box coordinate position to form the subtitle cropped video frame sequence.

[0052] It can be seen that, in the embodiment of the present application, the video frame sequence is sampled to obtain a video sample frame sequence, which reduces the amount of data to be processed while retaining key information; the subtitle region positioning model is used to position the subtitle region of each video sample frame in the video sample frame sequence to obtain the initial subtitle bounding box coordinate position corresponding to each video sample frame; the subtitle bounding box coordinate position is determined according to the initial subtitle bounding box coordinate position corresponding to each video sample frame in the video sample frame sequence; and the video frame sequence is subtitle cropped based on the subtitle bounding box coordinate position to obtain a subtitle cropped video frame sequence containing only the subtitle region, which removes a large amount of irrelevant background information and the like in the video frame, thereby improving the accuracy and efficiency of subtitle extraction.

[0053] In one possible implementation of the embodiment of the present application, the plurality of subtitle video frames in the subtitle cropped video frame sequence are grouped to obtain a plurality of subtitle groups, including: determining a subtitle switching frame in the plurality of subtitle video frames in the subtitle cropped video frame sequence by using a subtitle transition detection model; and grouping the plurality of subtitle video frames according to each subtitle switching frame to determine the plurality of subtitle groups.

[0054] It can be seen that, in the embodiment of the present application, the subtitle transition detection model is used to determine the subtitle switching frame in the plurality of subtitle video frames in the subtitle cropped video frame sequence; and the subtitle video frames that are continuous and have relatively stable subtitle content are divided into the same group, thereby improving the accuracy and processing efficiency of subtitle extraction.

[0055] Specifically, the plurality of subtitle video frames are grouped according to each subtitle switching frame to determine the plurality of subtitle groups, including: determining the frame index of each subtitle switching frame; determining the starting frame index and the ending frame index corresponding to each subtitle group according to the frame index of each subtitle switching frame; and determining the plurality of subtitle groups according to the starting frame index and the ending frame index corresponding to each subtitle group.

[0056] The subtitle cropped video frame sequence is input into the subtitle transition detection model for inference prediction to determine the subtitle switching frame in the plurality of subtitle video frames in the subtitle cropped video frame sequence.

[0057] The subtitle transition detection model is a lightweight AI model based on time series convolution and attention mechanism aggregation, which can perform binary classification probability prediction on a plurality of input subtitle video frames, and output the probability of whether each subtitle video frame is a subtitle switching frame. If a subtitle video frame is not a subtitle switching frame, the probability tends to 0, and if a subtitle video frame is a subtitle switching frame, the probability tends to 1. Through the subtitle transition detection model, the probability of whether each subtitle video frame in the subtitle cropped video frame sequence is a subtitle switching frame can be accurately predicted. If the probability is greater than 0.8, it is considered that a subtitle switching occurs in the subtitle video frame, and all subtitle switching frames are obtained. Further, the frame index of all subtitle switching frames is obtained.

[0058] Based on the subtitle switching frame index result, each subtitle group is sequentially divided to obtain the starting frame index and the ending frame index of each subtitle group; for example, if the frame indexes of the subtitle switching frames are 5, 9, 15 and 20 respectively, the first subtitle group is 1 frame-4 frame, the second subtitle group is 5 frame-8 frame, the third subtitle group is 9 frame-14 frame, the fourth subtitle group is 15 frame-19 frame, and the fifth subtitle group is 20 frame-25 frame.

[0059] It can be seen that in the embodiment of the present application, the frame indexes of the various subtitle switching frames are determined; the starting frame index and the ending frame index corresponding to each subtitle group are determined according to the frame indexes of the various subtitle switching frames, which clearly defines the range of each subtitle group in the video frame sequence; and the multiple subtitle groups are determined according to the starting frame index and the ending frame index corresponding to each subtitle group.

[0060] Further, for the subtitle transition detection model, in the embodiment of the present application, a large high-quality subtitle transition detection data set is first labeled, covering multiple types of videos (films, variety shows, animations, game recordings, classrooms, conferences, etc.), multiple language videos (Chinese / English / Japanese / Korean / Arabic, etc.), and multiple plate types (single / double lines, left / right dialogue, top / middle / bottom annotations, vertical arrangement, etc.).

[0061] In the present application, a lightweight subtitle transition detection model network structure based on time series convolution and attention mechanism aggregation is designed, the core of which includes: 1) time series convolution and pyramid time series receptive field: combining multi-scale time series convolution of short window and long window, it can capture the subtle dynamics of subtitle fade-in and fade-out and stroke change, and save long-time video sequence visual feature memory, and improve the robustness of switching positioning; 2) subtitle area attention mechanism: through intra-frame and inter-frame visual feature guidance, focusing on the time series changes of the subtitle area, and suppressing the interference brought by video background, video effects and lens changes. In the model training process, through time series data enhancement (frame speed change, frame insertion / loss, brightness / color jitter, compression noise injection, etc.), the generalization performance of the model in real scene videos is improved. Through the above strategies, the subtitle switching accuracy and recall rate are significantly better than the traditional visual threshold scheme, the subtitle time axis start and end error is reduced, and the subtitle grouping is more stable and accurate.

[0062] Further, in a possible implementation of the embodiment of the application, the subtitle transition detection model comprises a visual feature encoder, a pyramid temporal convolution module, a subtitle region attention module and a prediction module. Correspondingly, the subtitle transition detection model is used to determine the subtitle switching frame in the plurality of subtitle video frames in the subtitle cropped video frame sequence, which comprises: encoding the video frame sequence based on the visual feature encoder to obtain encoded features; inputting the encoded features into the pyramid temporal convolution module to obtain multi-scale temporal features; inputting the multi-scale temporal features and the video frame sequence into the subtitle region attention module to obtain weighted aggregated temporal features; inputting the weighted aggregated temporal features into the prediction module to obtain the respective subtitle transition probability of the plurality of subtitle video frames; and determining the subtitle switching frame from the plurality of subtitle video frames based on the respective subtitle transition probability of the plurality of subtitle video frames.

[0063] Referring to Figure 2 , Figure 2 A data processing flow diagram of a subtitle transition detection network model provided by the embodiment of the application is shown in FIG. 1.

[0064] The input video frame sequence (B, T, H, W, C) is shown in FIG. 2, wherein B represents BatchSize (batch size), T represents sequence length (the number of continuous video frames contained in a video sample), H represents the height of each frame image, W represents the width of each frame image, and C represents the number of channels.

[0065] The video frame sequence is encoded based on the visual feature encoder to obtain encoded features. Specifically, the visual feature encoder is responsible for extracting preliminary visual features from each input video image. In a possible implementation, a pre-trained lightweight CNN backbone network MobileNetV3 on the ImageNet dataset can be used. After the visual feature encoder, the video frame sequence (B, T, H, W, C) becomes a temporal feature sequence (B, T, H', W', C) i.e. encoded features, wherein H' and W' represent the feature map size after downsampling.

[0066] The encoded features are input into a pyramid temporal convolution module to obtain multi-scale temporal features, wherein parallel multi-scale convolution in the pyramid temporal convolution module can input the feature sequence (B, T, H', W', C) output by the previous layer in the time dimension T into two 1D temporal convolution layers (short window temporal convolution and long window temporal convolution) of different scales in parallel. Specifically, the short window temporal convolution can use a smaller convolution kernel (kernel_size=3) for convolution, which focuses on capturing fast and subtle dynamics such as subtitle fade-in and fade-out, color or stroke changes. The long window temporal convolution can use a larger convolution kernel (kernel_size=9) for convolution, and combine with the dilated convolution to expand the receptive field without increasing the amount of calculation. The long window temporal convolution is responsible for capturing visual feature associations in a longer time range, such as a long gradual transition or a scene change that causes a subtitle interruption, which helps to preserve long-term memory and improve the robustness of the model to complex transitions. Then, the feature maps output by the short window and long window convolutions are spliced in the channel dimension (C') to output an aggregated feature sequence (B, T, H', W', C_agg) containing multi-scale temporal information, i.e., multi-scale temporal features.

[0067] The subtitle region attention module includes an inter-frame spatial attention module, an inter-frame temporal attention module, and a feature fusion module. Specifically, the single-frame feature map (H', W', C') output by the visual feature encoder is learned by a convolution layer of the inter-frame spatial attention module to obtain a spatial attention map (H', W'), where each value in the map represents the probability that the spatial position is a subtitle region. This process enables the model to learn to locate the possible position of the subtitle in the single-frame image. The multi-scale temporal features (B, T, H', W', C_agg) output by the pyramid temporal convolution module are input into the inter-frame temporal attention module to calculate the difference between the features of adjacent frames or frames within a short window. This difference will significantly increase when the subtitle changes (appears, disappears, scrolls), while it is relatively stable in the background region, generating a temporal attention weight.

[0068] Through the feature fusion module, the spatial attention map learned by the inter-frame spatial attention module is multiplied by the temporal attention weight learned by the inter-frame temporal attention module to obtain a spatio-temporal attention matrix. Then, the spatio-temporal attention weight is multiplied by the output features of the pyramid temporal convolution module through a weighting parameter to obtain the final weighted aggregated features, so as to output an aggregated temporal feature (B, T, H', W', C_agg) weighted by attention.

[0069] The weighted aggregated temporal features are input into a prediction module to obtain a subtitle transition probability corresponding to each of the plurality of subtitle video frames. The weighted aggregated temporal features are first compressed in a spatial dimension (H', W') by a global average pooling to obtain (B, T, C agg), then information in a time dimension is further aggregated by using a plurality of 1D convolution, and then a probability value between 0 and 1 is output for each frame in the sequence by a fully connected layer and a sigmoid activation function. A (B, T, 1) probability sequence is output, wherein each value represents the confidence degree of the corresponding frame being a subtitle transition frame (start or end).

[0070] It can be seen that, in the embodiment of the present application, the video frame sequence is encoded based on the visual feature encoder to obtain encoded features; the encoded features are input into the pyramid temporal convolution module to obtain multi-scale temporal features, the pyramid temporal convolution module can capture the temporal relationship between video frames from different time scales, and the feature information is enriched; the multi-scale temporal features are input into the subtitle region attention module to obtain weighted aggregated temporal features, the subtitle region attention module can focus on the temporal / spatial features of the subtitle region; the weighted aggregated temporal features are input into the prediction module to obtain a subtitle transition probability corresponding to each of the plurality of subtitle video frames, the prediction module can accurately predict the probability of each subtitle video frame occurring subtitle transition according to the previous feature information; and the subtitle switching frame is determined from the plurality of subtitle video frames based on the subtitle transition probability corresponding to each of the plurality of subtitle video frames, so that the frame in which the subtitle changes can be accurately identified.

[0071] In one possible implementation of the embodiment of the present application, text recognition is performed on the subtitle video frames in the subtitle group to obtain subtext content, including: sampling the subtitle video frames in the subtitle group to obtain a plurality of subtitle sampling frames; sequentially performing text recognition on the plurality of subtitle sampling frames to obtain initial subtext content corresponding to each of the plurality of subtitle sampling frames; and determining the subtext content corresponding to the subtitle group according to the initial subtext content corresponding to each of the plurality of subtitle sampling frames and the corresponding time axis information.

[0072] In the embodiment of the present application, according to the start frame index and the end frame index of each subtitle group, sampling is performed within the frame index range of the subtitle group, text detection is performed on the random subtitle sampling frame, and after the text is detected, an OCR model is used to recognize the text region to obtain the initial subtext content of each sampling frame. After the text recognition of all the subtitle sampling frames is completed, the recognized text can also be standardized (special symbols are removed, full-width and half-width characters are converted, etc.), and then the initial subtext content with the highest occurrence frequency is selected as the subtext content of the subtitle group.

[0073] It can be understood that the subtext content of all subtitle groups is sequentially determined until the text of all subtitle groups is recognized, and the subtitle start time and the subtitle end time of each subtitle group can be calculated according to the start frame index and the end frame index of each subtitle group, and the subtext content corresponding to each subtitle group obtained carries time information.

[0074] In a possible implementation of the embodiment of the application, after the subtitle file is determined according to the subtext content of each subtitle group, the method further includes: proofreading the subtitle file based on a large language model to obtain a standard subtitle file.

[0075] In the embodiment of the application, the generated standard subtitle text content with time axis information is proofread based on a large language model, and the wrong characters that do not conform to semantics and context in the subtitle text are corrected to generate new corrected standard subtitle text content with time axis information; the new corrected standard subtitle text content with time axis information is written into a subtitle text file, and the subtitle file is output.

[0076] Based on any of the above embodiments, a hard-coded subtitle extraction method based on a subtitle transition detection AI model is proposed, as shown in Figure 3 The method realizes a full-AI-link hard-coded subtitle extraction scheme. First, the subtitle region positioning model can intelligently locate the region position of the hard-coded subtitle in the video, and then the subtitle transition detection model can automatically detect the subtitle switching position in the continuous video frames, and then divide the subtitle groups automatically through the subtitle switching detection result. Then, a few frames are randomly sampled in a single subtitle group for subtitle text detection and recognition, and then the best subtitle text is selected as the subtitle text of the subtitle group through the optimal subtitle selection strategy. Finally, the large language model is used for context and semantic proofreading of the subtitle text to ensure accuracy.

[0077] Compared with the existing hard-coded subtitle extraction method, the method provided in the embodiment of the application has the following effects:

[0078] (1) Avoiding frame-by-frame text detection and OCR text recognition, we train a subtitle transition detection AI model based on the characteristic that the subtitle does not change in continuous multiple frames, and then locate the subtitle switching frame through the subtitle transition detection model, and then divide the subtitle groups according to the subtitle switching frame index, and then only extract a few frames in each subtitle group for text detection and OCR recognition, the calculation amount only increases with the number of subtitles instead of the total number of video frames, avoiding text detection and OCR inference on a large number of invalid frames and repeated subtitle frames, which not only saves inference computing power, but also exponentially improves processing speed.

[0079] (2) The traditional visual threshold method for positioning the subtitle switching position scheme is prone to interference from video backgrounds and lens changes, difficult to distinguish between subtitle areas and non-subtitle areas, and unable to adaptively determine the threshold, and the application greatly improves the accuracy of the traditional visual threshold method for positioning the subtitle switching position scheme.

[0080] An apparatus provided by an embodiment of the application is described below, and the apparatus described below can be referred to each other in correspondence with the method described above. The apparatus of the embodiment is arranged in an electronic device, and the electronic device is described with reference to Figure 4 , Figure 4 is a structural block diagram of the apparatus of one embodiment of the application, which includes: an acquisition module 210, configured to acquire a video frame sequence; a cropping module 220, configured to crop subtitles from the video frame sequence to obtain a subtitle-cropped video frame sequence, the subtitle-cropped video frame sequence including a plurality of subtitle video frames, and each subtitle video frame including at least a subtitle area; a grouping module 230, configured to group the plurality of subtitle video frames in the subtitle-cropped video frame sequence to obtain a plurality of subtitle groups, and each subtitle group including a plurality of continuous subtitle video frames; an identification module 240, configured to perform text recognition on the subtitle video frames in the subtitle groups to obtain subtext content; and a determination module 250, configured to determine a subtitle file according to the subtext content of each subtitle group.

[0081] In an implementable manner, the cropping module 220 is configured to: sample the video frame sequence to obtain a video sample frame sequence; perform subtitle area positioning on each video sample frame in the video sample frame sequence by using a subtitle area positioning model to obtain an initial subtitle bounding box coordinate position corresponding to each video sample frame; determine a subtitle bounding box coordinate position according to the initial subtitle bounding box coordinate position corresponding to each video sample frame in the video sample frame sequence; and perform subtitle cropping on the video frame sequence based on the subtitle bounding box coordinate position to obtain the subtitle-cropped video frame sequence.

[0082] In an implementable manner, the grouping module 230 is configured to: determine a subtitle switching frame in the plurality of subtitle video frames in the subtitle-cropped video frame sequence by using a subtitle transition detection model; and group the plurality of subtitle video frames according to the subtitle switching frames to determine the plurality of subtitle groups.

[0083] In an implementable manner, the grouping module 230 is configured to: determine a frame index of each subtitle switching frame; determine a starting frame index and a terminal frame index corresponding to each subtitle group according to the frame index of each subtitle switching frame; and determine the plurality of subtitle groups according to the starting frame index and the terminal frame index corresponding to each subtitle group.

[0084] In an implementable manner, the subtitle transition detection model comprises a visual feature encoder, a pyramid temporal convolution module, a subtitle region attention module, and a prediction module; correspondingly, the grouping module 230 is configured to: encode the video frame sequence based on the visual feature encoder to obtain encoded features; input the encoded features into the pyramid temporal convolution module to obtain multi-scale temporal features; input the multi-scale temporal features and the video frame sequence into the subtitle region attention module to obtain weighted aggregated temporal features; input the weighted aggregated temporal features into the prediction module to obtain a plurality of subtitle transition probabilities corresponding to the plurality of subtitle video frames respectively; and determine the subtitle switching frame from the plurality of subtitle video frames based on the plurality of subtitle transition probabilities corresponding to the plurality of subtitle video frames respectively.

[0085] In an implementable manner, the identification module 240 is configured to: sample the subtitle video frames in the subtitle group to obtain a plurality of subtitle sample frames; sequentially perform text recognition on the plurality of subtitle sample frames to obtain initial subtext content corresponding to the plurality of subtitle sample frames respectively; and determine the subtext content corresponding to the subtitle group according to the initial subtext content corresponding to the plurality of subtitle sample frames respectively and the corresponding time axis information.

[0086] In an implementable manner, the method further comprises: proofreading the subtitle file based on the large language model to obtain a standard subtitle file.

[0087] Figure 5 A structural diagram of an electronic device provided by the embodiment of the present application is shown in FIG. 1. Figure 5 As shown in FIG. 1, the electronic device comprises a memory 60 configured to store a computer program, and a processor 61 configured to execute the computer program to implement the steps of the method of the above embodiment.

[0088] The electronic device provided by the embodiment of the present application can include but is not limited to a smart phone, a tablet computer, a notebook computer, or a desktop computer, etc.

[0089] The processor 61 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 61 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA), and the like. The processor 61 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU), and the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 61 can be integrated with a graphics processing unit (GPU) that is responsible for rendering and drawing content required to be displayed on the display screen. In some embodiments, the processor 61 can further include an artificial intelligence (AI) processor for processing computing operations related to machine learning.

[0090] The memory 60 can include one or more computer-readable storage media that can be non-transitory. The memory 60 can further include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In the present embodiment, the memory 60 is at least used to store the following computer program 601, wherein the computer program is loaded and executed by the processor 61, and can implement the related steps of the method disclosed in any of the preceding embodiments. In addition, the resources stored in the memory 60 can further include an operating system 602, data 603, and the like, and the storage mode can be temporary storage or permanent storage. The operating system 602 can include Windows, Unix, Linux, and the like.

[0091] In some embodiments, the electronic device can further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0092] Those skilled in the art can understand that, Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than those shown.

[0093] It can be understood that if the method in the above embodiment is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and performs all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM, a register, a hard disk, a removable magnetic disk, a CD-ROM, a magnetic disk or an optical disk, and various media that can store program codes.

[0094] Based on this, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to realize the steps of the above method.

[0095] Based on this, the embodiment of the present application further provides a computer program product, which includes computer programs / instructions. The computer programs / instructions are executed by a processor to realize the steps of the above method. It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps has no strict sequence limitation, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order is not necessarily sequential, but can be alternately or alternately executed with other steps or sub-steps or stages of other steps.

[0096] The above is only some embodiments of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method of extracting hard-coded subtitles, characterized by, The method comprises the following steps: obtaining a video frame sequence; subtitle cropping the video frame sequence to obtain a subtitle cropped video frame sequence, wherein the subtitle cropped video frame sequence comprises a plurality of subtitle video frames, and the subtitle video frames at least comprise a subtitle region; grouping the plurality of subtitle video frames in the subtitle cropped video frame sequence to obtain a plurality of subtitle groups, wherein the subtitle groups comprise a plurality of continuous subtitle video frames; performing text recognition on the subtitle video frames in the subtitle groups to obtain subtext content; determining a subtitle file according to the subtext content of each subtitle group.

2. The method of claim 1, wherein, The subtitle cropping of the video frame sequence to obtain the subtitle cropped video frame sequence comprises the following steps: sampling the video frame sequence to obtain a video sampling frame sequence; using a subtitle region positioning model to position a subtitle region in each video sampling frame in the video sampling frame sequence to obtain initial subtitle bounding box coordinate positions corresponding to each video sampling frame; determining subtitle bounding box coordinate positions according to the initial subtitle bounding box coordinate positions corresponding to each video sampling frame in the video sampling frame sequence; cropping the video frame sequence based on the subtitle bounding box coordinate positions to obtain the subtitle cropped video frame sequence.

3. The method of claim 1, wherein, The grouping of the plurality of subtitle video frames in the subtitle cropped video frame sequence to obtain the plurality of subtitle groups comprises the following steps: using a subtitle transition detection model to determine subtitle switching frames in the plurality of subtitle video frames in the subtitle cropped video frame sequence; grouping the plurality of subtitle video frames according to the subtitle switching frames to determine the plurality of subtitle groups.

4. The method of claim 3, wherein, The grouping of the plurality of subtitle video frames according to the subtitle switching frames to determine the plurality of subtitle groups comprises the following steps: determining frame indexes of the subtitle switching frames; determining starting frame indexes and ending frame indexes corresponding to each subtitle group according to the frame indexes of the subtitle switching frames; determining the plurality of subtitle groups according to the starting frame indexes and the ending frame indexes corresponding to each subtitle group.

5. The method of claim 3, wherein, The subtitle transition detection model comprises a visual feature encoder, a pyramid temporal convolution module, a subtitle region attention module, and a prediction module. Correspondingly, the determination of the subtitle switching frames in the plurality of subtitle video frames in the subtitle cropped video frame sequence using the subtitle transition detection model comprises the following steps: encoding the video frame sequence based on the visual feature encoder to obtain encoded features; inputting the encoded features into the pyramid temporal convolution module to obtain multi-scale temporal features; inputting the multi-scale temporal features and the video frame sequence into the subtitle region attention module to obtain weighted aggregated temporal features; inputting the weighted aggregated temporal features into the prediction module to obtain subtitle transition probabilities corresponding to the plurality of subtitle video frames respectively; determining the subtitle switching frames from the plurality of subtitle video frames based on the subtitle transition probabilities corresponding to the plurality of subtitle video frames respectively.

6. The method of claim 1, wherein, The text recognition on the subtitle video frames in the subtitle groups to obtain the subtext content comprises the following steps: sampling the subtitle video frames in the subtitle groups to obtain a plurality of subtitle sampling frames; performing character recognition on the plurality of subtitle sampling frames in sequence to obtain initial subtext content corresponding to the plurality of subtitle sampling frames respectively; According to the initial subtext content corresponding to each of the plurality of subtitle sampling frames and the corresponding time axis information, the subtext content corresponding to the subtitle group is determined.

7. The method of claim 1, wherein, After determining the subtext content of each subtitle group, the method further comprises: Based on the large language model, proofreading the subtitle file to obtain a standard subtitle file.

8. A hard-coded subtitle extraction apparatus characterized by comprising: Comprise: An acquisition module is configured to acquire a video frame sequence. A cropping module is configured to crop the video frame sequence to obtain a subtitle-cropped video frame sequence, wherein the subtitle-cropped video frame sequence comprises a plurality of subtitle video frames, and each subtitle video frame comprises at least a subtitle region. A grouping module is configured to group the plurality of subtitle video frames in the subtitle-cropped video frame sequence to obtain a plurality of subtitle groups, wherein each subtitle group comprises a plurality of continuous subtitle video frames. An identification module is configured to perform text recognition on the subtitle video frames in the subtitle groups to obtain subtext content. A determination module is configured to determine a subtitle file according to the subtext content of each subtitle group.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to perform the hard-coded subtitle extraction method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one program code, and the program code is loaded and executed by the processor to implement the hard-coded subtitle extraction method of any one of claims 1 to 7.

Images