Characteristic mining-based subtitle extraction method and device for barrier-free movie production

Abstract

The invention discloses a characteristic mining-based subtitle extraction method and device for barrier-free movie production, and the method comprises the steps: firstly, selecting a movie to be processed, and extracting a series of video frames in the movie to serve as samples for mining subtitle characteristics; then, characters in the samples are recognized through a general character recognition model, and position characteristics and subtitle color characteristics of the subtitles are mined according to the recognition result; on the basis, a movie image is accurately intercepted by utilizing the mined subtitle position characteristics, and an image only containing subtitles is obtained. And performing character recognition on the caption image by using a general character recognition model to obtain a caption recognition result. Afterwards, the obtained subtitle recognition results are processed, and the recognition results with the colors not meeting the preset requirements and the repeated recognition results are removed. And finally, combining the processed subtitle recognition result with a timestamp, and accurately writing into a subtitle file. According to the method, the subtitle recognition precision can be remarkably improved, and a basis is provided for compiling barrier-free side white based on the film subtitles in the subsequent barrier-free film production process.

Description

Technical Field

[0001] The invention relates to the technical field of media content processing, in particular to the technical field of barrier-free movie production, and relates to a method and device for extracting subtitles. Background Art

[0002] At present, the state and society are paying more and more attention to the material and spiritual needs of people with disabilities. Movies are an important part of the spiritual life of healthy people, but it is difficult for visually impaired people to obtain movie images. It is difficult for visually impaired people to fully understand the content of the movie by only listening to the audio. Barrier-free movies are a barrier-free form of expression of movies. By adding narration describing the pictures in the audio-free position of the movie, visually impaired people can understand the content of the movie pictures, thereby improving the viewing experience of visually impaired people. At present, most movies use the method of integrating subtitles with video sources, and it is difficult to obtain subtitles directly. The general process of barrier-free movie production is: select movies for healthy people, identify subtitles in movies, record subtitle text content and corresponding timestamps, add narration text describing the pictures in the time period without subtitles and generate narration audio, and finally synthesize the generated narration audio with the original movie to obtain barrier-free movies. In the production process of barrier-free movies, the accuracy of extracting subtitles determines the production quality of barrier-free movies. In practice, it is found that most of the existing methods for extracting movie subtitles ignore the characteristics of subtitles in movies (these characteristics can be used as prior knowledge to improve recognition accuracy), and directly apply text recognition methods in general scenarios to recognize movie subtitles. This results in insufficient subtitle extraction accuracy, missed recognition of subtitles, and mistaking text in movie scenes for subtitles, which brings difficulties to accessible movie processing. Summary of the invention

[0003] The present invention provides a method and device for extracting subtitles based on feature mining for barrier-free movie production. The method improves the accuracy of subtitle extraction in barrier-free movie production by mining the features of movie subtitles.

[0004] A first aspect of the present invention relates to a subtitle extraction method based on feature mining for barrier-free movie production, comprising:

[0005] S110: Obtain the movie to be processed.

[0006] S120: Detecting global characteristic information of the subtitles in combination with the text recognition model.

[0007] S130: Using the position information in the global characteristic information to capture the movie image, and obtain a subtitle image.

[0008] S140: Perform text recognition on the subtitle image obtained in S130 using a text recognition model to obtain a subtitle recognition result.

[0009] S150: Processing the subtitle recognition results to remove the recognition results whose colors do not meet the requirements and the duplicate recognition results.

[0010] S160: Combining the processed subtitle recognition result with the timestamp and writing it into a subtitle file.

[0011] Furthermore, in step S110, the movie to be processed is obtained. The specific form of the movie to be processed is:

[0012] The resolution is above 720P, the subtitles are in Chinese and English (Simplified) or Chinese (Simplified), and the subtitles are embedded in the movie screen in a hard-embedded manner. Examples of methods for obtaining movies include downloading from movie websites.

[0013] Furthermore, in step S120, the global characteristic information of the subtitles is detected in combination with the text recognition model, and the specific method includes:

[0014] S1201: Download and load the offline model for general text recognition. Examples of offline models for general text recognition include Baidu Paddle OCR model, etc. The process of this type of general text recognition model recognizing text in an image generally includes: using a text detection method to obtain text area polygon information, cropping and perspective transformation correction of the above text polygon area, converting the text area into a rectangular frame, and then using a direction classifier to correct the direction, and finally performing text recognition based on the rectangular frame containing the text area to obtain the final recognition result. The general text recognition model is called to recognize text in an image, and the return value is the text area coordinates (including four pixel coordinate points: upper left, upper right, lower right, and lower left) and the text content.

[0015] S1202: Extract video frames from the movie as a sample set for further extracting the global characteristics of subtitles. The extraction method is as follows: obtain the total number of frames M of the movie, divide the total number of frames M into N parts, obtain N video intervals, and remove the opening and closing intervals to avoid the interference of the opening production information and the closing cast list on the extraction of the global characteristics of subtitles, obtain N-2 video intervals, and then evenly extract C frames from each video interval. A total of (N-2)*C frame images are obtained. Here, M, N, and C are flexibly set according to the frequency of occurrence of movie subtitles and the computing power of the device. Examples are: M=10000, N=5, C=60.

[0016] In S1202, C frames are uniformly extracted from each video interval. Specifically, the number of interval frames (M / N) is divided by C to obtain the number of interval frames X, and the starting frame number of the interval plus C is accumulated in sequence as the position of each extraction.

[0017] S1203: calling the universal text recognition model loaded in S1201 for each video frame of the sample set obtained in S1202 to obtain the rectangular position of the text, the text content, and the pixel matrix of the rectangle where the text is located for all frames of the sample set.

[0018] S1204: Since the position of subtitles in movies is relatively fixed and the frequency of appearance is greater than that of text in the movie screen, this step uses a classification and sorting method for the rectangular positions of the text in all frames of the sample set obtained in S1203, and uses the upper and lower boundaries with the most frequent appearance as the upper and lower boundary positions of the subtitle area. Since the color of subtitles is relatively fixed, this step uses the k-means method to cluster the pixel matrix of the rectangle where the text is located in all frames of the sample set in S1203 into two categories: background color and text color, and uses the category with higher brightness as the color of the subtitle text.

[0019] The classification and sorting method adopted in S1204 is specifically implemented as follows: This step mainly counts the upper and lower boundary positions of the subtitles from the sample set, but since the general text recognition model will have errors in detecting the rectangular frame where the text is located when recognizing the text, such as the rectangular frame will be too high, too low, too large, or too small, and there may be fixed interfering texts such as station logos and watermarks in the movie. Therefore, for each frame image of the sample set, this step classifies the rectangular frame of the newly recognized text with all category rectangular frames, and classifies the upper and lower boundary positions that are less than 5% of the height of the rectangular frame from the original category boundary position into one category, but if the text is consistent with the original category, the recognition result is discarded. If it can be classified into one category, the corresponding category count value is increased by one, otherwise it is added to the category rectangular frame set as a new category. Finally, the upper and lower boundary positions corresponding to the category with the largest count value are used as the upper and lower boundary positions of the subtitle area.

[0020] Furthermore, the specific implementation of capturing an image using the position information in the global characteristic information obtained in S120 to obtain a subtitle image in step S130 is as follows:

[0021] For every 5 video frames in the movie, one video frame is extracted, and the timestamp of the video frame is obtained according to the current frame number, the total number of frames, and the length of the movie. Then, for each video frame, the position information in the global feature information, that is, the upper and lower boundary positions of the subtitles, is used to capture the image of the video frame as the image for further identifying the subtitles.

[0022] Furthermore, the specific implementation of performing text recognition on the subtitle image obtained in S130 using the text recognition model in step S140 is as follows:

[0023] According to the S1201 method, a general text recognition offline model is loaded, and text recognition is performed on each captured video frame to obtain the rectangular position where the text is located, the text content of the text, and the pixel matrix of the rectangle where the text is located.

[0024] Furthermore, the specific implementation of processing the recognition results included in step S150 to remove the recognition results whose colors do not meet the requirements and the repeated recognition results is as follows:

[0025] S1501: The specific implementation of processing the recognition results in S150 and removing the recognition results whose colors do not meet the requirements is as follows: the pixel matrix of the rectangle where the text is located obtained in S140 is clustered into two categories by channel using k-means, and the category with high brightness is used as the text color, and then compared with the subtitle color in the subtitle global information obtained in S120. If the change in the brightness value of each channel is less than 30, it is regarded as a correctly recognized subtitle, otherwise the recognition result is discarded.

[0026] S1502: The specific implementation of processing the recognition results in S150 and removing duplicate recognition results is as follows: the reason for generating duplicate recognition results is that the duration of the subtitles is greater than the sampling interval, and the duplicate recognition results are continuous in time. The present invention recognizes and processes the sampled video frames one by one, so the present invention defines variables in the program to temporarily store the recognition result and timestamp of the previous frame, and calculates the similarity between the text of the current frame recognition result and the text of the previous frame recognition result (for example: calling Python's difflib library). If it is greater than 0.8, it is determined to be a duplicate, and the recognition result is discarded, and the end timestamp of the previous subtitle is extended. If it is not a duplicate, the current recognition result is added to the subtitle list and the temporary variable is updated.

[0027] Furthermore, the specific implementation of writing the subtitle recognition result combined with the timestamp into the subtitle file in step S160 is as follows:

[0028] The subtitle text and timestamp in the subtitle list obtained in S150 are written into a csv file (for example, using the csv library of python).

[0029] The second aspect of the present invention relates to a subtitle extraction device based on feature mining for barrier-free film production, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a subtitle extraction method based on feature mining for barrier-free film production of the present invention.

[0030] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a subtitle extraction method based on feature mining for barrier-free movie production of the present invention.

[0031] The present invention first selects a movie to be processed and extracts a series of video frames therein as samples for mining subtitle characteristics. Subsequently, a universal text recognition model is used to recognize the text in the sample, and the position characteristics and color characteristics of the subtitles are mined according to the recognition results. On this basis, the mined subtitle position characteristics are used to accurately capture the movie image to obtain an image containing only subtitles. The universal text recognition model is then used to perform text recognition on the subtitle image to obtain a subtitle recognition result. Thereafter, the obtained subtitle recognition results are processed to eliminate the recognition results whose colors do not meet the preset requirements and the repeated recognition results. Finally, the processed subtitle recognition results are combined with the timestamp and accurately written into the subtitle file. Through this series of steps, this method can significantly improve the accuracy of subtitle recognition, and provide a basis for writing accessible narration based on movie subtitles in the subsequent barrier-free movie production process.

[0032] In summary, the present invention has created a subtitle extraction method based on subtitle feature mining for barrier-free movie production, which has the following beneficial effects: (1) by mining the position characteristics and color characteristics of movie subtitles and using them as prior knowledge to guide subtitle recognition, the accuracy of subtitle extraction is improved; (2) an accurate subtitle extraction method is provided for barrier-free movie production, providing support for the subsequent production of barrier-free movies. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flow chart of a subtitle extraction method based on feature mining for barrier-free movie production provided by the present invention.

[0035] Figure 2 It is a schematic diagram of the rectangle where the text recognized by the general text recognition model is located.

[0036] Figure 3 It is a statistical diagram of the pixel values ​​in the area where the subtitles are located.

[0037] Figure 4 It is a schematic diagram of subtitle positions mined based on subtitle characteristics.

[0038] Figure 5 It is a schematic diagram of the device of the present invention. Specific implementation methods

[0040] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] Example 1

[0042] Reference Figure 1 to Figure 3 This embodiment discloses a subtitle extraction method based on feature mining for barrier-free movie production. This example takes the recognition of subtitles of the movie "The Legend of 1900" as an example. The method specifically includes the following steps:

[0043] S110: Divide the movie into five video intervals, and evenly extract 100 video frames from each of the three middle video intervals to obtain a sample set for mining subtitle characteristics.

[0044] S120: Use the general text recognition model Paddle OCR to recognize the sample set video frame, and obtain the rectangular position of the text, the text content of the text, and the pixel matrix of the rectangular position of the text. Classify and sort the upper and lower boundaries of the rectangular position to obtain the upper and lower boundary position characteristics of the subtitles. Use k-means to cluster the pixel matrix into two categories according to the channel, and the color with high brightness is used as the subtitle color feature.

[0045] S130: For every 5 video frames in the movie, extract a video frame, and obtain the timestamp of the current video frame according to the current frame number, the total number of frames, and the length of the movie. Then, for each video frame, use the position information in the global feature information, that is, the upper and lower boundary positions of the subtitles, to capture the image of the video frame as an image for further identifying the subtitles.

[0046] S140: using a general text recognition model Paddle OCR to recognize the subtitle image captured in S130, and obtaining the rectangular position of the text, the text content of the text, and the pixel matrix of the rectangle where the text is located.

[0047] S150: Process the text recognition results, analyze the pixel matrix, remove the results whose text colors do not conform to the subtitle color characteristics, and merge the recognition results that are repeated with the previous frame of text content.

[0048] S160: Writing the subtitle recognition result into a subtitle file in combination with the timestamp.

[0049] Example 2

[0050] Reference Figure 5This embodiment relates to a subtitle extraction device based on feature mining for barrier-free movie production, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a subtitle extraction method based on feature mining for barrier-free movie production in embodiment 1.

[0051] Example 3

[0052] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a subtitle extraction method based on feature mining for barrier-free movie production of the present invention is implemented.

[0053] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0054] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A subtitle extraction method based on feature mining for accessible film production, Its characteristics include the following steps: S110: Obtain the movie to be processed; S120: Detecting global characteristic information of the subtitles in combination with the text recognition model; S130: intercepting the movie image using the position information in the global characteristic information to obtain a subtitle image; S140: Performing text recognition on the subtitle image to obtain a subtitle recognition result; S150: Processing the subtitle recognition results, removing the recognition results whose colors do not meet the requirements and the duplicate recognition results; S160: Combining the processed subtitle recognition result with the timestamp and writing it into a subtitle file.

2. The method according to claim 1, characterized in that: The step S110 obtains the movie to be processed, and the format requirements for the movie are: the resolution is above 720P, the subtitles are in Chinese and English or Chinese, and the subtitles are embedded in the movie screen in a hard-embedded manner.

3. The method according to claim 1, characterized in that: The specific scheme of step S120 for detecting the global characteristic information of the subtitles in combination with the text recognition model is: S1201: Download and load an offline model for general text recognition; S1202: extracting video frames from the movie as a sample set for extracting global characteristics of subtitles; S1203: calling a general text recognition model for each video frame in the sample set to obtain the rectangular position of the text, the text content of the text, and the pixel matrix of the matrix where the text is located; S1204: For each frame image of the sample set, the rectangle of the newly recognized text is classified with all the category rectangles, and the upper and lower boundary positions whose difference from the original category boundary position is less than 5% of the rectangle height are classified into one category. However, if the text is consistent with the original category, the recognition result is discarded; if it can be classified into one category, the corresponding category count value is increased by one, otherwise it is added to the category rectangle set as a new category; finally, the upper and lower boundary positions corresponding to the category with the largest count value are used as the upper and lower boundary positions of the subtitle area; the pixel point matrix of the rectangle where the text is located in all frames of the sample set is clustered into two categories of background color and text color by channel using the k-means method, and the category with higher brightness is used as the color of the subtitle text.

4. The method according to claim 1, characterized in that: The specific method of obtaining the subtitle image by intercepting the movie image with the position information in the global characteristic information in step S130 is as follows: For every 5 video frames in the movie, one video frame is extracted, and the timestamp of the current video frame is obtained by combining the current frame number, the total number of frames, and the length of the movie. For each video frame, the position information in the global feature information is used to capture the image of the video frame as the image for further identifying subtitles.

5. The method according to claim 1, characterized in that: The step S140 uses a text recognition model to perform text recognition on the subtitle image obtained in S130. The specific method of obtaining the subtitle recognition result is as follows: Load the general text recognition offline model, perform text recognition on each captured video frame, and obtain the rectangular position of the text, the text content, and the pixel matrix of the rectangle where the text is located.

6. The method according to claim 1, characterized in that: The specific method of processing the subtitle recognition results in step S150 and removing the recognition results whose colors do not meet the requirements and the repeated recognition results is as follows: S1501: clustering the pixel matrix of the rectangle where the text is located obtained in S140 into two categories by channel using k-means, taking the category with higher brightness as the text color, and then comparing it with the subtitle color in the subtitle global information obtained in S120. If the change in brightness value of each channel is within 30, it is regarded as a correctly recognized subtitle, otherwise the recognition result is discarded; S1502: Define variables in the program to temporarily store the recognition result and timestamp of the previous frame, calculate the text similarity between the current frame recognition result and the previous frame recognition result, if the similarity is greater than 0.8, it is determined to be a duplicate, discard the recognition result, and extend the end timestamp of the previous subtitle. If it is not a duplicate, add the current recognition result to the subtitle list and update the recognition variable temporarily stored in the previous frame.

7. The method according to claim 1, characterized in that: In step S160, the subtitle recognition result is written into a subtitle file in combination with the timestamp. The format of the subtitle file is a csv table.

8. A subtitle extraction device based on feature mining for barrier-free movie production, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement a subtitle extraction method based on feature mining for barrier-free movie production as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a subtitle extraction method based on feature mining for barrier-free movie production as described in any one of claims 1 to 7 is implemented.

Images