A method for detecting movie narration intervals based on computer vision

Abstract

A computer vision-based method for detecting movie narration intervals includes: first, opening a movie to obtain images of continuous frames; second, sampling the obtained images and obtaining the coordinates of the subtitles through text recognition; then, detecting whether subtitles appear within the coordinates for all image frames, calculating the time intervals in which the subtitles appear, and filling the remaining time intervals with the time of narration; finally, fine-tuning the detection results, and giving a recommended number of narration words to be inserted based on the narration interval and the normal speaking speed. According to the above steps, all the narration intervals of the movie and the recommended number of inserted words for each narration can be determined. The present invention can be used to recommend narrations for movies, help better express the content of the movie, and provide more choices and possibilities for movie editors.

Description

Technical field:

[0001] The present invention belongs to the field of information accessibility and relates to a method for detecting narration intervals in a movie, and is particularly used to help visually impaired users better understand and read movie content. Background technology:

[0002] With the continuous improvement of people's quality of life, watching movies has become a common way for people to pursue spiritual life. In traditional movie-watching, people need to fully mobilize their senses and feel the impact of the film on their vision and hearing. For transitions without actors' lines, directors will also shoot some scene clips to enhance the atmosphere of the film to better create and convey the artistic conception, so that the audience can resonate more deeply and watch the film more immersively.

[0003] However, traditional movies are often designed without considering vulnerable groups in society, such as the visually impaired and the elderly. When watching traditional movies, they face an additional obstacle in obtaining information compared to normal users. For example, the visually impaired cannot see the actors' expressions, changes in scenes, etc. Therefore, it is difficult for these vulnerable groups to extract all the information the director wants to express when watching a movie. They can only hear the actors' lines and background music, resulting in a poor viewing experience.

[0004] Therefore, in order to improve the movie-watching experience of some vulnerable groups, lower the threshold for them to obtain the information that the film wants to convey, and better enjoy the rich spiritual life brought by movies like everyone else, we need to do a good job in building barrier-free movies.

[0005] The goal of making movies accessible is to enable people with disabilities to extract the various information conveyed in a film just like non-subtitled viewers, thereby enhancing their viewing experience. To achieve this goal, the key is to improve users' ability to extract information from "non-subtitle" sections. To process these sections, the first thing required is to determine which time intervals are non-subtitle sections, also known as "narration sections." Therefore, how to quickly and accurately detect narration sections becomes a critical issue. Summary of the invention:

[0006] To achieve this, the present invention proposes a computer vision-based method for detecting movie narration intervals. This method can detect narration areas within subtitled films and intelligently recommend the number of words that can be inserted into these narration areas based on the speech rate of the surrounding context. This method accurately determines the narration intervals within a movie and further intelligently recommends the number of words that can be inserted into the narration, thereby providing a foundation and guarantee for subsequent film processing to enhance the viewing experience.

[0007] A method for detecting movie narration intervals based on computer vision comprises the following steps:

[0008] S1: Input the file path of a movie in other video formats;

[0009] S2: Read the video image, read the image at a certain frame rate and analyze the image group to determine the upper and lower boundary positions of the subtitles in the entire picture;

[0010] S21: Perform OCR recognition on the read image to determine whether there is text in the image;

[0011] S22: If the image obtained in S21 contains text, perform gradient detection and calculate the tilt angle of the detected rectangular frame;

[0012] S23: If the tilt angle calculated in S22 is less than a certain threshold, it is marked as a valid subtitle, and the coordinates of the upper left corner and the lower right corner of the rectangle are recorded;

[0013] S24: When a certain number of valid subtitles are found, the subtitle position detection ends, and the largest rectangle in the record is taken as the detected subtitle position;

[0014] S3: Detect the video images of the entire movie at a certain frequency to obtain the start and end times of all subtitles;

[0015] S31: Cutting each frame of the image according to the upper and lower boundaries of the subtitle position obtained in S2, and performing OCR text recognition on the cut image to determine whether there is text in the image;

[0016] S32: If text is recognized in S31, gradient detection is performed and the tilt angle of the detected rectangular frame is calculated;

[0017] S33: If the tilt angle calculated in S32 is less than a certain threshold and the center of the subtitle is near the center of the picture, it is marked as a valid subtitle;

[0018] S34: If valid subtitles are detected in S33, some punctuation marks in the text are normalized, including normalizing some half-width symbols into full-width symbols, so as to facilitate the calculation of the speech rate of the narration near the subtitle and the word count recommendation based on the word count and time of the subtitle.

[0019] S35: judging the subtitle status of the current frame image according to the result of S34 and the processing of the previous frame image;

[0020] S351: If no subtitles were detected in the previous frame, but subtitles were detected in the current frame, it means that this is a new subtitle, and the start time of the subtitle is recorded;

[0021] S352: If subtitles were detected in the previous frame and not in the current frame, it means that the subtitles have disappeared. The end time of the subtitles is recorded, and a complete subtitle record is formed.

[0022] S353: If subtitles are detected in the previous frame image and subtitles are also detected in the current frame image, it is necessary to determine whether the two subtitles are the same subtitle;

[0023] S3531: Remove punctuation marks from the detected subtitles and perform similarity detection, with the detection result being between [0, 1];

[0024] S3532: If the similarity of the detection result is less than a certain threshold, it is considered a new subtitle, and the end time of the subtitle is recorded, forming a complete subtitle record;

[0025] S3533: If the similarity of the detection results is greater than a certain threshold, it is considered to be the same subtitle;

[0026] S4: traverse all detected subtitles, and insert a narration between two adjacent subtitles with a certain time interval, so as to form a narration interval list of the movie;

[0027] S5: Traverse each narration, and obtain the subtitle speed based on the word count and start and end time of the subtitle of the narration, and obtain the recommended word count of the narration based on the speech speed and the start and end time of the narration.

[0028] Preferably, the certain frame rate in steps S2 and S3 is 3 frames per second.

[0029] Preferably, the slope parameter of the tilt angle in steps S23 and S33 is 0.1.

[0030] Preferably, in step S33, the subtitle center is between 0.4 and 0.6 of the screen width.

[0031] Preferably, in step S24, the representative parameter of effective monitoring is 10 effective subtitles or more than 1000 images are inspected.

[0032] Preferably, in step S3532, the similarity threshold is 0.7.

[0033] Preferably, in step S3531, the similarity calculation formula of two subtitles is:

[0034]

[0035] Where a and b are the string representations of the two subtitles, and T represents the sum of the lengths of all matching segments.

[0036] Preferably, in steps S3532 and S3533, the similarity threshold is 0.7.

[0037] Preferably, in step S4, the parameter of the certain time interval is 1 second.

[0038] In summary, the present invention has created a method for detecting movie narration intervals based on computer vision, which has the following beneficial effects: (1) Through the movie narration interval detection method, the narration interval can be quickly and accurately fed back to the producer, facilitating their secondary processing and providing a way for visually impaired people to watch movies smoothly. (2) It is the first method to comprehensively use inter-frame comparison method, text recognition algorithm and subtitle position detection to assist in narration interval detection, and at the same time, intelligently recommend the current narration word count based on the speech speed of the previous and next subtitles, providing different ideas for the subsequent generation and modification of barrier-free movies. Description of the drawings:

[0039] Figure 1 This is an overall flow chart of the movie narration interval detection algorithm based on computer vision provided by the present invention.

[0040] Figure 2 This is a flowchart for obtaining the detection position of movie subtitles in the overall flowchart of the movie narration interval detection algorithm based on computer vision provided by the present invention.

[0041] Figure 3 This is a flowchart of subtitle transcription in the overall flowchart of the movie narration interval detection algorithm based on computer vision provided by the present invention. Specific implementation method:

[0042] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] This example uses a movie as an example. A computer vision-based method for detecting movie narration intervals includes the following specific steps:

[0044] S1: Input the file path of a movie in other video formats;

[0045] S2: Read the video image, read the image at a certain frame rate and analyze the image group to determine the upper and lower boundary positions of the subtitles in the entire picture;

[0046] S21: Perform OCR recognition on the read image to determine whether there is text in the image;

[0047] S22: If the image obtained in S21 contains text, perform gradient detection and calculate the tilt angle of the detected rectangular frame;

[0048] S23: If the tilt angle calculated in S22 is less than a certain threshold, it is marked as a valid subtitle, and the coordinates of the upper left corner and the lower right corner of the rectangle are recorded;

[0049] S24: When a certain number of valid subtitles are found, the subtitle position detection ends, and the largest recorded rectangle is taken as the detected subtitle position;

[0050] S3: Detect the video images of the entire movie at a certain frequency to obtain the start and end times of all subtitles;

[0051] S31: Cutting each frame of the image according to the upper and lower boundaries of the subtitle position obtained in S2, and performing OCR text recognition on the cut image to determine whether there is text in the image;

[0052] S32: If text is recognized in S31, gradient detection is performed and the tilt angle of the detected rectangular frame is calculated;

[0053] S33: If the tilt angle calculated in S32 is less than a certain threshold and the center of the subtitle is near the center of the picture, it is marked as a valid subtitle;

[0054] S34: If valid subtitles are detected in S33, some punctuation marks in the text are normalized, including normalizing some half-width symbols into full-width symbols, so as to facilitate the calculation of the speech rate of the narration near the subtitle and the word count recommendation based on the word count and time of the subtitle.

[0055] S35: judging the subtitle status of the current frame image according to the result of S34 and the processing of the previous frame image;

[0056] S351: If no subtitles were detected in the previous frame, but subtitles were detected in the current frame, it means that this is a new subtitle, and the start time of the subtitle is recorded;

[0057] S352: If subtitles were detected in the previous frame and not in the current frame, it means that the subtitles have disappeared. The end time of the subtitles is recorded, and a complete subtitle record is formed.

[0058] S353: If subtitles are detected in the previous frame image and subtitles are also detected in the current frame image, it is necessary to determine whether the two subtitles are the same subtitle;

[0059] S3531: Remove punctuation marks from the detected subtitles and perform similarity detection, with the detection result being between [0, 1];

[0060] S3532: If the similarity of the detection result is less than a certain threshold, it is considered a new subtitle, and the end time of the subtitle is recorded, forming a complete subtitle record;

[0061] S3533: If the similarity of the detection results is greater than a certain threshold, it is considered to be the same subtitle;

[0062] S4: traverse all detected subtitles, and insert a narration between two adjacent subtitles with a certain time interval, so as to form a narration interval list of the movie;

[0063] S5: Traverse each narration, and obtain the subtitle speed based on the word count and start and end time of the subtitle of the narration, and obtain the recommended word count of the narration based on the speech speed and the start and end time of the narration.

[0064] Figure 1 This is an overall flow chart of the movie narration interval detection algorithm based on computer vision provided by the present invention.

[0065] Figure 2 This is a flowchart for obtaining the detection position of movie subtitles in the overall flowchart of the movie narration interval detection algorithm based on computer vision provided by the present invention.

[0066] S21: Perform OCR recognition on the read image to determine whether there is text in the image;

[0067] S22: If the image obtained in S21 contains text, perform gradient detection and calculate the tilt angle of the detected rectangular frame;

[0068] S23: If the tilt angle calculated in S22 is less than a certain threshold, it is marked as a valid subtitle, and the coordinates of the upper left corner and the lower right corner of the rectangle are recorded;

[0069] S24: When a certain number of valid subtitles are found, the subtitle position detection ends, and the largest recorded rectangle is taken as the detected subtitle position.

[0070] Figure 3 This is a flowchart of subtitle transcription in the overall flowchart of the movie narration interval detection algorithm based on computer vision provided by the present invention.

[0071] S3: Detect the video images of the entire movie at a certain frequency to obtain the start and end times of all subtitles;

[0072] S31: Cutting each frame of the image according to the upper and lower boundaries of the subtitle position obtained in S2, and performing OCR text recognition on the cut image to determine whether there is text in the image;

[0073] S32: If text is recognized in S31, gradient detection is performed and the tilt angle of the detected rectangular frame is calculated;

[0074] S33: If the tilt angle calculated in S32 is less than a certain threshold and the center of the subtitle is near the center of the picture, it is marked as a valid subtitle;

[0075] S34: If valid subtitles are detected in S33, some punctuation marks in the text are normalized, including normalizing some half-width symbols into full-width symbols, so as to facilitate the calculation of the speech rate of the narration near the subtitle and the word count recommendation based on the word count and time of the subtitle.

[0076] S35: judging the subtitle status of the current frame image according to the result of S34 and the processing of the previous frame image;

[0077] S351: If no subtitles were detected in the previous frame, but subtitles were detected in the current frame, it means that this is a new subtitle, and the start time of the subtitle is recorded;

[0078] S352: If subtitles were detected in the previous frame and not in the current frame, it means that the subtitles have disappeared. The end time of the subtitles is recorded, and a complete subtitle record is formed.

[0079] S353: If subtitles are detected in the previous frame image and subtitles are also detected in the current frame image, it is necessary to determine whether the two subtitles are the same subtitle;

[0080] S3531: Remove punctuation marks from the detected subtitles and perform similarity detection, with the detection result being between [0, 1];

[0081] S3532: If the similarity of the detection result is less than a certain threshold, it is considered a new subtitle, and the end time of the subtitle is recorded, forming a complete subtitle record;

[0082] S3533: If the similarity of the detection results is greater than a certain threshold, it is considered to be the same subtitle.

[0083] This invention combines traditional image processing methods with deep learning to detect gaps in movie subtitles, determine the time intervals for narration, and recommend the number of words to insert. This method achieves a high recall rate and low error rate for narration interval detection. This invention can be applied to recommend narrations for movies, helping to better convey film content and providing more options and possibilities for film editors.

Claims

1. A method for detecting movie narration intervals based on computer vision, characterized in that: The following steps are involved: S1: Input the file path of a movie in other video formats; S2: Read the video image, read the image at a certain frame rate and analyze the image group to determine the upper and lower boundary positions of the subtitles in the entire picture; S21: Perform OCR recognition on the read image to determine whether there is text in the image; S22: If the image obtained in S21 contains text, perform gradient detection and calculate the tilt angle of the detected rectangular frame; S23: If the tilt angle calculated in S22 is less than a certain threshold, it is marked as a valid subtitle, and the coordinates of the upper left corner and the lower right corner of the rectangle are recorded; S24: When a certain number of valid subtitles are found, the subtitle position detection ends, and the largest rectangle in the record is taken as the detected subtitle position; S3: Detect the video image of the entire movie at a certain frame rate to obtain the start and end times of all subtitles; S31: Cutting each frame of the image according to the upper and lower boundaries of the subtitle position obtained in S2, and performing OCR text recognition on the cut image to determine whether there is text in the image; S32: If text is recognized in S31, gradient detection is performed and the tilt angle of the detected rectangular frame is calculated; S33: If the tilt angle calculated in S32 is less than a certain threshold and the center of the subtitle is near the center of the picture, it is marked as a valid subtitle; S34: If valid subtitles are detected in S33, some punctuation marks in the text are normalized, including normalizing some half-width symbols into full-width symbols, so as to facilitate the calculation of the speech rate of the narration near the subtitle and the word count recommendation based on the word count and time of the subtitle. S35: judging the subtitle status of the current frame image according to the result of S34 and the processing of the previous frame image; S351: If no subtitles were detected in the previous frame, but subtitles were detected in the current frame, it means that this is a new subtitle, and the start time of the subtitle is recorded; S352: If subtitles were detected in the previous frame and not in the current frame, it means that the subtitles have disappeared. The end time of the subtitles is recorded, and a complete subtitle record is formed. S353: If subtitles are detected in the previous frame image and subtitles are also detected in the current frame image, it is necessary to determine whether the two subtitles are the same subtitle; S3531: Remove punctuation marks from the detected subtitles and perform similarity detection, with the detection result being between [0, 1]; S3532: If the similarity of the detection result is less than a certain threshold, it is considered a new subtitle, and the end time of the subtitle is recorded, forming a complete subtitle record; S3533: If the similarity of the detection results is greater than a certain threshold, it is considered to be the same subtitle; S4: traverse all detected subtitles, and insert a narration between two adjacent subtitles with a certain time interval, so as to form a narration interval list of the movie; S5: Traverse each narration, and obtain the subtitle speed based on the word count and start and end time of the subtitle of the narration, and obtain the recommended word count of the narration based on the speech speed and the start and end time of the narration.

2. The method for detecting movie narration intervals based on computer vision according to claim 1, wherein: The certain frame rate in steps S2 and S3 is 3 frames per second.

3. The method for detecting movie narration intervals based on computer vision according to claim 1, wherein: The slope parameter of the tilt angle in steps S23 and S33 is 0.

1.

4. The method for detecting movie narration intervals based on computer vision according to claim 1, wherein: In step S33, the subtitle center is between 0.4 and 0.6 of the screen width.

5. The method for detecting movie narration intervals based on computer vision according to claim 1, wherein: In step S24, representative parameters of effective detection are 10 effective subtitles or checking more than 1000 images.

6. The method for detecting movie narration intervals based on computer vision according to claim 1, wherein: In step S3531, the similarity calculation formula of two subtitles is as shown in formula (1): Where a and b are the string representations of the two subtitles, and T represents the sum of the lengths of all matching segments.

7. The method for detecting movie narration intervals based on computer vision according to claim 1, wherein: In steps S3532 and S3533, the similarity threshold is 0.

7.

8. The method for detecting movie narration intervals based on computer vision according to claim 1, wherein: In step S4, the parameter of the certain time interval is 1 second.

Images