De-duplication and extraction method applied to video continuous frame subtitle sequence

Through spatial coordinate projection merging and repeated subtitle merging technology, the problem of unfixed repetitive text and position in video subtitles is solved, and the deduplication and extraction of video subtitles is realized, and the complete sequence of repetitive texts is generated, which improves video text analysis efficiency.

CN119052531APending Publication Date: 2024-11-29BEIJING YANGLIAN ICT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411149097.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

When processing video subtitles, there are problems such as a large number of repetitive text, irregular subtitles position, and rolling subtitles mergers, making it difficult for the prior art to effectively identify and analyze text data in videos.

Method used

The spatial coordinate projection merging and repeated subtitle merging technology are used to establish a projection coordinate array and traverse the frame list, judge the intersection degree of subtitles and area, and realize the de-repellent merging and form a complete text field.

Benefits of technology

It effectively removes the duplicate subtitle text in the video, generates a sequence of repetitive subtitle texts that can be quickly analyzed, solves the problems of irregular subtitle position and scroll subtitle merging, and improves the accuracy and efficiency of video text recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119052531A_ABST
    Figure CN119052531A_ABST
Patent Text Reader

Abstract

The invention relates to the field of deduplication of video continuous frame subtitle sequences, in particular to a deduplication and extraction method applied to a video continuous frame subtitle sequence, which comprises space coordinate projection merging and repeated subtitle merging. According to the method, subtitles in videos form a section of readable non-repeated characters through a space coordinate projection merging technology and a repeated subtitle merging technology, so that the videos can be quickly analyzed, subtitle identification data of a continuous frame sequence can be subjected to region merging and character deduplication, and the subtitle recognition efficiency is improved. The deduplication and extraction method applied to the video continuous frame subtitle sequence is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of deduplication and extraction of subtitle sequences of continuous video frames, and in particular to a deduplication and extraction method applied to subtitle sequences of continuous video frames. Background Art

[0002] As we enter the era of universal Internet and universal short videos, massive amounts of video data will be generated every day. In order to analyze these video data, OCR technology is widely used to identify text in videos and convert visual content into editable text.

[0003] However, there are some challenges when dealing with subtitles, such as:

[0004] Due to the characteristics of the video, a large amount of repetitive text will appear in the recognition results;

[0005] In the era of short videos, there are various video styles, and subtitles do not only appear in fixed positions, but also appear in various positions of the video frame;

[0006] Merge scrolling subtitles in video.

[0007] Therefore, those skilled in the art provide a deduplication and extraction method for subtitle sequences of continuous video frames to solve the problems raised in the above background technology. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a deduplication and extraction method for subtitle sequences of continuous video frames.

[0009] include

[0010] Space coordinate projection merging, coordinate projection merging is mainly responsible for merging each frame with the same coordinate area. Generally, the subtitles in the video will appear in a fixed area, but the length and height of each subtitle will be different. In order to merge the subtitles in a certain area into a complete and smooth text, space coordinate projection merging is required. First, create a projection empty array to record the subtitle position information, traverse the frame list and traverse the subtitle array of each frame to determine the position;

[0011] After repeating the subtitle merging and obtaining the projection coordinate array of the entire video, theoretically each subtitle should fall into a certain projection area, and continue a round of cycles to remove duplicates and merge each block of text to form a complete text segment for each block.

[0012] Preferably, the method for merging the spatial coordinate projections is as follows:

[0013] S1-1: Check whether the frame sequence is empty;

[0014] S1-2: When the frame sequence is empty, it ends;

[0015] S1-3: When the frame sequence is not empty, get the frame subtitle array;

[0016] S1-4: Determine whether the subtitle array is empty;

[0017] S1-5: When the subtitle array is empty, return to S1-1;

[0018] S1-6: When the subtitle array is not empty, obtain the subtitle position information;

[0019] S1-7: Determine whether the projection array is empty;

[0020] S1-8: When the projection array is empty, add the subtitle position to the projection array and return to S1-4;

[0021] S1-9: When the projection array is not empty, obtain the projection position information;

[0022] S1-10: Compare the projection position and subtitle position;

[0023] S1-11: Determine whether the projection position includes the subtitle position;

[0024] S1-12: When the projection position includes the subtitle position, jump out of the loop to process the next subtitle and return to S1-4;

[0025] S1-13: When the projection position does not include the subtitle position, determine whether the projection position intersects with the subtitle position;

[0026] S1-14: When the projection position does not intersect with the subtitle position, add the subtitle position to the projection array and return to S1-4;

[0027] S1-15: When the projection position intersects with the subtitle position, determining whether the intersection of the projection position area and the subtitle area exceeds 80%;

[0028] S1-16: When the projection position area intersects with the subtitle area by more than 80%, the projection coordinates are updated to the coordinates of the rectangle after the projection rectangle and the subtitle rectangle are combined, and the loop is jumped out to process the next subtitle, and then returns to S1-4;

[0029] S1-17: When the intersection of the projection position area and the subtitle area does not exceed 80%, add the subtitle position to the projection array and return to S1-4;

[0030] S1-18: Form a projection coordinate array.

[0031] Preferably: the method for deduplicating subtitles is as follows:

[0032] S2-1: Check whether the frame sequence is empty;

[0033] S2-2: When the frame sequence is empty, it ends;

[0034] S2-3: When the frame sequence is not empty, obtain the frame subtitle array;

[0035] S2-4: Determine whether the subtitle array is empty;

[0036] S2-5: When the subtitle array is empty, return to S2-1;

[0037] S2-6: When the subtitle array is not empty, obtain the subtitle position information;

[0038] S2-7: Determine whether the projection array is empty;

[0039] S2-8: When the projection array is empty, return to S2-4;

[0040] S2-9: When the projection array is not empty, obtain the projection position information, and compare the projection position with the subtitle position;

[0041] S2-10: Determine whether the projection position includes the subtitle position;

[0042] S2-11: When the projection position includes the subtitle position, merge the subtitle into the projection subtitle, and then return to S2-4;

[0043] S2-12: When the projection position does not include the subtitle position, directly return to S2-4.

[0044] Preferably: the method for merging subtitles is as follows;

[0045] S3-1: Get the projection string x, the length is l;

[0046] S3-2: Get subtitle string y, length is k;

[0047] S3-3: Determine whether l is greater than 0;

[0048] S3-4: When l is not greater than 0, set the moving length of x to l 1, and determine whether l 1 / k is greater than 0.9;

[0049] S3-5: When l 1 / k is greater than 0.9, it means that the projected string basically contains the subtitles, and the process ends;

[0050] S3-6: When l 1 / k is not greater than 0.9, combine different character strings into the projection character string, x+=y[l 1:k], and end.

[0051] S3-7: When l is greater than 0, determine whether k is greater than 0;

[0052] S3-8: When l > 0 and k <= 0, return to S3-4 again;

[0053] S3-9: When l > 0 and k > 0, compare whether the elements in x[l-1] and y[k-1] are equal;

[0054] S3-10: When the elements in x[l-1] and y[k-1] are equal, calculate according to the formulas l = l - 1 and k = k - 1, and then return to S3-3 again;

[0055] S3-11: When the elements in x[l-1] and y[k-1] are not equal, calculate according to the formula k = k - 1, and then return to S3-7.

[0056] This solution uses an algorithm to remove duplicates and integrate the subtitle sequences based on consecutive video frames, enabling it to output non-repeated subtitle text segments in the entire video. Through OCR processing of each frame of the video, the subtitles in each frame of the video can be converted into a structured object sequence. In this sequence, it contains the subtitle string and position information. Each frame of the image will generate an object array, for example, [{ "text": "text", "addr": { "lefttop": [0, 100], "rightbottom": [100, 200]}},...]. Each object element represents the subtitle information recognized by OCR technology, including the text and the coordinate information of the two-dimensional plane where the text is located in the image. The entire video will form a two-dimensional array structure, and the above data structure is the input data of this method.

[0057] Preferably, the input needs to include the subtitles and subtitle position information of each frame.

[0058] Preferably, the spatial coordinate projection merging includes calculating the intersection ratio of the area of the rectangle formed by the subtitles in each frame and the projection rectangle of the previous subtitles in the projection plane. If there is no intersection, it is a new subtitle projection and is added to the projection subtitle array. If the intersection is greater than 80%, it is considered subtitles in the same area. For the new rectangle coordinates, the x value of the upper left point takes the minimum of the two, and the y value takes the minimum of the two. The x value of the lower right point of the new rectangle coordinates takes the maximum of the two, and the y value takes the maximum of the two. Update the subtitle position in the projection array with the new rectangle coordinates determined by the upper left and lower right.

[0059] Preferably, the function of the spatial coordinate projection merging is to merge the subtitle projections in all frames and reduce the two-dimensional array to a one-dimensional array.

[0060] Preferably, for duplicate subtitle merging, when dealing with scrolling subtitles, according to our reading habit from left to right, the scrolling of the subtitles is from right to left, so character comparison starts from the end of the string.

[0061] Preferably, the subtitles are merged repeatedly, and when the subtitles in the projection area are empty, the subtitles are directly added. After the subtitles are compared, the moving distance is skipped and the character string after the subtitles is added after the subtitles in the projection area.

[0062] Preferably, the subtitle merging is repeated, and considering the accuracy of the OCR recognition technology, 10% of the characters are allowed to have recognition errors without affecting the subtitle merging.

[0063] Technical effects and advantages of the present invention:

[0064] The spatial coordinate projection merging technology and repeated subtitle merging technology provided in the present invention form the subtitles in the video into a readable non-repetitive text so as to enable rapid analysis of these videos. The present invention can perform regional merging and character deduplication on the subtitle recognition data of a continuous frame sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a subtitle region merging flow chart of a method for deduplication and extraction of subtitle sequences of continuous video frames provided in an embodiment of the present application;

[0066] Figure 2 It is a subtitle deduplication flow chart of a deduplication and extraction method for a subtitle sequence of continuous video frames provided in an embodiment of the present application;

[0067] Figure 3 It is a subtitle merging flow chart of a method for deduplication and extraction of subtitle sequences of continuous video frames provided in an embodiment of the present application; DETAILED DESCRIPTION

[0068] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0069] Example 1

[0070] See also Figures 1 to 3 In this embodiment, a deduplication and extraction method for subtitle sequence of continuous frames of a video is provided, comprising:

[0071] Space coordinate projection merging, coordinate projection merging is mainly responsible for merging each frame with the same coordinate area. Generally, the subtitles in the video will appear in a fixed area, but the length and height of each subtitle will be different. In order to merge the subtitles in a certain area into a complete and smooth text, space coordinate projection merging is required. First, create a projection empty array to record the subtitle position information, traverse the frame list and traverse the subtitle array of each frame to determine the position;

[0072] After repeating the subtitle merging and obtaining the projection coordinate array of the entire video, theoretically each subtitle should fall into a certain projection area, and continue a round of cycles to remove duplicates and merge each block of text to form a complete text segment for each block.

[0073] Preferably, the method for merging the spatial coordinate projections is as follows:

[0074] S1-1: Check whether the frame sequence is empty;

[0075] S1-2: When the frame sequence is empty, it ends;

[0076] S1-3: When the frame sequence is not empty, get the frame subtitle array;

[0077] S1-4: Determine whether the subtitle array is empty;

[0078] S1-5: When the subtitle array is empty, return to S1-1;

[0079] S1-6: When the subtitle array is not empty, obtain the subtitle position information;

[0080] S1-7: Determine whether the projection array is empty;

[0081] S1-8: When the projection array is empty, add the subtitle position to the projection array and return to S1-4;

[0082] S1-9: When the projection array is not empty, obtain the projection position information;

[0083] S1-10: Compare the projection position and subtitle position;

[0084] S1-11: Determine whether the projection position includes the subtitle position;

[0085] S1-12: When the projection position includes the subtitle position, jump out of the loop to process the next subtitle and return to S1-4;

[0086] S1-13: When the projection position does not include the subtitle position, determine whether the projection position intersects with the subtitle position;

[0087] S1-14: When the projection position does not intersect with the subtitle position, add the subtitle position to the projection array and return to S1-4;

[0088] S1-15: When the projection position intersects with the subtitle position, determining whether the intersection of the projection position area and the subtitle area exceeds 80%;

[0089] S1-16: When the projection position area intersects with the subtitle area by more than 80%, the projection coordinates are updated to the coordinates of the rectangle after the projection rectangle and the subtitle rectangle are combined, and the loop is jumped out to process the next subtitle, and then returns to S1-4;

[0090] S1-17: When the intersection of the projection position area and the subtitle area does not exceed 80%, add the subtitle position to the projection array and return to S1-4;

[0091] S1-18: Form a projection coordinate array.

[0092] Preferably, the method for deduplicating subtitles is as follows:

[0093] S2-1: Check whether the frame sequence is empty;

[0094] S2-2: When the frame sequence is empty, it ends;

[0095] S2-3: When the frame sequence is not empty, obtain the frame subtitle array;

[0096] S2-4: Determine whether the subtitle array is empty;

[0097] S2-5: When the subtitle array is empty, return to S2-1;

[0098] S2-6: When the subtitle array is not empty, obtain the subtitle position information;

[0099] S2-7: Determine whether the projection array is empty;

[0100] S2-8: When the projection array is empty, return to S2-4;

[0101] S2-9: When the projection array is not empty, obtain the projection position information, and compare the projection position with the subtitle position;

[0102] S2-10: Determine whether the projection position includes the subtitle position;

[0103] S2-11: When the projection position includes the subtitle position, merge the subtitle into the projection subtitle, and then return to S2-4;

[0104] S2-12: When the projection position does not include the subtitle position, directly return to S2-4.

[0105] Preferably: the method for merging subtitles is as follows;

[0106] S3-1: Get the projection string x, the length is l;

[0107] S3-2: Get subtitle string y, length is k;

[0108] S3-3: Determine whether l is greater than 0;

[0109] S3-4: When l is not greater than 0, set the moving length of x to l 1, and determine whether l 1 / k is greater than 0.9;

[0110] S3-5: When l 1 / k is greater than 0.9, it means that the projected string basically contains the subtitles, and the process ends;

[0111] S3-6: When l 1 / k is not greater than 0.9, combine different character strings into the projection character string, x+=y[l 1:k], and end.

[0112] S3-7: When l is greater than 0, determine whether k is greater than 0;

[0113] S3-8: When l is greater than 0 and k is not greater than 0, return to S3-4;

[0114] S3-9: When l is greater than 0 and k is greater than 0, compare whether the elements in x[l-1] and y[k-1] are equal;

[0115] S3-10: When the elements in x[l-1] and y[k-1] are equal, calculate according to the formula l=l-1, k=k-1, and then return to S3-3;

[0116] S3-11: When the elements in x[l-1] and y[k-1] are not equal, calculate according to the formula k=k-1, and then return to S3-7.

[0117] When using the present invention, firstly, spatial coordinates are projected and merged: characters are input, whether the frame sequence is empty is checked, whether the subtitle array is empty is determined, projection position information is obtained, the projection position is compared with the subtitle position, whether the projection position includes the subtitle position, whether the projection position and the subtitle position intersect, and the projection coordinates are updated to the rectangular coordinates after the projection rectangle and the subtitle rectangle are merged to form a projection coordinate array;

[0118] De-duplicate the subtitles: check whether the frame sequence is empty, whether the subtitle array is empty, whether the projection array is empty, and whether the projection position contains the subtitle position;

[0119] Merge subtitles: Get the projected string x, length l; get the subtitle string y, length k, and determine whether l is greater than 0; when l is not greater than 0, set the x moving length to l 1, and determine whether l 1 / k is greater than 0.9; when l 1 / k is greater than 0.9, it means that the projected string basically contains the subtitles, and the end; when l 1 / k is not greater than 0.9, combine different strings into the projected string, x+=y[l 1:k], end; when l is greater than 0, determine whether k is greater than 0; when l is greater than 0 and k is not greater than 0, return to S3-4; when l is greater than 0 and k is greater than 0, compare whether the elements in x[l-1] and y[k-1] are equal; when the elements in x[l-1] and y[k-1] are equal, calculate according to the formula l=l-1, k=k-1, and then return to S3-3; when the elements in x[l-1] and y[k-1] are not equal, calculate according to the formula k=k-1, and then return to S3-7.

[0120] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention are implemented according to the conventional means in the field unless otherwise specified and limited.

Claims

1. A method for deduplication and extraction of subtitle sequences of continuous video frames, comprising: Space coordinate projection merging is responsible for merging the same coordinate area of ​​each frame, merging the subtitle projections in all frames, and reducing the two-dimensional array to a one-dimensional array; Repeat the subtitle merging to obtain the projection coordinate array of the entire video, then remove the duplicate text in each block and merge it to form a complete text segment for each block.

2. The method for deduplication and extraction of subtitle sequences of continuous video frames according to claim 1, characterized in that: The method for merging the spatial coordinate projections is as follows: S1-1: Check whether the frame sequence is empty; S1-2: When the frame sequence is empty, it ends; S1-3: When the frame sequence is not empty, get the frame subtitle array; S1-4: Determine whether the subtitle array is empty; S1-5: When the subtitle array is empty, return to S1-1; S1-6: When the subtitle array is not empty, obtain the subtitle position information; S1-7: Determine whether the projection array is empty; S1-8: When the projection array is empty, add the subtitle position to the projection array and return to S1-4; S1-9: When the projection array is not empty, obtain the projection position information; S1-10: Compare the projection position and subtitle position; S1-11: Determine whether the projection position includes the subtitle position; S1-12: When the projection position includes the subtitle position, jump out of the loop to process the next subtitle and return to S1-4; S1-13: When the projection position does not include the subtitle position, determine whether the projection position intersects with the subtitle position; S1-14: When the projection position does not intersect with the subtitle position, add the subtitle position to the projection array and return to S1-4; S1-15: When the projection position intersects with the subtitle position, determining whether the intersection of the projection position area and the subtitle area exceeds 80%; S1-16: When the projection position area intersects with the subtitle area by more than 80%, the projection coordinates are updated to the coordinates of the rectangle after the projection rectangle and the subtitle rectangle are combined, and the loop is jumped out to process the next subtitle, and then returns to S1-4; S1-17: When the intersection of the projection position area and the subtitle area does not exceed 80%, add the subtitle position to the projection array and return to S1-4; S1-18: Form a projection coordinate array.

3. The method for deduplication and extraction of subtitle sequences of continuous video frames according to claim 1, characterized in that: The method for deduplicating subtitles is as follows: S2-1: Check whether the frame sequence is empty; S2-2: When the frame sequence is empty, it ends; S2-3: When the frame sequence is not empty, obtain the frame subtitle array; S2-4: Determine whether the subtitle array is empty; S2-5: When the subtitle array is empty, return to S2-1; S2-6: When the subtitle array is not empty, obtain the subtitle position information; S2-7: Determine whether the projection array is empty; S2-8: When the projection array is empty, return to S2-4; S2-9: When the projection array is not empty, obtain the projection position information, and compare the projection position with the subtitle position; S2-10: Determine whether the projection position includes the subtitle position; S2-11: When the projection position includes the subtitle position, merge the subtitle into the projection subtitle, and then return to S2-4; S2-12: When the projection position does not include the subtitle position, directly return to S2-4.

4. The method for deduplication and extraction of subtitle sequences of continuous video frames according to claim 3, characterized in that: The method of subtitle merging is as follows: S3-1: Get the projection string x, the length is l; S3-2: Get subtitle string y, length is k; S3-3: Determine whether l is greater than 0; S3-4: When l is not greater than 0, set the moving length of x to l 1, and determine whether l 1 / k is greater than 0.9; S3-5: When l 1 / k is greater than 0.9, it means that the projected string basically contains the subtitles, and the process ends; S3-6: When l 1 / k is not greater than 0.9, combine different character strings into the projection character string, x+=y[l 1:k], and end; S3-7: When l is greater than 0, determine whether k is greater than 0; S3-8: When l is greater than 0 and k is not greater than 0, return to S3-4; S3-9: When l is greater than 0 and k is greater than 0, compare whether the elements in x[l-1] and y[k-1] are equal; S3-10: When the elements in x[l-1] and y[k-1] are equal, calculate according to the formula l=l-1, k=k-1, and then return to S3-3; S3-11: When the elements in x[l-1] and y[k-1] are not equal, calculate according to the formula k=k-1, and then return to S3-7.

5. The method for deduplication and extraction of subtitle sequences of continuous video frames according to claim 2, characterized in that: Calculate the intersection ratio between the area of ​​the rectangle formed by the subtitles in each frame and the projection rectangle of the previous subtitles in the projection plane. If there is no intersection, it is a new subtitle projection and added to the projection subtitle array. If the intersection is greater than 80%, it is considered to be subtitles in the same area. The new rectangle coordinates take the minimum value of the upper left point x and the minimum value of the y. The new rectangle coordinates take the maximum value of the lower right point x and the maximum value of the y. Update the subtitle position in the projection array with the new rectangle coordinates determined by the upper left and lower right.

6. The method for deduplication and extraction of subtitle sequences of continuous video frames according to claim 4, characterized in that: In the subtitle merging process, character comparison is started from the end of the character string.

7. The method for deduplication and extraction of subtitle sequences of continuous video frames according to claim 4, characterized in that: When the subtitles in the projection area are empty, the subtitles are directly appended. After the subtitles are compared, the moving distance is skipped and the string after the subtitles is appended to the subtitles in the projection area.