Method for detecting video caption and related product

Abstract

The application discloses a video subtitle detection method and related products. The method comprises the following steps: acquiring at least one first text in a to-be-detected video and a position of the at least one first text in the to-be-detected video; dividing the at least one first text to obtain at least one text class by classifying the first texts with the same position in the to-be-detected video; determining a text class with a quantity of first texts contained in the at least one text class greater than or equal to a first threshold value as a target text class; and taking the first texts in the target text class as subtitles of the to-be-detected video.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for detecting video subtitles and related products. Background Technology

[0002] With the explosive growth of various videos, video management has become increasingly difficult. However, when subtitles are present in a video, they are usually related to the video content. Therefore, subtitles can be used as a basis for video management, thereby reducing the difficulty of video management. For example, video recommendations or searches can be based on subtitles. Therefore, extracting subtitle text from videos with subtitles is of great importance. Current methods involve performing optical character recognition (OCR) on video frames to obtain the text within the frames, and then manually judging whether the obtained text contains subtitles. This method has low subtitle detection efficiency. Summary of the Invention

[0003] This application provides a method for detecting video subtitles and related products.

[0004] Firstly, a method for detecting video subtitles is provided, the method comprising:

[0005] Obtain at least one first text in the video to be detected and the position of the at least one first text in the video to be detected;

[0006] By grouping the first texts that are located in the same position in the video to be detected into one category, at least one text class is obtained by dividing the at least one first text.

[0007] The text class containing a number of first texts greater than or equal to a first threshold is identified as the target text class.

[0008] The first text in the target text class is used as the subtitle of the video to be detected.

[0009] In conjunction with any embodiment of this application, after obtaining at least one first text in the video to be detected and the position of the at least one first text in the video to be detected, and before dividing the at least one first text into at least one text class by grouping the first texts that are located in the same position in the video to be detected into one class, the method further includes:

[0010] If the number of first texts is greater than 1, the intersection-union ratio of the areas of any two first texts among the at least one first text is determined based on the position of the at least one first text in the video to be detected;

[0011] If the cross-union ratio is greater than or equal to the second threshold, it is determined that the two first texts corresponding to the cross-union ratio are in the same position in the video to be detected.

[0012] In any embodiment of this application, the pixel coordinate system of the video to be detected includes a horizontal axis and a vertical axis, wherein the direction of the horizontal axis is horizontal and the direction of the vertical axis is vertical.

[0013] After determining the intersection-union ratio of the areas of any two first texts in the at least one first text, the method further includes:

[0014] When the intersection-union ratio is less than the second threshold, based on the positions of the two first texts corresponding to the intersection-union ratio in the video to be detected, the center-horizontal distance between the centers of the two first texts corresponding to the intersection-union ratio in the horizontal direction is determined, as well as the center-vertical distance between the centers of the two first texts corresponding to the intersection-union ratio in the vertical direction is determined, as well as the maximum horizontal distance between the two first texts corresponding to the intersection-union ratio in the horizontal direction is determined, as well as the maximum vertical distance between the two first texts corresponding to the intersection-union ratio in the vertical direction is determined;

[0015] If the ratio of the center horizontal distance to the maximum horizontal distance is greater than or equal to a third threshold, and the ratio of the center vertical distance to the maximum vertical distance is greater than or equal to the third threshold, then the two first texts corresponding to the intersection-union ratio are determined to be in the same position in the video to be detected.

[0016] In conjunction with any embodiment of this application, obtaining at least one first text in the video to be detected includes:

[0017] Obtain at least one second text in the video to be detected and the duration of the at least one second text in the video to be detected;

[0018] The at least one first text is obtained by selecting texts whose duration is within a preset range from the at least one second text.

[0019] In conjunction with any embodiment of this application, obtaining at least one second text in the video to be detected includes:

[0020] Acquire the video to be detected;

[0021] The video to be detected is subjected to text detection to obtain at least one third text.

[0022] Merge the identical third texts in the at least one third text to obtain the at least one second text.

[0023] In conjunction with any embodiment of this application, obtaining the duration of the at least one second text in the video to be detected includes:

[0024] Use the timestamp of the video frame corresponding to the third text as the timestamp of the third text;

[0025] Determine the minimum timestamp of the third text corresponding to the second text as the start time of the second text;

[0026] The maximum timestamp of the third text corresponding to the second text is determined as the end time of the second text;

[0027] The duration of the at least one second text in the video to be detected is obtained based on the start time and end time of the at least one second text.

[0028] In conjunction with any embodiment of this application, the step of merging the identical third texts in the at least one third text to obtain the at least one second text includes:

[0029] Merge the identical third texts in the at least one third text to obtain the at least one fourth text;

[0030] If the number of fourth texts is greater than 1, and there are fifth and sixth texts in the at least one fourth text with a similarity greater than or equal to a fourth threshold, then the fifth and sixth texts in the at least one fourth text are merged to obtain the at least one second text;

[0031] If there are no two fourth texts in the at least one fourth text that have a similarity greater than or equal to a fourth threshold, then the at least one fourth text shall be regarded as the at least one second text.

[0032] In conjunction with any embodiment of this application, the step of merging the fifth and sixth texts in the at least one fourth text to obtain the at least one second text includes:

[0033] Obtain the start time of the fifth text, the end time of the fifth text, the start time of the sixth text, and the end time of the sixth text, wherein the start time of the fifth text is earlier than the start time of the sixth text;

[0034] Determine the first time difference between the start time of the fifth text and the end time of the sixth text;

[0035] The median time of the fifth text is obtained by determining the median time of the fifth text by taking the median start time and the median end time of the fifth text.

[0036] The median time of the sixth text is obtained by determining the median time of the sixth text by taking the median start time and the median end time of the sixth text.

[0037] Determine a second time difference between the median time of the fifth text and the median time of the sixth text;

[0038] If the ratio of the first time difference to the second time difference is less than or equal to the fifth threshold, the fifth text and the sixth text in the at least one fourth text are merged to obtain the at least one second text.

[0039] Secondly, a video subtitle detection device is provided, the detection device comprising:

[0040] An acquisition unit is configured to acquire at least one first text in a video to be detected and the position of the at least one first text in the video to be detected;

[0041] The segmentation unit is used to segment the at least one first text by grouping the first texts that are located in the same position in the video to be detected into one category, thereby obtaining at least one text class;

[0042] A determining unit is configured to determine a text class whose number of first texts contained in the at least one text class is greater than or equal to a first threshold as a target text class;

[0043] The processing unit is used to use the first text in the target text class as the subtitle of the video to be detected.

[0044] The determining unit is further configured to:

[0045] If the number of first texts is greater than 1, the intersection-union ratio of the areas of any two first texts among the at least one first text is determined based on the position of the at least one first text in the video to be detected;

[0046] If the cross-union ratio is greater than or equal to the second threshold, it is determined that the two first texts corresponding to the cross-union ratio are in the same position in the video to be detected.

[0047] In any embodiment of this application, the pixel coordinate system of the video to be detected includes a horizontal axis and a vertical axis, wherein the direction of the horizontal axis is horizontal and the direction of the vertical axis is vertical.

[0048] The determining unit is further configured to:

[0049] When the intersection-union ratio is less than the second threshold, based on the positions of the two first texts corresponding to the intersection-union ratio in the video to be detected, the center-horizontal distance between the centers of the two first texts corresponding to the intersection-union ratio in the horizontal direction is determined, as well as the center-vertical distance between the centers of the two first texts corresponding to the intersection-union ratio in the vertical direction is determined, as well as the maximum horizontal distance between the two first texts corresponding to the intersection-union ratio in the horizontal direction is determined, as well as the maximum vertical distance between the two first texts corresponding to the intersection-union ratio in the vertical direction is determined;

[0050] If the ratio of the center horizontal distance to the maximum horizontal distance is greater than or equal to a third threshold, and the ratio of the center vertical distance to the maximum vertical distance is greater than or equal to the third threshold, then the two first texts corresponding to the intersection-union ratio are determined to be in the same position in the video to be detected.

[0051] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0052] Obtain at least one second text in the video to be detected and the duration of the at least one second text in the video to be detected;

[0053] The at least one first text is obtained by selecting texts whose duration is within a preset range from the at least one second text.

[0054] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0055] Acquire the video to be detected;

[0056] The video to be detected is subjected to text detection to obtain at least one third text.

[0057] Merge the identical third texts in the at least one third text to obtain the at least one second text.

[0058] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0059] Use the timestamp of the video frame corresponding to the third text as the timestamp of the third text;

[0060] Determine the minimum timestamp of the third text corresponding to the second text as the start time of the second text;

[0061] The maximum timestamp of the third text corresponding to the second text is determined as the end time of the second text;

[0062] The duration of the at least one second text in the video to be detected is obtained based on the start time and end time of the at least one second text.

[0063] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0064] Merge the identical third texts in the at least one third text to obtain the at least one fourth text;

[0065] If the number of fourth texts is greater than 1, and there are fifth and sixth texts in the at least one fourth text with a similarity greater than or equal to a fourth threshold, then the fifth and sixth texts in the at least one fourth text are merged to obtain the at least one second text;

[0066] If there are no two fourth texts in the at least one fourth text that have a similarity greater than or equal to a fourth threshold, then the at least one fourth text shall be regarded as the at least one second text.

[0067] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0068] Obtain the start time of the fifth text, the end time of the fifth text, the start time of the sixth text, and the end time of the sixth text, wherein the start time of the fifth text is earlier than the start time of the sixth text;

[0069] Determine the first time difference between the start time of the fifth text and the end time of the sixth text;

[0070] The median time of the fifth text is obtained by determining the median time of the fifth text by taking the median start time and the median end time of the fifth text.

[0071] The median time of the sixth text is obtained by determining the median time of the sixth text by taking the median start time and the median end time of the sixth text.

[0072] Determine a second time difference between the median time of the fifth text and the median time of the sixth text;

[0073] If the ratio of the first time difference to the second time difference is less than or equal to the fifth threshold, the fifth text and the sixth text in the at least one fourth text are merged to obtain the at least one second text.

[0074] Thirdly, an electronic device is provided, characterized in that it comprises: a processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0075] Fourthly, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs as described in the first aspect and any of its embodiments.

[0076] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform the first aspect and any of its embodiments described above.

[0077] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions that, when the computer program or instructions are executed on a computer, cause the computer to perform the first aspect described above and any of its embodiments.

[0078] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0079] In this application, after acquiring at least one first text in a video to be detected and its position within the video, the detection device divides the at least one first text into at least one text class by grouping first texts with the same position in the video into one category. Then, by identifying target text classes from the at least one text class that contain a number of first texts greater than or equal to a first threshold, the device determines the target text class whose corresponding position is the location where the subtitle appears. Finally, the first text in the target text class is used as the subtitle of the video to be detected, thereby detecting subtitles in the video and improving the efficiency of subtitle detection. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0081] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0082] Figure 1 A flowchart illustrating a video subtitle detection method provided in an embodiment of this application;

[0083] Figure 2 A flowchart illustrating another video subtitle detection method provided in this application embodiment;

[0084] Figure 3 A flowchart illustrating the process of removing duplicate fields provided in this application embodiment;

[0085] Figure 4 A schematic diagram of a clustering process provided for an embodiment of this application;

[0086] Figure 5 A schematic diagram of the structure of a video subtitle detection device provided in an embodiment of this application;

[0087] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0088] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0089] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0090] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0091] The execution subject of this application embodiment is a video subtitle detection device (hereinafter referred to as the detection device), wherein the detection device can be any electronic device capable of executing the technical solution disclosed in the method embodiment of this application. Optionally, the detection device can be one of the following: a computer, a server.

[0092] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a video subtitle detection method provided in an embodiment of this application.

[0093] 101. Obtain at least one first text in the video to be detected and the position of the at least one first text in the video to be detected.

[0094] In this embodiment, the video to be detected can be any video. The video to be detected can be an offline video or an online video. Offline videos can be videos acquired through a camera or mobile smart device. Online videos can be videos acquired in real-time by a camera. The video to be detected can be a video containing any content; for example, it could be a movie clip, an instructional video, or a sports competition video.

[0095] In this embodiment, the number of first texts is one or more, and all first texts are texts in the video to be detected. For example, the video to be detected includes image a and image b, where image a includes text c and image b includes text d. Then text c and text d are both first texts, that is, at least one first text includes text c and text d.

[0096] At least one location of the first text in the video to be detected, including the location of each first text in the video to be detected. In one possible implementation, the location of the first text in the video to be detected is the location of the text box containing the first text in the video to be detected. For example, by performing OCR on the video to be detected, the text box containing the first text can be determined from the video to be detected, and the location of the first text in the video to be detected is the location of the text box containing the first text in the video to be detected.

[0097] Optionally, the position in the video to be detected is the position in the pixel coordinate system of the video to be detected. In this case, the position of the first text in the video to be detected is the position of the first text in the pixel coordinate system to be detected.

[0098] In one implementation of acquiring at least one first text, the detection device receives at least one first text input by a user through an input component. The input component includes at least one of the following: a keyboard, a mouse, a touchscreen, a touchpad, or an audio input device.

[0099] In another implementation of acquiring at least one first text, the detection device receives at least one first text sent by a terminal. The terminal can be any of the following: a mobile phone, a computer, a tablet computer, or a server.

[0100] In another implementation of obtaining at least one first text, the detection device, upon obtaining the video to be detected, extracts at least one first text from the video by performing OCR on the video to be detected.

[0101] In one implementation of obtaining the position of at least one first text in a video to be detected, the detection device receives the position of at least one first text in the video to be detected, input by a user through an input component.

[0102] In another implementation of obtaining the position of at least one first text in the video to be detected, the detection device receives the position of at least one first text in the video to be detected sent by the terminal.

[0103] In another implementation of obtaining the position of at least one first text in a video to be detected, the detection device, upon obtaining the video to be detected, extracts at least one first text from the video to be detected by performing OCR on the video to be detected, and determines the position of at least one first text in the video to be detected.

[0104] It should be understood that the steps of the detection device acquiring at least one first text in the video to be detected and acquiring the position of at least one first text in the video to be detected can be performed separately or simultaneously, and this application does not limit this.

[0105] 102. By grouping the first texts that are in the same position in the above-mentioned video to be detected into one category, the above-mentioned at least one first text is divided to obtain at least one text class.

[0106] In step 102, the detection device classifies at least one text based on the position of the first text in the video to be detected. Specifically, first texts with the same position in the video to be detected are grouped into the same category. In this way, by classifying at least one first text, at least one text class can be obtained. The number of text classes is one or more, and each text class includes one or more first texts.

[0107] Optionally, two first texts being in the same position in the video to be detected means that the distance between the two first texts in the video to be detected is less than or equal to a sixth threshold. For example, if the sixth threshold is 2, based on the positions of first text 'a' and first text 'b' in the video to be detected, the distance between first text 'a' and first text 'b' in the video to be detected is determined to be 'd'. Then, if 'd' is less than or equal to 2, the positions of first text 'a' and first text 'b' in the video to be detected are determined to be the same; if 'd' is greater than 2, the positions of first text 'a' and first text 'b' in the video to be detected are determined to be different.

[0108] 103. Determine the target text class as the text class in which the number of first texts contained in at least one of the above text classes is greater than or equal to the first threshold.

[0109] Subtitles in a video usually appear in the same position, or different subtitles appear in close positions. Moreover, the time interval between two adjacent frames in a video is short, while the appearance of subtitles usually lasts for a period of time (such as 3 to 15 seconds). That is, subtitles usually appear in multiple video frames. Therefore, the number of texts located in the positions where subtitles appear in the video should be large, and conversely, the number of texts not located in the positions where subtitles appear in the video should be small.

[0110] By executing step 102, the detection device has classified at least one first text located in the video to be detected into at least one text class according to its position. Therefore, the detection device can determine whether the position corresponding to the text class is the position where the subtitle appears based on the number of first texts in the text class.

[0111] In this embodiment, the detection device uses a first threshold to determine whether the number of first texts in a text class is large or small, thereby determining whether the position corresponding to the text class is the location where subtitles appear. Specifically, if the number of first texts in a text class is less than the first threshold, it indicates that the number of first texts in the text class is small, thus determining that the position corresponding to the text class is not the location where subtitles appear. Conversely, if the number of first texts in a text class is greater than or equal to the first threshold, it indicates that the number of first texts in the text class is large, thus determining that the position corresponding to the text class is the location where subtitles appear. In this embodiment, the text class whose corresponding position is the location where subtitles appear is called the target text class.

[0112] For example, at least one text class includes text class a and text class b, where the number of first texts in text class a is 1, and the number of first texts in text class b is 40. If the first threshold is 5, then the number of first texts in text class a is less than the first threshold, meaning text class a is not the target text class, while the number of first texts in text class b is greater than the first threshold, meaning text class b is the target text class.

[0113] 104. Use the first text in the above target text class as the subtitle of the above video to be detected.

[0114] Since the target text class corresponds to the location where the subtitle appears, and the first text in the target text class is the subtitle, the detection device can use the first text in the target text class as the subtitle of the video to be detected.

[0115] In this embodiment, after acquiring at least one first text in the video to be detected and its position within the video, the detection device divides the at least one first text into at least one text class by grouping the first texts located at the same position in the video. Then, by identifying target text classes from the at least one text class that contain a number of first texts greater than or equal to a first threshold, the device determines the target text class whose corresponding position is the location where the subtitle appears. Finally, the first text in the target text class is used as the subtitle of the video to be detected, thereby detecting the subtitles in the video and improving the efficiency of subtitle detection.

[0116] As an optional implementation, the detection device further performs the following steps after performing step 101 and before performing step 102:

[0117] 201. When the number of first texts is greater than 1, the intersection over union (IoU) ratio of the areas of any two of the first texts is determined based on the position of the first text in the video to be detected.

[0118] In this embodiment of the application, the IoU of the area of ​​the two first texts is calculated as follows: the area of ​​the intersection of the pixel regions covered by the two first texts / the area of ​​the union of the pixel regions covered by the two first texts.

[0119] When the number of first texts is greater than one, the detection device can determine the pixel area covered by the first text in the video to be detected based on its position. Therefore, based on the position of each first text in the video to be detected, the pixel area covered by each individual first text can be determined. After determining the pixel areas covered by each individual first text in the video to be detected, the IoU (Intersection over Union) of the areas of any two first texts can be determined.

[0120] In one possible implementation, the position of the first text in the video to be detected is the position of the text box containing the first text in the video to be detected. Based on the positions of any two first texts in the video to be detected, the detection device can determine the intersection and union of the text boxes corresponding to these two first texts. Furthermore, it can obtain the Area of ​​Union (IoU) of the two first texts based on the area of ​​their intersection and the area of ​​their union.

[0121] Optionally, if the number of second texts is 1, the detection device may treat the second text as the first text.

[0122] 202. If the above IoU is greater than or equal to the second threshold, it is determined that the two first texts corresponding to the above IoU are in the same position in the above video to be detected.

[0123] A high IoU between two primary texts indicates a high degree of overlap between them in the video being tested, while a low IoU indicates a low degree of overlap. When two primary texts have a high degree of overlap, it can be determined that they are located in the same position within the video.

[0124] In this embodiment, the detection device uses a second threshold to determine whether the IoU is large or small, thereby determining whether the two first texts corresponding to the IoU are in the same position in the video to be detected. Specifically, if the IoU is greater than or equal to the second threshold, it indicates that the IoU is large, thus determining that the two first texts corresponding to the IoU are in the same position in the video to be detected; conversely, if the IoU is less than the second threshold, it indicates that the IoU is small, thus determining that the overlap between the two first texts corresponding to the IoU is low.

[0125] In this embodiment, when the number of first texts is greater than 1, the detection device determines the Area of ​​Integration (IoU) of any two first texts within the at least one first text based on the position of the at least one first text in the video to be detected. Then, it can determine whether the positions of the two first texts in the video to be detected are the same based on the IoU. Specifically, if the IoU is greater than or equal to a second threshold, it is determined that the two first texts corresponding to the IoU are in the same position in the video to be detected, thereby improving the accuracy of determining that the positions of the two first texts in the video to be detected are the same.

[0126] When a subtitle contains a large amount of text, it may be displayed in at least two parts. For example, in the video being tested, if the subtitle contains a large amount of text, it might be displayed in two lines, top and bottom. Clearly, when a subtitle is divided into at least two parts, different content within the subtitle is displayed in different areas. This means that text in different areas belongs to the same subtitle, and these different areas together constitute the area for displaying the subtitle. Therefore, the text in different areas should be grouped into the same category; in other words, different text within the same subtitle should be considered to be in the same position within the video being tested.

[0127] When the subtitle text is displayed in at least two parts, the text in different regions may not overlap, resulting in low overlap and a small Intersection over Union (IoU). In other words, if the IoU of two first texts is small, determining that they are in different positions in the video to be detected may lead to misjudgment. Therefore, when the IoU of two first texts is less than a second threshold, the detection device can determine whether the positions of two first texts in the video to be detected are the same by performing the following optional implementation method.

[0128] As an optional implementation, the pixel coordinate system of the video to be detected includes a horizontal axis and a vertical axis, wherein the direction of the horizontal axis is horizontal and the direction of the vertical axis is vertical. After determining the intersection-union ratio of the areas of any two first texts in at least one first text, the detection device further performs the following steps:

[0129] 301. When the IoU is less than the second threshold, based on the positions of the two first texts corresponding to the IoU in the video to be detected, determine the center-to-center horizontal distance of the two first texts corresponding to the IoU in the horizontal direction, determine the center-to-center vertical distance of the two first texts corresponding to the IoU in the vertical direction, determine the maximum horizontal distance of the two first texts corresponding to the IoU in the horizontal direction, and determine the maximum vertical distance of the two first texts corresponding to the IoU in the vertical direction.

[0130] In this embodiment, the center of the first text can be the geometric center of the pixel area covered by the first text. Optionally, the center of the first text can be the geometric center of the text box containing the first text.

[0131] In this embodiment, the horizontal distance between the centers of two first texts is called the horizontal center distance. Specifically, the horizontal distance between the centers of two first texts is the difference in the horizontal coordinates of their centers. For example, if the horizontal coordinate of the center of first text a is 2 and the horizontal coordinate of the center of first text b is 5, then the difference in the horizontal coordinates of the centers of first text a and first text b is (5-2) = 3, meaning the horizontal center distance between first text a and first text b is 3.

[0132] In this embodiment, the vertical distance between the centers of two first texts is called the vertical center distance. Specifically, the vertical distance between the centers of two first texts is the difference in the distance between their ordinates. For example, if the ordinate of the center of first text a is 3 and the ordinate of the center of first text b is 8, then the difference in distance between the ordinates of the centers of first text a and first text b is (8-3) = 5, that is, the vertical center distance between first text a and first text b is 5.

[0133] In this embodiment, the maximum horizontal distance between two first texts is called the maximum horizontal distance. Specifically, the maximum horizontal distance between two first texts is the difference between the maximum and minimum horizontal coordinates of the two first texts. For example, the minimum horizontal coordinate of the pixel area covered by first text a is 2, and the maximum horizontal coordinate of the pixel area covered by first text a is 10. The minimum horizontal coordinate of the pixel area covered by first text b is 6, and the maximum horizontal coordinate of the pixel area covered by first text a is 13. In this case, the minimum horizontal coordinate of first text a and first text b is 2, and the maximum horizontal coordinate of first text a and first text b is 13. Therefore, the maximum horizontal distance between first text a and first text b is (13-2) = 9.

[0134] In this embodiment, the maximum vertical distance between two first texts is called the maximum vertical distance. Specifically, the maximum vertical distance between two first texts is the difference between the maximum and minimum ordinates of the two first texts. For example, the minimum ordinate of the pixel area covered by first text a is 7, the maximum ordinate of the pixel area covered by first text a is 19, the minimum ordinate of the pixel area covered by first text b is 13, and the maximum ordinate of the pixel area covered by first text a is 20. In this case, the minimum ordinate of first text a and first text b is 7, the maximum ordinate of first text a and first text b is 20, and the maximum vertical distance between first text a and first text b is (20-7) = 13.

[0135] 302. If the ratio of the horizontal distance of the center to the maximum horizontal distance is greater than or equal to the third threshold, and the ratio of the vertical distance of the center to the maximum vertical distance is greater than or equal to the third threshold, it is determined that the two first texts corresponding to the IoU are in the same position in the video to be detected.

[0136] Since the text of a subtitle is displayed in at least two parts, the distance between text in different areas is usually small, meaning that the overlap between text in different areas is high both horizontally and vertically. Therefore, based on the overlap between two first texts in the horizontal and vertical directions, it can be determined whether the positions of the two first texts in the video to be detected are the same.

[0137] The larger the ratio of the horizontal distance between the centers of two first texts to their maximum horizontal distance, the higher the degree of overlap between the two first texts in the horizontal direction. Similarly, the larger the ratio of the vertical distance between the centers of two first texts to their maximum vertical distance, the higher the degree of overlap between the two first texts in the vertical direction. Therefore, when both the ratio of the horizontal distance between the centers of two first texts to their maximum horizontal distance and the ratio of the vertical distance between the centers of two first texts to their maximum vertical distance are large, the detection device can determine that the two first texts are located in the same position in the video to be detected.

[0138] In this embodiment, the detection device uses a third threshold to determine whether the ratio of the horizontal distance between the centers of two first texts to the maximum horizontal distance between the two first texts is large or small. It can also use the third threshold to determine whether the ratio of the vertical distance between the centers of two first texts to the maximum vertical distance between the two first texts is large or small, thereby determining whether the two first texts are in the same position in the video to be detected. Specifically, if the ratio of the horizontal distance between the centers of two first texts to the maximum horizontal distance between the two first texts is greater than or equal to the third threshold, and the ratio of the vertical distance between the centers of two first texts to the maximum vertical distance between the two first texts is also greater than or equal to the third threshold, the detection device determines that the ratio of the vertical distance between the centers of two first texts to the maximum vertical distance between the two first texts is large, and thus determines that the two first texts are in the same position in the video to be detected.

[0139] In this embodiment, when the cross-union ratio (CUNR) is less than a second threshold, the detection device determines the center-horizontal distance, center-vertical distance, maximum horizontal distance, and maximum vertical distance of the two first texts corresponding to the CUNR in the video to be detected based on their positions in the video. Then, based on the center-horizontal distance, center-vertical distance, maximum horizontal distance, and maximum vertical distance of the two first texts corresponding to the CUNR, it determines that the two first texts corresponding to the CUNR are in the same position in the video to be detected, thereby improving the accuracy of determining that the two first texts are in the same position in the video to be detected.

[0140] As an optional implementation, the detection device acquires at least one first text in the video to be detected by performing the following steps:

[0141] 401. Obtain at least one second text in the video to be detected and the duration of the at least one second text in the video to be detected.

[0142] In this embodiment, the number of second texts is one or more, and all second texts are texts in the video to be detected. For example, the video to be detected includes image a and image b, where image a includes text c, and image b includes text d and text e. Then text c, text d, and text e are all second texts, that is, at least one second text includes text c, text d, and text e.

[0143] In this embodiment, the duration of the second text in the video to be detected refers to the duration of the second text appearing in the video to be detected. For example, if the second text first appears in the tenth frame of the video to be detected and last appears in the thirtieth frame, then the duration of the second text is the duration from the timestamp of the tenth frame to the timestamp of the thirtieth frame. The duration of at least one second text includes the duration of each individual second text.

[0144] In one implementation of acquiring at least one second text, the detection device receives at least one second text input by a user through an input component.

[0145] In another implementation of acquiring at least one second text, the detection device receives at least one second text sent by the terminal.

[0146] In another implementation of obtaining at least one second text, the detection device, upon obtaining the video to be detected, extracts at least one second text from the video by performing OCR on the video to be detected.

[0147] In one implementation of obtaining the duration of at least one second text, the detection device receives the duration of at least one second text input by a user through an input component.

[0148] In another implementation of obtaining the duration of at least one second text, the detection device receives the duration of at least one second text sent by the terminal.

[0149] It should be understood that the steps of the detection device acquiring at least one second text in the video to be detected and acquiring the duration of at least one second text can be performed separately or simultaneously, and this application does not limit this.

[0150] 402. Select a text whose duration is within a preset range from the above at least one second text to obtain the above at least one first text.

[0151] The text in a video includes both subtitles and non-subtitles. For example, the watermark in the video is also considered text. Therefore, it is necessary to filter the text in the video to identify the subtitle text.

[0152] As described in step 103, the appearance of subtitles in a video usually lasts for a period of time. Therefore, the duration can be used as a basis to filter out the text belonging to the subtitles from at least one second text in the video to be detected. Specifically, the detection device selects texts whose duration falls within a preset range from at least one second text to obtain the text belonging to the subtitles (i.e., at least one first text). Optionally, the preset range is 3 seconds to 15 seconds.

[0153] In this embodiment, after acquiring at least one second text in the video to be detected and the duration of the at least one second text in the video to be detected, the detection device selects a text whose duration is within a preset range from the at least one second text as at least one first text, which can increase the probability that at least one first text is a subtitle.

[0154] As an optional implementation, the detection device acquires at least one second text in the video to be detected by performing the following steps:

[0155] 501. Obtain the video to be tested.

[0156] In one implementation of acquiring the video to be tested, the detection device receives the video to be tested input by the user through an input component. In another implementation, the detection device receives the video to be tested sent by a terminal. In yet another implementation, the detection device obtains the video to be tested by downloading video from the internet. In yet yet another implementation, the detection device has a communication connection with a camera, and the camera acquires the video captured by the camera as the video to be tested through this communication connection.

[0157] 502. Perform OCR on the above-mentioned video to be detected to obtain at least one third-party text.

[0158] In this embodiment, the number of third texts is one or more. The detection device performs OCR on each video frame in the video to be detected to obtain at least one third text. That is, at least one third text includes the text in each video frame. For example, if the video to be detected includes a first frame and a second frame, the detection device obtains text a and text b by performing OCR on the first frame, and obtains text b and text c by performing OCR on the second frame. In this case, at least one third text includes text a, text b from the first frame, text b from the second frame, and text c. In other words, even if the text content in different video frames is the same, two different third texts can be obtained by performing OCR on the video to be detected.

[0159] It should be understood that by performing OCR on the video to be detected, not only can at least one third text be obtained, but also the timestamp of the third text and the position of the third text in the video to be detected. The timestamp of the third text is the timestamp of the video frame containing the third text.

[0160] 503. Merge the identical third texts in at least one of the above third texts to obtain at least one of the above second texts.

[0161] When there is only one third text, the detection device treats the third text as the second text. When there is more than one third text, the detection device obtains at least one second text by merging identical texts from all the third texts.

[0162] In this embodiment of the application, merging identical third texts means retaining the text information of two or more identical third texts, and determining the duration of the retained text information based on the timestamps of the two or more identical third texts to obtain the second text. The duration of the retained text information is the time difference between the minimum and maximum timestamps of the two or more identical third texts.

[0163] For example, third texts a, b, and c are three identical third texts, with timestamps t1, t2, and t3 for text a and c, respectively. If t1 is less than t2 and t2 is less than t3, merging third texts a, b, and c results in second text d. The duration of second text d is (t3-t1), where the text information of second text d is identical to any one of the text information of third texts a, b, and c.

[0164] In this embodiment, after acquiring the video to be detected, the detection device performs text detection on the video to obtain at least one third text. Then, by merging identical third texts within the at least one third text, at least one second text can be obtained.

[0165] It should be understood that in this embodiment, the duration of the second text can be determined by merging the same third text in at least one third text to obtain at least one second text, that is, the duration of the second text can also be determined through this embodiment.

[0166] As an optional implementation, the detection device obtains the duration of at least one second text in the video to be detected by performing the following steps:

[0167] 601. Use the timestamp of the video frame corresponding to the third text as the timestamp of the third text.

[0168] In other words, the timestamp of the video frame containing the third text is used as the timestamp of the third text.

[0169] 602. Determine the minimum timestamp of the third text corresponding to the second text as the start time of the second text.

[0170] In this embodiment, the second text is obtained by merging identical third texts. Therefore, the third text corresponding to the second text is the merged third text. For example, the detection device obtains the second text c by merging third text a and third text b. Then, the third text corresponding to the second text c includes third text a and third text b.

[0171] There are two or more third texts corresponding to the second text, and the timestamps of these different third texts are different. Therefore, the minimum timestamp of the third text corresponding to the second text is the minimum timestamp among all the third texts corresponding to the second text. For example, the third texts corresponding to the second text 'a' include third text 'b' and third text 'c', where the timestamp of third text 'b' is t1 and the timestamp of third text 'c' is t2. If t1 is less than t2, then the minimum timestamp of the third text corresponding to the second text 'a' is t1.

[0172] 603. Determine the maximum timestamp of the third text corresponding to the second text as the end time of the second text.

[0173] The maximum timestamp of the third text corresponding to the second text is the maximum timestamp among all the third texts corresponding to the second text. For example, the third texts corresponding to the second text a include third text b and third text c, where the timestamp of third text b is t1 and the timestamp of third text c is t2. If t1 is less than t2, then the maximum timestamp of the third text corresponding to the second text a is t2.

[0174] Based on steps 601 to 603, the detection device can determine the start and end times of any second text, that is, through steps 601 to 603, the start and end times of at least one second text can be determined.

[0175] 604. Based on the start time and end time of the at least one second text, the duration of the at least one second text in the video to be detected is obtained.

[0176] The detection device determines the time difference between the end time of the second text and the start time of the second text, thereby obtaining the duration of the second text in the video to be detected.

[0177] By implementing this method, the detection device can determine the duration of at least one second text in the video to be detected.

[0178] As an optional implementation, the detection device performs the following steps during step 503:

[0179] 701. Merge the identical third texts in at least one of the above third texts to obtain at least one of the above fourth texts.

[0180] In this step, the same third text refers to third texts whose text information is identical as determined by OCR recognition. The implementation method for merging identical third texts in this step can be found in step 503, and will not be repeated here. It should be understood that in step 503, the detection device obtains at least one second text by merging identical third texts in at least one third text; however, in this step, it obtains not at least one second text, but at least one fourth text, wherein the number of fourth texts is one or more.

[0181] 702. If the number of fourth texts is greater than 1, and there are fifth and sixth texts in at least one of the aforementioned fourth texts with a similarity greater than or equal to the fourth threshold, merge the aforementioned fifth and sixth texts in at least one of the aforementioned fourth texts to obtain at least one of the aforementioned second texts.

[0182] In this embodiment of the application, a high degree of similarity between two fourth texts indicates a high degree of semantic matching between the two fourth texts, that is, a high degree of matching of the information carried by the texts in the two fourth texts. In this case, the two fourth texts can be regarded as the same fourth text.

[0183] In this embodiment of the application, the detection device uses a fourth threshold to determine whether the similarity between two fourth texts is large or small. Specifically, if the similarity between two fourth texts is greater than or equal to the fourth threshold, it means that the similarity between the two fourth texts is large; conversely, if the similarity between two fourth texts is less than the fourth threshold, it means that the similarity between the two fourth texts is small.

[0184] In this embodiment, when the number of fourth texts is greater than 1, the fifth and sixth texts are two different fourth texts among all the fourth texts, and the similarity between the fifth and sixth texts is greater than or equal to a fourth threshold, that is, the fifth and sixth texts are the same text. Therefore, by executing step 702, the detection device obtains at least one second text, which can improve the accuracy of at least one second text.

[0185] It should be understood that the fifth and sixth texts are merely descriptive objects selected for concisely describing the technical solutions provided in the embodiments of this application, and should not be construed as meaning that, when the number of fourth texts is greater than 1, at least one fourth text contains only fifth and sixth texts with a similarity greater than or equal to a fourth threshold, nor should it be construed as simply merging the fifth and sixth texts in at least one fourth text. In practical applications, at least one fourth text may contain one or more pairs of fourth texts with a similarity greater than or equal to a fourth threshold. The detection device can obtain at least one second text by merging all pairs of fourth texts in at least one fourth text with a similarity greater than or equal to a fourth threshold.

[0186] 703. If there are no two fourth texts with a similarity greater than or equal to the fourth threshold among the above-mentioned at least one fourth text, the above-mentioned at least one fourth text shall be regarded as the above-mentioned at least one second text.

[0187] If there are no two fourth texts with a similarity greater than or equal to the fourth threshold in at least one fourth text, it means that there are no two identical fourth texts among at least two fourth texts. Therefore, the detection device will treat at least one fourth text as at least one second text.

[0188] In this embodiment, after merging identical third texts in at least one third text to obtain at least one fourth text, the detection device further determines whether there are identical fourth texts among the at least one fourth text based on the similarity between the fourth texts. If it is determined that there are identical fourth texts among the at least one fourth text, at least one second text is obtained by merging the identical fourth texts among the at least one fourth text. If it is determined that there are no identical fourth texts among the at least one fourth text, the at least one fourth text is used as at least one second text, thereby improving the accuracy of the at least one second text.

[0189] As an optional implementation, if the number of fourth texts is greater than 1, and at least one fourth text contains fifth and sixth texts with a similarity greater than or equal to a fourth threshold, the detection device merges the fifth and sixth texts in at least one fourth text to obtain at least one second text by performing the following steps:

[0190] 801. Obtain the start time of the fifth text, the end time of the fifth text, the start time of the sixth text, and the end time of the sixth text, wherein the start time of the fifth text is earlier than the start time of the sixth text.

[0191] 802. Determine the first time difference between the start time of the fifth text and the end time of the sixth text.

[0192] In this embodiment of the application, the time difference between the start time of the fifth text and the end time of the sixth text is called the first time difference.

[0193] 803. Determine the median of the start time and end time of the fifth text to obtain the median time of the fifth text.

[0194] For example, if the start time of the fifth text is t1 and the end time of the fifth text is t2, then the median time of the fifth text is (t2-t1) / 2.

[0195] 804. Determine the median of the start time and end time of the sixth text above to obtain the median time of the sixth text above.

[0196] For example, if the start time of the sixth text is t1 and the end time of the sixth text is t2, then the median time of the sixth text is (t2-t1) / 2.

[0197] 805. Determine the second time difference between the median time of the fifth text and the median time of the sixth text.

[0198] In this embodiment of the application, the time difference between the median time of the fifth text and the median time of the sixth text is called the second time difference.

[0199] 806. If the ratio of the first time difference to the second time difference is less than or equal to the fifth threshold, the fifth text and the sixth text in the at least one fourth text are merged to obtain the at least one second text.

[0200] A video may contain the same subtitle repeatedly at different times. If the time interval between the appearances of the same subtitle is large, merging two identical subtitles into one is clearly unreasonable. For example, subtitle A may appear between 1 minute and 1 minute 20 seconds into the video being tested, and also between 15 minutes and 15 minutes 20 seconds. In this case, merging the two subtitles A appearing at different times into one is obviously unreasonable.

[0201] Therefore, when the detection device determines that two fourth texts are the same fourth text, it can further determine whether the time interval between the two fourth texts is large or small, and merge the two fourth texts if the time interval between the two fourth texts is large, and not merge the two fourth texts if the time interval between the two fourth texts is small.

[0202] In this embodiment, if the ratio of the first time difference to the second time difference is less than or equal to the fifth threshold, it indicates that the interval between the appearance time of the fifth text and the appearance time of the sixth text in the video to be detected is small. In this case, it can be determined that the fifth text and the sixth text are the same text, and at least one second text can be obtained by merging the fifth text and the sixth text in at least one fourth text. Conversely, if the ratio of the first time difference to the second time difference is large, it indicates that the interval between the appearance time of the fifth text and the appearance time of the sixth text in the video to be detected is large. In this case, it can be determined that the fifth text and the sixth text are different texts, and therefore the fifth text and the sixth text are not merged.

[0203] In this embodiment, when the number of fourth texts is greater than 1, and at least one of the fourth texts contains a fifth text and a sixth text with a similarity greater than or equal to a fourth threshold, the detection device acquires the start time, end time, start time, and end time of the fifth text, where the start time of the fifth text is earlier than the start time of the sixth text. Then, a first time difference between the start time of the fifth text and the end time of the sixth text is determined; the median time of the fifth text is obtained by determining the median time of the fifth text by determining the median time of the sixth text by determining the median time of the sixth text by determining the median time of the sixth text by determining the median time of the fifth text and the median time of the sixth text; and a second time difference between the median time of the fifth text and the median time of the sixth text is determined.

[0204] Finally, based on the ratio of the first time difference to the second time difference, it is determined whether the interval between the appearance times of the fifth text and the sixth text in the video to be detected is large or small. Specifically, if the ratio of the first time difference to the second time difference is less than or equal to a fifth threshold, it is determined that the interval between the appearance times of the fifth text and the sixth text in the video to be detected is small. The fifth and sixth texts from at least one fourth text are then merged to obtain at least one second text. This improves the accuracy of at least one second text.

[0205] Based on the technical solutions provided in the embodiments of this application, the embodiments of this application also provide a possible implementation method. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating another video subtitle detection method provided in an embodiment of this application. Figure 2 As shown, the detection device first performs frame extraction on the video to be processed to obtain the video to be detected. Optionally, the detection device extracts a frame from the video to be processed every 0.2 seconds. Then, OCR recognition is performed on the frame extraction result (i.e., the video to be detected) to obtain at least one third text. The text information in the third text is used as the key, and the position of the third text in the video to be detected and the timestamp of the third text are stored in a dictionary as values.

[0206] At least one fourth text is obtained by merging identical third texts from at least one third text. Duplicate fields caused by OCR recognition errors are removed from at least one fourth text. Figure 2 (Removing duplicate fields caused by OCR recognition errors) For example, if there are cursive or handwritten characters in the video to be detected, OCR is prone to detecting cursive or handwritten characters incorrectly.

[0207] Optional, please see Figure 3 , Figure 3 This is a schematic diagram illustrating a process for removing duplicate fields, provided as an embodiment of this application. For example... Figure 3 As shown, the detection device first compares field similarity (i.e., determines the similarity between any two fourth texts). If a field pair with similarity exceeding a threshold is found, it determines whether the field pair intersects on the timeline. The process for determining the existence of field pairs with similarity exceeding the threshold can be found in the previously described process for determining that at least one fourth text contains a fifth and a sixth text with a similarity greater than or equal to the fourth threshold. The process for determining whether the field pair intersects on the timeline can be found in the previously described process for determining whether the interval between the appearance times of the fifth and sixth texts in the video to be detected is large or small. Specifically, if the interval between the appearance times of the fifth and sixth texts in the video to be detected is small, it is determined that the fifth and sixth texts intersect on the timeline, thus determining that the fifth and sixth texts are duplicate fields (i.e., the fifth and sixth texts are the same text), and then merging the fifth and sixth texts. If the time interval between the appearance of the fifth text and the appearance of the sixth text in the video to be detected is large, it is determined that the fifth text and the sixth text do not overlap on the timeline, and thus it is judged that the fifth text and the sixth text are two sentences from different times (i.e., the fifth text and the sixth text are different texts), and therefore the fifth text and the sixth text are not merged.

[0208] After obtaining at least one second text by removing duplicate fields caused by OCR recognition errors, remove fields with excessively long or short durations, i.e., remove second texts whose duration is outside the preset range, to obtain at least one first text. The implementation of this process can be found in step 402 above.

[0209] After obtaining at least one first text, the text positions recognized by OCR are clustered to obtain at least one text class. That is, based on the position of the first text in the video to be detected, at least one text is classified to obtain at least one text class. The implementation of this process can be found in step 102. Clustering results with fewer than a threshold of samples are removed to obtain the OCR result of the subtitle. The implementation of this process can be found in steps 103 and 104.

[0210] Optional, Figure 4 This is a schematic diagram of a clustering process provided in an embodiment of this application. In the process of removing clustering results with fewer than a threshold samples to obtain the OCR results for the subtitles, the detection device can execute... Figure 4 The clustering process shown is used for clustering. For example... Figure 4 As shown, the detection device first compares the horizontal intersection-union ratio (IU / U) of the text boxes, that is, compares the IU / U of any two first text text boxes in the horizontal direction. If the horizontal IU / U is less than a preset value, it further compares the vertical IU / U of the text boxes, that is, compares the vertical IU / U of the two first text text boxes. If the vertical IU / U is also less than the preset value, and the IU / U of the area of ​​the text boxes is determined to be less than or equal to a second threshold by comparing the IU / U of the area of ​​the text boxes, the text boxes are clustered. Based on the clustering results, clusters with fewer samples are deleted to obtain the subtitle cluster. Here, the IU / U of the area of ​​the text boxes being less than or equal to the second threshold means that the IU / U of the area of ​​the two first text text boxes is less than or equal to the second threshold. Clustering the text boxes determines that the two first texts are in the same position in the video to be detected, and thus classifies the two first texts into the same text class. The implementation of deleting clusters with fewer samples based on the clustering results to obtain the subtitle cluster can be found in step 104.

[0211] Based on the technical solutions provided in the embodiments of this application, several possible application scenarios are also provided. With the rapid development of user-generated content (UGC) communities, the applications of UGC communities are becoming increasingly widespread. This includes users sharing videos through UGC communities. Specifically, users upload videos to UGC communities so that all users within the community can view the video. Since different users are interested in different videos, it is necessary to manage the videos uploaded to UGC (hereinafter referred to as uploaded videos) to present the videos that users are interested in. If the server running the UGC community is used as the aforementioned detection device, then when the server stores uploaded videos, the server can determine the subtitles in the uploaded videos by executing the video subtitle detection method described above, and then use these subtitles as tags for the uploaded videos. In this way, the server can determine the videos that users are interested in based on the tags of the uploaded subtitles.

[0212] In one possible scenario, after obtaining user information (such as age, interests, gender, etc.), the server identifies uploaded videos with tags matching the user's information as videos of interest to the user. Here, the user information can be used to identify the user. In another possible scenario, the user sends a video search command to the server via their terminal. This video search command includes search terms for the videos being searched. Upon receiving the video search command, the server identifies uploaded videos with tags matching the search terms in the video search command as videos of interest to the user.

[0213] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0214] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, while using clear signs / information to inform users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0215] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0216] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a video subtitle detection device 1 provided in an embodiment of this application. The detection device 1 includes: an acquisition unit 11, a segmentation unit 12, a determination unit 13, and a processing unit 14. Specifically:

[0217] Acquisition unit 11 is used to acquire at least one first text in the video to be detected and the position of the at least one first text in the video to be detected;

[0218] The segmentation unit 12 is used to segment the at least one first text by grouping the first texts that are located in the same position in the video to be detected into one category, thereby obtaining at least one text class;

[0219] Determining unit 13 is used to determine the text class containing a number of first texts greater than or equal to a first threshold as the target text class;

[0220] Processing unit 14 is used to use the first text in the target text class as the subtitle of the video to be detected.

[0221] The determining unit 13 is further configured to:

[0222] If the number of first texts is greater than 1, the intersection-union ratio of the areas of any two first texts among the at least one first text is determined based on the position of the at least one first text in the video to be detected;

[0223] If the cross-union ratio is greater than or equal to the second threshold, it is determined that the two first texts corresponding to the cross-union ratio are in the same position in the video to be detected.

[0224] In any embodiment of this application, the pixel coordinate system of the video to be detected includes a horizontal axis and a vertical axis, wherein the direction of the horizontal axis is horizontal and the direction of the vertical axis is vertical.

[0225] The determining unit 13 is further configured to:

[0226] When the intersection-union ratio is less than the second threshold, based on the positions of the two first texts corresponding to the intersection-union ratio in the video to be detected, the center-horizontal distance between the centers of the two first texts corresponding to the intersection-union ratio in the horizontal direction is determined, as well as the center-vertical distance between the centers of the two first texts corresponding to the intersection-union ratio in the vertical direction is determined, as well as the maximum horizontal distance between the two first texts corresponding to the intersection-union ratio in the horizontal direction is determined, as well as the maximum vertical distance between the two first texts corresponding to the intersection-union ratio in the vertical direction is determined;

[0227] If the ratio of the center horizontal distance to the maximum horizontal distance is greater than or equal to a third threshold, and the ratio of the center vertical distance to the maximum vertical distance is greater than or equal to the third threshold, then the two first texts corresponding to the intersection-union ratio are determined to be in the same position in the video to be detected.

[0228] In any embodiment of this application, the acquisition unit 11 is used for:

[0229] Obtain at least one second text in the video to be detected and the duration of the at least one second text in the video to be detected;

[0230] The at least one first text is obtained by selecting texts whose duration is within a preset range from the at least one second text.

[0231] In any embodiment of this application, the acquisition unit 11 is used for:

[0232] Acquire the video to be detected;

[0233] The video to be detected is subjected to text detection to obtain at least one third text.

[0234] Merge the identical third texts in the at least one third text to obtain the at least one second text.

[0235] In any embodiment of this application, the acquisition unit 11 is used for:

[0236] Use the timestamp of the video frame corresponding to the third text as the timestamp of the third text;

[0237] Determine the minimum timestamp of the third text corresponding to the second text as the start time of the second text;

[0238] The maximum timestamp of the third text corresponding to the second text is determined as the end time of the second text;

[0239] The duration of the at least one second text in the video to be detected is obtained based on the start time and end time of the at least one second text.

[0240] In any embodiment of this application, the acquisition unit 11 is used for:

[0241] Merge the identical third texts in the at least one third text to obtain the at least one fourth text;

[0242] If the number of fourth texts is greater than 1, and there are fifth and sixth texts in the at least one fourth text with a similarity greater than or equal to a fourth threshold, then the fifth and sixth texts in the at least one fourth text are merged to obtain the at least one second text;

[0243] If there are no two fourth texts in the at least one fourth text that have a similarity greater than or equal to a fourth threshold, then the at least one fourth text shall be regarded as the at least one second text.

[0244] In any embodiment of this application, the acquisition unit 11 is used for:

[0245] Obtain the start time of the fifth text, the end time of the fifth text, the start time of the sixth text, and the end time of the sixth text, wherein the start time of the fifth text is earlier than the start time of the sixth text;

[0246] Determine the first time difference between the start time of the fifth text and the end time of the sixth text;

[0247] The median time of the fifth text is obtained by determining the median time of the fifth text by taking the median start time and the median end time of the fifth text.

[0248] The median time of the sixth text is obtained by determining the median time of the sixth text by taking the median start time and the median end time of the sixth text.

[0249] Determine a second time difference between the median time of the fifth text and the median time of the sixth text;

[0250] If the ratio of the first time difference to the second time difference is less than or equal to the fifth threshold, the fifth text and the sixth text in the at least one fourth text are merged to obtain the at least one second text.

[0251] In this embodiment, after acquiring at least one first text in the video to be detected and its position within the video, the detection device divides the at least one first text into at least one text class by grouping the first texts located at the same position in the video. Then, by identifying target text classes from the at least one text class that contain a number of first texts greater than or equal to a first threshold, the device determines the target text class whose corresponding position is the location where the subtitle appears. Finally, the first text in the target text class is used as the subtitle of the video to be detected, thereby detecting the subtitles in the video and improving the efficiency of subtitle detection.

[0252] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0253] Figure 6This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0254] Processor 21 may include one or more processors, such as one or more central processing units (CPUs). If the processor is a CPU, it may be a single-core CPU or a multi-core CPU. Optionally, processor 21 may be a processor group consisting of multiple CPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in this embodiment.

[0255] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0256] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0257] It is understood that in this embodiment of the application, the memory 22 can be used not only to store related instructions, but also to store related data. This embodiment of the application does not limit the specific data stored in the memory.

[0258] Understandable Figure 6This is merely a simplified design of an electronic device. In practical applications, the electronic device may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of this application are within the protection scope of this application.

[0259] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0260] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0261] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0262] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0263] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0264] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0265] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for detecting video subtitles, characterized in that, The method includes: The start time, end time, start time, and end time of the fifth text, and the sixth text are obtained, wherein the start time of the fifth text is earlier than the start time of the sixth text; the fifth text and the sixth text are two texts with a similarity greater than or equal to a fourth threshold among at least one fourth text, wherein the at least one fourth text is obtained by merging the same third text among at least one third text, and the at least one third text is obtained by performing text detection on the video to be detected; Determine the first time difference between the start time of the fifth text and the end time of the sixth text; The median time of the fifth text is obtained by determining the median time of the fifth text by taking the median start time and the median end time of the fifth text. The median time of the sixth text is obtained by determining the median time of the sixth text by taking the median start time and the median end time of the sixth text. Determine a second time difference between the median time of the fifth text and the median time of the sixth text; If the ratio of the first time difference to the second time difference is less than or equal to the fifth threshold, the fifth text and the sixth text are merged to obtain at least one second text; At least one first text is selected from the at least one second text, wherein the duration of the at least one first text in the video to be detected is within a preset range; Obtain the position of the at least one first text in the video to be detected; By grouping the first texts that are located in the same position in the video to be detected into one category, at least one text class is obtained by dividing the at least one first text. The text class containing a number of first texts greater than or equal to a first threshold is identified as the target text class. The first text in the target text class is used as the subtitle of the video to be detected.

2. The method according to claim 1, characterized in that, After obtaining at least one first text in the video to be detected and the position of the at least one first text in the video to be detected, and before dividing the at least one first text into at least one text class by grouping the first texts that are located in the same position in the video to be detected into one class, the method further includes: If the number of first texts is greater than 1, the intersection-union ratio of the areas of any two first texts among the at least one first text is determined based on the position of the at least one first text in the video to be detected; If the cross-union ratio is greater than or equal to the second threshold, it is determined that the two first texts corresponding to the cross-union ratio are in the same position in the video to be detected.

3. The method according to claim 1, characterized in that, Before selecting at least one first text from the at least one second text, the method further includes: Use the timestamp of the video frame corresponding to the third text as the timestamp of the third text; Determine the minimum timestamp of the third text corresponding to the second text as the start time of the second text; The maximum timestamp of the third text corresponding to the second text is determined as the end time of the second text; The duration of the at least one second text in the video to be detected is obtained based on the start time and end time of the at least one second text.

4. A device for detecting video subtitles, characterized in that, The detection device includes: The acquisition unit is used to acquire the start time of the fifth text, the end time of the fifth text, the start time of the sixth text, and the end time of the sixth text, wherein the start time of the fifth text is earlier than the start time of the sixth text; the fifth text and the sixth text are two texts with a similarity greater than or equal to a fourth threshold among at least one fourth text, wherein the at least one fourth text is obtained by merging the same third text among at least one third text, and the at least one third text is obtained by performing text detection on the video to be detected; A determining unit is configured to determine a first time difference between the start time of the fifth text and the end time of the sixth text; The determining unit is further configured to determine the median of the start time and the end time of the fifth text, thereby obtaining the median time of the fifth text; The determining unit is further configured to determine the median of the start time and the end time of the sixth text, thereby obtaining the median time of the sixth text; The determining unit is further configured to determine a second time difference between the median time of the fifth text and the median time of the sixth text; The merging unit is configured to merge the fifth text and the sixth text to obtain at least one second text when the ratio of the first time difference to the second time difference is less than or equal to a fifth threshold. A selection unit is configured to select at least one first text from the at least one second text, wherein the duration of the at least one first text in the video to be detected is within a preset range. The acquisition unit is further configured to acquire the position of the at least one first text in the video to be detected; The segmentation unit is used to segment the at least one first text by grouping the first texts that are located in the same position in the video to be detected into one category, thereby obtaining at least one text class; A determining unit is configured to determine a text class whose number of first texts contained in the at least one text class is greater than or equal to a first threshold as a target text class; The processing unit is used to use the first text in the target text class as the subtitle of the video to be detected.

5. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 3.

Images