A method and device for determining video subtitle area
By extracting image frames from the video, identifying text areas and calculating the change rate, and determining the subtitle area, the problem of excessive interfering content in video subtitle recognition is solved, and the accuracy of subtitle extraction is improved.
Patent Information
- Application Number
- CN202311865215.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-12-29
AI Technical Summary
The prior art is difficult to accurately determine the video subtitle area, resulting in a large amount of interfering content in subtitle extraction, reducing the accuracy of subtitle recognition.
By extracting multiple image frames from the target video, identifying text areas, and calculating text change rates in adjacent image frames, determining subtitle areas based on the overall text change rate, and filtering interfering content using the preset rate of change threshold and average area, accurately Position the subtitle area.
It improves the recognition accuracy of video subtitles, reduces the impact of interfering content, and enhances the accuracy of subtitle extraction.
Smart Images

Figure CN117912000B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method and device for determining a video subtitle area. Background Art
[0002] For video files with voice, the accuracy of subtitles extracted from the video is much higher than that of subtitles obtained by transcribing the audio. In addition, some subtitles contain special nouns, punctuation marks, and sentence breaks, which will inevitably lead to higher accuracy of subtitles extracted from the video.
[0003] However, when extracting subtitles from video images, there will be many interference items, such as news titles, column names, rolling advertisements / news, pictures with text, etc. These will inevitably lead to a decrease in the accuracy of subtitles obtained based on text extraction from video images.
[0004] As to how to effectively improve the accuracy of subtitle extraction in videos, no effective solution has been proposed so far. Summary of the invention
[0005] The purpose of the present application is to provide a method and device for determining a video subtitle area, which can achieve accurate positioning of the video subtitle area, thereby improving the accuracy of video subtitle recognition.
[0006] In a first aspect, the present application provides a method for determining a subtitle area of a video, the method comprising:
[0007] Get the target video;
[0008] Extracting a plurality of image frames from the target video;
[0009] identifying a text region in each of the plurality of image frames;
[0010] Calculate the text change rate of each text area in adjacent image frames;
[0011] Determining an overall text change rate of each text area in the plurality of image frames according to a text change rate of each text area in adjacent image frames;
[0012] The subtitle area of the target video is determined according to the overall text change rate of each text area.
[0013] In some embodiments of the present application,
[0014] Determining the overall text change rate of each text area in the plurality of image frames according to the text change rate of each text area in adjacent image frames includes:
[0015] Accumulate and sum the text change rates of adjacent image frames corresponding to the current text area or perform the cumulative summing and then averaging;
[0016] The cumulative summation result or the cumulative summation and averaging result is used as the overall text change rate of the current text area in the multiple image frames to obtain the overall text change rate of each text area in the multiple image frames.
[0017] In some embodiments of the present application, determining the subtitle area of the target video according to the overall text change rate of each text area includes:
[0018] Get the preset upper limit of the change rate threshold;
[0019] Eliminate text regions whose overall text change rate is higher than the upper limit of the change rate threshold from the text regions, and use the remaining text regions as candidate subtitle regions;
[0020] A target subtitle area is selected from the candidate subtitle areas as the subtitle area of the target video.
[0021] In some embodiments of the present application, determining the subtitle area of the target video according to the overall text change rate of each text area includes:
[0022] Get the preset upper and lower limits of the change rate threshold;
[0023] Eliminate, from the text regions, text regions whose overall text change rate is higher than an upper limit of a change rate threshold, and text regions whose overall text change rate is lower than a lower limit of a change rate threshold, and use the remaining text regions as candidate subtitle regions;
[0024] A target subtitle area is selected from the candidate subtitle areas as the subtitle area of the target video.
[0025] In some embodiments of the present application, selecting a target subtitle area from the candidate subtitle areas as the subtitle area of the target video includes:
[0026] Determine the average area of each candidate subtitle region;
[0027] From the candidate subtitle regions, a candidate subtitle region with the largest average area is selected as the subtitle region of the target video.
[0028] In some embodiments of the present application, the step of calculating the average area of each candidate subtitle region includes:
[0029] The average area of the current candidate subtitle region is obtained by dividing the overall text change rate of the current candidate subtitle region by the area of the current text region.
[0030] In some embodiments of the present application, determining the subtitle area of the target video according to the overall text change rate of each text area includes:
[0031] Get the preset upper and lower limits of the change rate threshold;
[0032] The text areas whose overall text change rate is higher than the upper limit of the change rate threshold are eliminated from the text areas, and the text areas whose overall text change rate is lower than the lower limit of the change rate threshold are eliminated, and the remaining text areas are used as the subtitle areas of the target video.
[0033] In some embodiments of the present application, the upper limit of the change rate threshold is determined based on the duration of the target video and a given number of words spoken per unit time.
[0034] In some embodiments of the present application, before determining the subtitle area of the target video, the determination method further includes:
[0035] Determine the average area of each text region;
[0036] Eliminate text regions whose average area is smaller than a given area threshold.
[0037] In some embodiments of the present application, identifying a text area in each of the multiple image frames includes:
[0038] Convert the current image frame to a grayscale image;
[0039] Performing binarization processing on each pixel in the grayscale image according to increasing threshold values;
[0040] As the threshold continues to increase, the text content changes from distinguishable to indistinguishable, so as to identify the boundary of the text content;
[0041] The boundaries of the text content are merged and divided to obtain one or more text regions in the current image frame.
[0042] In some embodiments of the present application, calculating the text change rate of each text area in adjacent image frames includes:
[0043] Obtaining the region boundary and text content of each text region in the first frame of the adjacent image frames;
[0044] Acquire the region boundary and text content of each text region in the second frame of the adjacent image frames;
[0045] For each text area in the first frame: determine the area change ratio and the number of text content transformation operation steps required to transform the current text area in the first frame into the corresponding text area in the second frame, and calculate the text change rate based on the area change ratio and the number of text content transformation operation steps.
[0046] In some embodiments of the present application, after calculating the text change rate of each text area in adjacent image frames, the method further includes:
[0047] The determination method further comprises:
[0048] Generating a pure color and equal-size feature map for the multiple image frames;
[0049] The text change rate of each text area is displayed in the corresponding area of the feature map in the form of a heat map, wherein different text change rates in the heat map correspond to different color displays, and continuous changes in the text change rate correspond to continuous color transitions.
[0050] In a second aspect, an embodiment of the present application provides a device for determining a video subtitle area, including:
[0051] An acquisition module, used to acquire a target video;
[0052] A segmentation module, used for extracting a plurality of image frames from the target video;
[0053] A recognition module, used for recognizing a text area in each of the plurality of image frames;
[0054] A calculation module, used for calculating the text change rate of each text area in adjacent image frames;
[0055] A first determination module, configured to determine an overall text change rate of each text region in the plurality of image frames according to a text change rate of each text region in adjacent image frames;
[0056] The second determination module is used to determine the subtitle area of the target video according to the overall text change rate of each text area.
[0057] In some embodiments of the present application, the first determining module is specifically used to:
[0058] Accumulate and sum the text change rates of adjacent image frames corresponding to the current text area or perform the cumulative summing and then averaging;
[0059] The cumulative summation result or the cumulative summation and averaging result is used as the overall text change rate of the current text area in the multiple image frames to obtain the overall text change rate of each text area in the multiple image frames.
[0060] In some embodiments of the present application, the second determining module includes:
[0061] An acquisition unit, used for acquiring a preset upper limit of a change rate threshold and a lower limit of a change rate threshold;
[0062] A removal unit, used to remove text areas whose overall text change rate is higher than an upper limit of a change rate threshold, and remove text areas whose overall text change rate is lower than a lower limit of a change rate threshold from the text areas, and use the remaining text areas as candidate subtitle areas;
[0063] A determination unit is used to select a target subtitle area from the candidate subtitle areas as the subtitle area of the target video.
[0064] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory for storing instructions executable by the processor, wherein the processor implements the steps of the above method when executing the instructions.
[0065] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when the computer program / instruction is executed by a processor.
[0066] The method and device for determining the subtitle area of a video provided by the present application extract multiple image frames from the target video, then identify the text area in each of the multiple image frames, calculate the text change rate of each text area in adjacent image frames, determine the overall text change rate of each text area in the multiple image frames based on the text change rate of each text area in the adjacent image frames, and then determine the subtitle area of the target video based on the overall text change rate of each text area. That is, combining the inter-frame text change rate characteristics of each text area to accurately identify which area in the video is the subtitle area, the above scheme solves the existing technical problem of being unable to accurately determine the subtitle area, resulting in too much interference content in the final generated subtitles, and achieves the technical effect of accurately locating the subtitle area to improve the recognition accuracy of the video subtitles. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use are briefly introduced below. It should be clear that the drawings in the following description only cover some embodiments of the present application, and ordinary technicians in this field can obtain other drawings based on these drawings without performing creative work. The purpose of these drawings is to better illustrate the technical details to help understand the implementation methods of the present application, among which:
[0068] Figure 1 It is a method flow chart of an embodiment of a method for determining a video subtitle area provided by the present application;
[0069] Figure 2 It is a schematic diagram of dividing a video provided by the present application into multiple frames;
[0070] Figure 3is a schematic diagram of identifying a text area in an image frame provided by the present application;
[0071] Figure 4 is a schematic diagram of calculating the text change rate of adjacent key frames provided by the present application;
[0072] Figure 5 is a method flow chart of another embodiment of the method for determining the video subtitle area provided by the present application;
[0073] Figure 6 This is a schematic diagram of the module structure of an embodiment of a device for determining a video subtitle area provided by the present application;
[0074] Figure 7 This is a hardware structure block diagram of an electronic device for a method for determining a video subtitle area provided by the present application. DETAILED DESCRIPTION
[0075] The exemplary embodiments of the present application will be described in detail below, and the illustrations of the examples have been shown in the accompanying drawings. When referring to the drawings, unless otherwise specified, the same numbers or marks in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Instead, they are only examples of some aspects of the devices and methods covered by one or more embodiments of this specification as detailed in the claims of this application.
[0076] The term "including" and its variations used in this specification are all intended to be broadly inclusive, i.e., "including but not limited to" the items listed. Unless otherwise stated, the term "or" means "and / or", the term "based on" means relying on, or at least partially relying on, the terms "an example embodiment" and "an embodiment" refer to at least one example embodiment, and the term "another embodiment" means at least one different embodiment. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0077] In the embodiments of the present application, there are sometimes station logos, rolling news, and floating watermarks in the video, which may lead to reduced accuracy of subtitle recognition. In this example, in order to improve the accuracy of the final subtitle recognition, it is necessary to first accurately determine the area where the subtitles are located in the video screen.
[0078] Figure 1It is a method flow chart of an embodiment of the method for determining the video subtitle area provided by the present application. Although the present application provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present application and shown in the drawings. When the method or module structure is applied to an actual device or terminal product, it can be connected according to the method or module structure shown in the embodiments or drawings for sequential execution or parallel execution (for example, a parallel processor or a multi-threaded processing environment, or even a distributed processing environment).
[0079] Specifically, Figure 1 As shown, the above-mentioned method for determining the video subtitle area may include the following steps:
[0080] S110: Acquire target video;
[0081] For example, there is a video, which can be a movie video or a short video. For these videos, there is usually a subtitle area, and for each video, the location of the subtitle area is generally fixed. However, for the video screen, there are sometimes some text content or station logos, which do not need to be recognized as subtitles. In order to improve the accuracy of subtitle recognition, it is necessary to accurately mark the subtitle area.
[0082] S120: extracting multiple image frames from the target video;
[0083] When implementing, image frames can be extracted according to a preset time interval. The time interval can be set according to actual needs and the characteristics of each video. The target video can be divided into Figure 2 Multiple image frames are shown (image frame 1, image frame 2, image frame 3...image frame m, image frame m+1, image frame m+2).
[0084] In another embodiment, all image frames of the target video may be directly extracted.
[0085] S130: Identify a text area in each of the multiple image frames;
[0086] When implementing, the region containing text in each image frame can be identified as the text region by means of text recognition and the like, such as Figure 3As shown, assuming that text area 1, text area 2, and text area 3 are identified in image frame 1, there are texts in these areas. It is necessary to identify which area is the subtitle area from these text areas. This is the purpose of the solution provided in this example.
[0087] S140: Calculating text change rates of each text area in adjacent image frames;
[0088] like Figure 4 As shown, for adjacent key frames, the text change rate can be calculated. For example, the text change rate of each text area is calculated based on image frame 1 and image frame 2, and the text change rate of each text area is calculated based on image frame 2 and image frame 3. The text change rate of each text area in each adjacent image frame can be calculated in this way.
[0089] Specifically, when calculating the text change rate of each text area in the adjacent image frames, the area boundary and text content of each text area in the first frame of the adjacent image frames can be obtained; the area boundary and text content of each text area in the second frame of the adjacent image frames can be obtained; for each text area in the first frame: determine the area change ratio and the number of text content transformation operation steps required for the current text area in the first frame to be transformed into the corresponding text area in the second frame, and calculate the text change rate according to the area change ratio and the number of text content transformation operation steps. That is, based on the area change ratio and the number of text content transformation operation steps, the text change rate is calculated. A weight value can be set for the area, and a weight value can be set for the text content. The two changes are weighted and summed to obtain the text change rate. For example, if the area change weight is set to 1 and the text content change weight is set to 0, then only the area change is used for judgment. If the area change weight is set to 0 and the text content change weight is set to 1, then only the text content change is used for judgment. If the area change weight is set to 0.8 and the text content change weight is set to 0.2, then the area change is used as the main judgment basis.
[0090] Among them, the change ratio of the area can be an increase or decrease in the area. The area of the first frame can be used as a reference to determine the increase or decrease ratio of the area of the corresponding area in the second frame, and this ratio value is used as the change ratio of the area; wherein, the number of operation steps for transforming the text content can be understood as the number of steps required to transform from the text content of the first frame to the text content of the second frame. For example: if the first frame is "Hello, little friend!" and the second frame is also "Hello, little friend!", then the number of steps executed is 0, because no change is required. If the first frame is "Hello, little friend!" and the second frame is "Are you a little friend!", then "good" needs to be changed to "yes", "," is removed, and "may" is added. This requires executing three steps, that is, the number of steps executed is 3.
[0091] S150: determining the overall text change rate of each text area in the plurality of image frames according to the text change rate of each text area in adjacent image frames;
[0092] S160: Determine the subtitle area of the target video according to the overall text change rate of each text area.
[0093] Furthermore, when determining the overall text change rate of each text area in the multiple image frames based on the text change rate of each text area in the adjacent image frames, the text change rates of the adjacent image frames corresponding to the current text area can be accumulated and summed; the accumulated and summed result is used as the overall text change rate of the current text area in the multiple image frames to obtain the overall text change rate of each text area in the multiple image frames. In another embodiment, the overall text change rate can be the text change rate of the adjacent frames accumulated and summed and then averaged, and the present application is not limited here. That is, for the video screen, the text area and the text change rate calculated above can be accumulated (or averaged after accumulation) based on each image frame, so that this change can be displayed from an overall dimension.
[0094] For example, for a fixed logo, the overall text change rate is almost 0, because it does not change at all. Even if it flashes occasionally, the value is very small when accumulated. For scrolling news or advertisements, it changes all the time, resulting in a large overall text change rate. For the subtitle area, the subtitles change in stages, that is, the content of this sentence needs to be displayed before switching to the next sentence. Therefore, the final accumulated overall text change rate is medium.
[0095] Based on this, when determining the subtitle area of the target video according to the overall text change rate of each text area, the station logo, fixed watermark, rolling advertisement and news can be removed by setting the upper and lower limits of the change rate threshold, that is, obtaining the preset change rate threshold upper limit and change rate threshold lower limit; removing the text area whose overall text change rate is higher than the change rate threshold upper limit from each text area, removing the text area whose overall text change rate is lower than the change rate threshold lower limit, and taking the remaining text area as the candidate subtitle area; selecting the target subtitle area from the candidate subtitle area as the subtitle area of the target video. In a preferred embodiment of the present application, the change rate threshold upper limit is determined based on the duration of the target video and the given number of words spoken per unit time. Specifically, the change rate threshold upper limit can be determined by the video duration x the number of words spoken per unit time. In another embodiment, additional parameters or margins or coefficients can also be added, and the present application does not limit this. By setting the change rate threshold upper limit in this way, it can be effectively used as a means of filtering rolling subtitles.
[0096] Furthermore, considering that there is a floating moving watermark, this type of watermark can be excluded by calculating the average area. This is mainly because the floating watermark keeps jumping in various places on the screen, which makes the cumulative change rate of the text in these text areas lower (because in most cases, there is no text in this area), and the area corresponding to the area is larger, which makes the final average area smaller.
[0097] Based on this, selecting a target subtitle area from the candidate subtitle areas as the subtitle area of the target video may include: calculating the average area of each candidate subtitle area in the following manner: dividing the overall text change rate of the current candidate subtitle area by the area of the current text area to obtain the average area of the current candidate subtitle area; and selecting a candidate subtitle area with the largest average area from the candidate subtitle areas as the subtitle area of the target video.
[0098] For the above-mentioned identification of text areas in each of the multiple image frames, the text recognition technology can be used to identify the area where the text is located and the specific text content of the text. In this example, a method for identifying the area where text content exists in the video is provided, which converts the current image frame into a grayscale image; performs binarization processing on each pixel in the grayscale image according to the threshold value increasing from 0 to 255 in sequence; based on the characteristic that the grayscale value of the text content is consistent, in the process of continuous increase of the threshold, the text content changes from distinguishable to indistinguishable when the threshold value increases to the grayscale value of the text content, and identifies the boundary of the text content; merges and divides the boundary of the text content to obtain one or more text areas in the current image frame. That is, when implementing, first determine the boundary of each text content by grayscale value, and then merge the text areas whose distance is less than the threshold, so as to obtain one or more text areas.
[0099] Further, in order to more clearly characterize the text change rate of each text area or the overall text change rate, a heat map can be used for characterization. For example, after calculating the text change rate of each text area in adjacent image frames, a large feature map of a pure color (such as pure white) can be generated for the image frame; the text change rate of each text area is displayed in the corresponding area of the feature map in the form of a heat map, for example, the text change rate of each adjacent frame is superimposed in the form of a heat map (the superimposed heat map represents the overall text change rate), wherein different text change rates in the heat map correspond to different color displays, and continuous text change rate changes correspond to continuous color transitions. The change of the text change rate can be more clearly and intuitively characterized by the heat map. Such a heat map can be used to help determine the subtitle area of the target video, for example, by directly judging which text area is the subtitle area through the superimposed heat map. As an explanation, the color and / or depth of the superimposed heat map can characterize the size of the change rate.
[0100] The above method is described below in conjunction with a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.
[0101] Sometimes there may be station logos, rolling news, and floating watermarks in the video, which will inevitably lead to a decrease in the accuracy of subtitle recognition. In this case, in order to improve the accuracy of the final subtitle recognition, it is necessary to first accurately determine the area where the subtitles are located in the video screen, that is, it is necessary to distinguish between "station logos", "rolling news", "floating watermarks", "fixed watermarks", etc., to determine which are subtitle areas and which are interference areas rather than subtitle areas. After accurately determining the subtitle area, subtitle recognition is performed on the text in the determined subtitle area, which can effectively improve the accuracy of subtitle recognition.
[0102] Furthermore, the subtitle area may be determined by using a variety of different combinations of the upper limit of the change rate threshold, the lower limit of the change rate threshold, and the average area, so as to obtain a variety of embodiments of the present application.
[0103] In some embodiments, the subtitle area can be determined using only the upper limit of the change rate threshold. In this embodiment, the subtitle area of the target video is determined based on the overall text change rate of each text area, including: obtaining a preset upper limit of the change rate threshold; removing text areas whose overall text change rate is higher than the upper limit of the change rate threshold from each text area, and using the remaining text areas as candidate subtitle areas; and selecting a target subtitle area from the candidate subtitle areas as the subtitle area of the target video.
[0104] In a further embodiment, the subtitle area can be determined by combining the upper limit of the change rate threshold with the average area. Here, the subtitle area of the target video is determined based on the overall text change rate of each text area, including: obtaining a preset upper limit of the change rate threshold; removing text areas whose overall text change rate is higher than the upper limit of the change rate threshold from each text area, and using the remaining text areas as candidate subtitle areas; selecting a target subtitle area from the candidate subtitle areas as the subtitle area of the target video, specifically including: determining the average area of each candidate subtitle area; selecting the candidate subtitle area with the largest average area from the candidate subtitle areas as the subtitle area of the target video. Here, the average area of each candidate subtitle area can be determined by the means described in the embodiments of the present application or any other suitable means.
[0105] In other embodiments, the subtitle area can be determined by combining the upper change rate threshold with the lower change rate threshold. In this embodiment, the upper change rate threshold and the lower change rate threshold that are preset are obtained; the text areas whose overall text change rate is higher than the upper change rate threshold are removed from the text areas, and the text areas whose overall text change rate is lower than the lower change rate threshold are removed, and the remaining text areas are used as the subtitle areas of the target video.
[0106] In some embodiments, the average area may be used to additionally or additionally filter text regions (including candidate subtitle regions). In this embodiment, before determining the subtitle region of the target video, the determination method further includes: determining the average area of each text region; and eliminating text regions whose average area is less than a given area threshold.
[0107] In order to accurately determine the subtitle area, a method for determining the subtitle area of a video is provided in a specific example. Figure 5 As shown, the following steps may be included:
[0108] S510: Obtain a target video file;
[0109] S520: extracting frame images from the target video file at predetermined time intervals;
[0110] S530: Perform text recognition on each frame of image to determine the text area in each frame of image and the text content in each text area;
[0111] S540: Compare text changes in adjacent frames in the same area, thereby calculating the text change rate of each text area (overall text change rate), and map the text change rate to the corresponding area of the image model to form a heat map of the image area;
[0112] S550: setting an upper limit and a lower limit of the text change rate threshold, and determining an area above the upper limit of the text change rate threshold or below the lower limit of the text change rate threshold as an invalid area, so that rolling news, rolling advertisements and station logo areas can be eliminated;
[0113] Considering that rolling news is always changing, while the station logo is always the same, for this purpose, the upper and lower limits of the text change rate threshold can be set. For those exceeding the upper limit of the text change rate threshold, they can be determined as rolling news or rolling advertisement areas, and for those below the lower limit of the text change rate threshold, they can be determined as station logos, that is, these are not subtitle areas and can be eliminated based on this method. By setting the upper and lower limits of the text change rate, rolling news (faster speed) and station logos (same speed) can be eliminated respectively.
[0114] S560: For the remaining multiple candidate regions, an average area of each candidate region may be calculated, wherein the average area may be calculated by the accumulated text change rate of the region / area of the region;
[0115] S570: Taking the candidate region with the largest average area as the recognized subtitle region.
[0116] This is mainly because the floating watermark appears in various places on the screen all the time, which makes the cumulative change rate of the text in these text areas lower (because in most cases, there is no text in this area), and the area corresponding to the area is larger, which makes the final average area smaller.
[0117] In the above example, a method for locating the subtitle area of a video is provided. First, the text in the image is identified through text recognition technology. By calculating the change rate of the text as a representation of different areas of the image, different thresholds are used to effectively exclude interference areas and finally determine the effective subtitle area. Furthermore, the calculated text change rate can be used as a feature to generate a heat map, thereby improving the accuracy of the prediction. By setting different thresholds, different types of interference areas can be filtered. For example: by setting the upper and lower limits of the text change rate, rolling news (faster speed) and station logos (same speed) can be eliminated respectively. By calculating the average area of the text box, floating and jumping watermarks can be eliminated.
[0118] In some embodiments of the present application, at the software level, such as Figure 6 As shown, the embodiment of the present application provides a device 600 for determining a video subtitle area, including:
[0119] An acquisition module 601 is used to acquire a target video;
[0120] A segmentation module 602 is used to extract multiple image frames from the target video;
[0121] The recognition module 603 is used to recognize the text area in each of the multiple image frames;
[0122] A calculation module 604 is used to calculate the text change rate of each text area in adjacent image frames;
[0123] A first determination module 605, configured to determine the overall text change rate of each text region in the plurality of image frames according to the text change rate of each text region in adjacent image frames;
[0124] The second determination module 606 is used to determine the subtitle area of the target video according to the overall text change rate of each text area.
[0125] In one embodiment, the first determination module 605 can be specifically used to accumulate and sum, or accumulate and sum and then average, the text change rates of adjacent image frames corresponding to the current text area; the accumulated sum result or the accumulated sum and then average result is used as the overall text change rate of the current text area in the multiple image frames to obtain the overall text change rate of each text area in the multiple image frames.
[0126] In one embodiment, the second determination module 606 may include: an acquisition unit, used to obtain a preset change rate threshold upper limit and change rate threshold lower limit; a removal unit, used to remove text areas whose overall text change rate is higher than the change rate threshold upper limit from the text areas, and remove text areas whose overall text change rate is lower than the change rate threshold lower limit, and use the remaining text areas as candidate subtitle areas; a determination unit, used to select a target subtitle area from the candidate subtitle areas as the subtitle area of the target video.
[0127] In one embodiment, the above-mentioned determination unit can be specifically used to calculate the average area of each candidate subtitle area in the following manner: divide the overall text change rate of the current candidate subtitle area by the area of the current text area to obtain the average area of the current candidate subtitle area; from the candidate subtitle areas, select the candidate subtitle area with the largest average area as the subtitle area of the target video.
[0128] In one embodiment, the above-mentioned recognition module 603 is specifically used to convert the current image frame into a grayscale image; perform binarization processing on each pixel in the grayscale image in an increasing order of threshold values from 0 to 255; based on the characteristic that the grayscale value of the text content is consistent, as the threshold continues to increase, the text content changes from distinguishable to indistinguishable when the threshold increases to the grayscale value of the text content, identify the boundary of the text content; merge and divide the boundaries of the text content to obtain one or more text areas in the current image frame.
[0129] In one embodiment, the calculation module 604 can be specifically used to obtain the area boundaries and text contents of each text area in the first frame of adjacent image frames; obtain the area boundaries and text contents of each text area in the second frame of adjacent image frames; for each text area in the first frame: determine the area change ratio and the number of text content transformation operation steps required to transform the current text area in the first frame to the corresponding text area in the second frame, and calculate the text change rate based on the area change ratio and the number of text content transformation operation steps.
[0130] In one embodiment, after calculating the text change rate of each text area in adjacent image frames, a large feature map of a pure color, such as pure white, can also be generated for the image frame; the text change rate of each text area is displayed in the corresponding area of the feature map in the form of a heat map, wherein different text change rates in the heat map correspond to different color displays, and continuous changes in text change rate correspond to continuous color transitions.
[0131] Figure 7 A schematic diagram of an electronic device 700 that can be used to implement the method of an embodiment of the present application or to implement an embodiment of the present application is shown. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, cloud or distributed electronic devices may also be used for implementation.
[0132] like Figure 7 As shown, the electronic device 700 includes a processor 710 and a memory 720. The processor is used to execute programs stored in the memory, and these programs can implement the methods, steps or functions described in the above embodiments when executed by the computer. The processor 710 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 710 and the memory 720 are interconnected via a bus 730. The bus 730 can also be connected to an input / output (I / O) interface, etc.
[0133] When the processor executes the computer program, the following steps are implemented:
[0134] Step 1: Get the target video;
[0135] Step 2: extracting multiple image frames from the target video;
[0136] Step 3: Identify a text area in each of the multiple image frames;
[0137] Step 4: Calculate the text change rate of each text area in adjacent image frames;
[0138] Step 5: determining the overall text change rate of each text area in the plurality of image frames according to the text change rate of each text area in adjacent image frames;
[0139] Step 6: Determine the subtitle area of the target video according to the overall text change rate of each text area.
[0140] The embodiment of the present application also provides a computer-readable storage medium capable of implementing all the steps in the method for determining a video subtitle area in the above embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps in the method for determining a video subtitle area in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:
[0141] Step 1: Get the target video;
[0142] Step 2: extracting multiple image frames from the target video;
[0143] Step 3: Identify a text area in each of the multiple image frames;
[0144] Step 4: Calculate the text change rate of each text area in adjacent image frames;
[0145] Step 5: determining the overall text change rate of each text area in the plurality of image frames according to the text change rate of each text area in adjacent image frames;
[0146] Step 6: Determine the subtitle area of the target video according to the overall text change rate of each text area.
[0147] From the above description, it can be seen that the embodiment of the present application extracts multiple image frames from the target video, then identifies the text area in each of the multiple image frames, and then calculates the text change rate of each text area in adjacent image frames. Based on the text change rate of each text area in adjacent image frames, the overall text change rate of each text area in the multiple image frames is determined, and then based on the overall text change rate of each text area, the subtitle area of the target video is determined. That is, the inter-frame text change rate characteristics of each text area are combined to accurately identify which area in the video is the subtitle area. The above solution solves the existing technical problem of being unable to accurately determine the subtitle area, resulting in too much interference content in the final generated subtitles, and achieves the technical effect of accurately locating the subtitle area to improve the recognition accuracy of the video subtitles.
[0148] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0149] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0150] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).
[0151] Although the present specification embodiment provides the method operation steps as described in the embodiment or flow chart, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way in the order of execution of many steps, and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel (such as a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment) according to the method shown in the embodiment or the accompanying drawings. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.
[0152] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0153] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.
[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0157] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0158] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0159] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0160] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0161] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0162] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.
[0163] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.
Claims
1. A method for determining a video subtitle area, characterized in that: The determination method comprises: Get the target video; Extracting a plurality of image frames from the target video; identifying a text region in each of the plurality of image frames; Calculating the text change rate of each text area in adjacent image frames; including: setting weights for each text area area and text content in adjacent image frames, respectively, and performing weighted summation of the change ratio of each text area area and the number of transformation operation steps of the text content in the adjacent image frames based on the weights to obtain the text change rate; Determining an overall text change rate of each text area in the plurality of image frames according to a text change rate of each text area in adjacent image frames; The subtitle area of the target video is determined according to the overall text change rate and the average area of each text area, including: using multiple different combinations of the change rate threshold upper limit, the change rate threshold lower limit and the average area of the overall text change rate of each text area to determine the subtitle area.
2. The method according to claim 1, characterized in that Determining the overall text change rate of each text area in the plurality of image frames according to the text change rate of each text area in adjacent image frames includes: Accumulate and sum the text change rates of adjacent image frames corresponding to the current text area or perform the cumulative summing and then averaging; The cumulative summation result or the cumulative summation and averaging result is used as the overall text change rate of the current text area in the multiple image frames to obtain the overall text change rate of each text area in the multiple image frames.
3. The method according to claim 2, characterized in that Determining the subtitle area of the target video according to the overall text change rate and the average area of each text area includes: Get the preset upper limit of the change rate threshold; Eliminate text regions whose overall text change rate is higher than the upper limit of the change rate threshold from the text regions, and use the remaining text regions as candidate subtitle regions; A target subtitle area is selected from the candidate subtitle areas as the subtitle area of the target video.
4. The method according to claim 2, characterized in that: Determining the subtitle area of the target video according to the overall text change rate and the average area of each text area includes: Get the preset upper and lower limits of the change rate threshold; Eliminate, from the text regions, text regions whose overall text change rate is higher than an upper limit of a change rate threshold, and text regions whose overall text change rate is lower than a lower limit of a change rate threshold, and use the remaining text regions as candidate subtitle regions; A target subtitle area is selected from the candidate subtitle areas as the subtitle area of the target video.
5. The method according to claim 3 or 4, characterized in that: Selecting a target subtitle area from the candidate subtitle areas as the subtitle area of the target video includes: Determine the average area of each candidate subtitle region; From the candidate subtitle regions, a candidate subtitle region with the largest average area is selected as the subtitle region of the target video.
6. The method according to claim 5, characterized in that The calculating the average area of each candidate subtitle region comprises: The average area of the current candidate subtitle region is obtained by dividing the overall text change rate of the current candidate subtitle region by the area of the current text region.
7. The method according to claim 3 or 4, characterized in that: The upper limit of the change rate threshold is determined based on the duration of the target video and a given number of words spoken per unit time.
8. The method according to claim 1, characterized in that Before determining the subtitle area of the target video, the determination method further includes: Determine the average area of each text region; Eliminate text regions whose average area is smaller than a given area threshold.
9. The method according to claim 1, characterized in that: Identifying a text region in each of the plurality of image frames comprises: Convert the current image frame to a grayscale image; Performing binarization processing on each pixel in the grayscale image according to increasing threshold values; As the threshold continues to increase, the text content changes from distinguishable to indistinguishable, so as to identify the boundary of the text content; The boundaries of the text content are merged and divided to obtain one or more text regions in the current image frame.
10. The method according to claim 1, characterized in that After calculating the text change rate of each text area in adjacent image frames, the determination method further includes: Generating a pure color and equal-size feature map for the multiple image frames; The text change rate of each text area is displayed in the corresponding area of the feature map in the form of a heat map, wherein different text change rates in the heat map correspond to different color displays, and continuous changes in the text change rate correspond to continuous color transitions.
11. A device for determining a video subtitle area, characterized in that: include: An acquisition module, used to acquire a target video; A segmentation module, used for extracting a plurality of image frames from the target video; A recognition module, used for recognizing a text area in each of the plurality of image frames; A calculation module, used for calculating the text change rate of each text area in adjacent image frames; The method includes: setting weights for the area of each text region and the text content in adjacent image frames respectively, and performing weighted summation on the change ratio of the area of each text region and the number of transformation operation steps of the text content in adjacent image frames based on the weights to obtain the text change rate; A first determination module, configured to determine an overall text change rate of each text region in the plurality of image frames according to a text change rate of each text region in adjacent image frames; The second determination module is used to determine the subtitle area of the target video according to the overall text change rate and the average area of each text area, wherein the second determination module is further used to determine the subtitle area using multiple different combinations of an upper limit of a change rate threshold, a lower limit of a change rate threshold and an average area of the overall text change rate of each text area.
12. The device according to claim 11, characterized in that The first determining module is specifically used for: Accumulate and sum the text change rates of adjacent image frames corresponding to the current text area or perform the cumulative summing and then averaging; The cumulative summation result or the cumulative summation and averaging result is used as the overall text change rate of the current text area in the multiple image frames to obtain the overall text change rate of each text area in the multiple image frames.
13. The device according to claim 12, characterized in that The second determining module comprises: An acquisition unit, used for acquiring a preset upper limit of a change rate threshold and a lower limit of a change rate threshold; A removal unit, used to remove text areas whose overall text change rate is higher than an upper limit of a change rate threshold, and remove text areas whose overall text change rate is lower than a lower limit of a change rate threshold from the text areas, and use the remaining text areas as candidate subtitle areas; A determination unit is used to select a target subtitle area from the candidate subtitle areas as the subtitle area of the target video.
14. An electronic device comprising a processor and a memory for storing instructions executable by the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 10 are implemented.
15. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
