A method and system for automatic positioning of video hard subtitle positions

Abstract

The present application relates to the technical field of video processing, and particularly relates to a video hard subtitle position automatic positioning method and system. The method first extracts key frames and identifies all suspected hard subtitle regions, determines a visual independence evaluation index based on the feature point displacement difference of the suspected hard subtitle region and the background scene and the edge pixel time sequence mutation feature, performs context semantic association analysis on the text content in the suspected region to obtain a semantic continuity evaluation index, and determines a hard subtitle confidence probability and performs positioning by comprehensively considering the visual independence evaluation index and the semantic continuity evaluation index. The present application effectively distinguishes and excludes the interference of scene original text by fusing the visual motion mutation feature and the context semantic coherence feature of the text in the video space-time sequence in multiple dimensions, realizes automatic and accurate positioning of the video hard subtitle, and improves the practicability and generalization ability of the automatic positioning of the video hard subtitle.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and specifically to a method and system for automatically locating hard subtitles in video. Background Technology

[0002] Hard captions in videos are text information layers that are manually superimposed onto the screen during the post-production editing stage of a video. They are burned into the pixels of the screen. Because they carry the key function of conveying information, the automatic extraction and recognition of hard captions has become an important part of intelligent video analysis and intelligent creation.

[0003] Existing hard caption extraction methods mostly rely on pre-defined, fixed spatial regions for segmentation and recognition. For example, they might directly designate the lower-middle area of ​​the screen as the caption retrieval zone to limit the detection range and extract text. However, these methods ignore dynamic hard captions located in other irregular positions within the screen. Furthermore, during the hard caption localization process, text inherent in the video scene can severely interfere with caption recognition. For instance, text from road signs, billboards, clothing, and other scene text within the video frame cannot effectively distinguish between artificially added hard captions and interfering text from the scene itself, leading to decreased recognition accuracy in the hard caption region and ultimately insufficient accuracy in hard caption localization. Summary of the Invention

[0004] To address the technical problem in existing technologies that fail to effectively distinguish between manually added hard-subtitles and interfering text within the scene, leading to decreased accuracy in hard-subtitle recognition, this invention aims to provide an automatic hard-subtitle location method and system for videos. The specific technical solution adopted is as follows: This invention provides a method for automatically locating hard subtitles in a video, the method comprising: Acquire consecutive frame images of the video, extract keyframes from the consecutive frame images, and identify suspected hard caption areas containing character text in the keyframes; Based on the difference in feature point displacement between the suspected hard caption region and the background scene in consecutive frames, and the abrupt change characteristics of the edge pixels of the suspected hard caption region in time, the visual independence evaluation index of the suspected hard caption region is determined. Based on the text content within the suspected hard caption region in consecutive frame images, a contextual semantic association analysis is performed on the suspected hard caption region to obtain a semantic continuity evaluation index for the suspected hard caption region. Based on the visual independence assessment index and the semantic continuity assessment index, the confidence probability of hard subtitles in suspected hard subtitle regions is determined; and the target hard subtitle position in the keyframe is determined based on the hard subtitle confidence probability.

[0005] Furthermore, the method for obtaining the suspected hard-subtitle region includes: OCR technology is used to recognize the character text in the keyframe. Each character text region is selected by using a detection box. The selected region is taken as the suspected hard text region, and the position coordinate information of the suspected hard text region in the corresponding keyframe is recorded.

[0006] Furthermore, the method for obtaining the visual independence assessment index includes: For any suspected hard caption region, the keyframe containing the suspected hard caption region and all other frames between the keyframes before and after it are taken as the neighboring image set. For the keyframe where the suspected hard caption region is located, the motion consistency metric between the suspected hard caption region and the background is determined based on the displacement difference of feature points in the adjacent image set; the distribution characteristics of the number of edge pixels of the suspected hard caption region in each frame of the adjacent image set are analyzed to determine the clarity abruptness metric of text switching in the suspected hard caption region. By combining motion consistency and clarity abruptness metrics, a visual independence assessment index for the suspected hard caption region is obtained.

[0007] Furthermore, the method for obtaining the motion consistency metric includes: In images containing suspected hard captions, a uniform grid is created, and feature points within each grid are extracted; areas that are not suspected hard captions are treated as background areas. By combining position tracking algorithms, background displacement vectors of feature points in the background region between adjacent frames and suspected region displacement vectors of feature points in the suspected hard subtitle region are obtained. Calculate the cosine similarity between the mean of the background displacement vector and the mean of the displacement vector of the suspected region between adjacent frames, and fuse the cosine similarity between all adjacent frames in the adjacent image set to obtain the motion consistency metric.

[0008] Furthermore, the method for obtaining the clear mutation metric includes: Within the suspected hard caption area, edge detection is performed on each frame of the adjacent image set, and the number of edge pixels is counted. The number of edge pixels in each frame is obtained, and the maximum inter-class variance method is used to divide the frame into a set of clear edges and a set of blurry edges based on the number of edge pixels. The sharpness abruptness metric is determined based on the proportion of frames in the edge-sharp class set.

[0009] Furthermore, the method for obtaining the semantic continuity evaluation index includes: For any suspected hard caption region, obtain multiple adjacent keyframes of the keyframe where the suspected hard caption region is located within a preset adjacent window, and use word frequency statistics to convert the text content of the keyframe where the suspected hard caption region is located and the suspected hard caption region in the adjacent keyframes into corresponding word vectors respectively. The similarity between the word vector of the keyframe containing the suspected hard caption region and the word vector of each adjacent keyframe is calculated, and the maximum similarity is used as the semantic continuity evaluation index of the suspected hard caption region.

[0010] Furthermore, the confidence probability of the hard caption is a weighted sum of the visual independence assessment index and the semantic continuity assessment index.

[0011] Furthermore, determining the target hard caption position of the keyframe based on the hard caption confidence probability includes: When the confidence probability of a hard subtitle is greater than a preset confidence threshold, the corresponding suspected hard subtitle area is determined to be a hard subtitle area, and the position coordinates of the hard subtitle area are extracted as the target hard subtitle position.

[0012] Furthermore, the method for obtaining keyframes includes: extracting multiple frames from consecutive frame images at preset time intervals as keyframes.

[0013] The present invention also provides an automatic positioning system for video hard captions, the system comprising: The suspected region extraction module is used to acquire consecutive frame images of a video, extract keyframes from the consecutive frame images, and identify suspected hard caption regions containing character text in the keyframes. The motion correlation analysis module is used to determine the visual independence evaluation index of the suspected hard caption region based on the difference in feature point displacement between the suspected hard caption region and the background scene in consecutive frames, as well as the abrupt change characteristics of the edge pixels of the suspected hard caption region in time. The semantic association analysis module is used to perform contextual semantic association analysis on the suspected hard caption regions based on the text content in the suspected hard caption regions in consecutive frame images, and obtain the semantic continuity evaluation index of the suspected hard caption regions. The hard caption localization module is used to determine the hard caption confidence probability of suspected hard caption regions based on visual independence evaluation index and semantic continuity evaluation index; and to determine the target hard caption position of keyframes based on the hard caption confidence probability.

[0014] The present invention has the following beneficial effects: This invention first captures subtitle text at any location in a video by extracting keyframes and identifying all suspected hard subtitle regions. Second, it determines a visual independence evaluation index based on the difference in feature point displacement and temporal abrupt changes in edge pixels between the suspected hard subtitle regions and the background scene, effectively eliminating interfering text that moves with the scene. Simultaneously, it obtains a semantic continuity evaluation index by performing contextual semantic association analysis on the text content within the suspected regions. This leverages the difference between hard subtitles, which are semantically coherent in time, and scene text, which is semantically random and unrelated, to improve the accuracy of hard subtitle region recognition. Finally, it combines the visual independence evaluation index and the semantic continuity evaluation index to determine the confidence probability of the hard subtitles and performs localization. This invention effectively distinguishes and eliminates interference from native scene text by multidimensionally fusing the visual motion abrupt change features and contextual semantic coherence features of text in the video's spatiotemporal sequence, achieving automatic and accurate localization of video hard subtitles and improving the practicality and generalization ability of automatic video hard subtitle localization. Attached Figure Description

[0015] Figure 1 A flowchart of a video hard caption position automatic positioning method provided in one embodiment of the present invention; Figure 2 A flowchart illustrating a method for evaluating visual independence in one embodiment of the present invention; Figure 3 This is a structural diagram of an automatic video hard caption positioning system provided in one embodiment of the present invention. Detailed Implementation

[0016] Example 1 Please see Figure 1 The diagram illustrates a flowchart of an automatic video hard caption positioning method according to an embodiment of the present invention, which includes the following steps: S1: Acquire consecutive frame images of the video, extract keyframes from the consecutive frame images, and identify suspected hard caption areas containing character text in the keyframes.

[0017] Hard captions are text information layers artificially superimposed onto the image during video post-production, and they are fundamentally different from the text on the scene in the original shooting. To locate hard captions in a video stream, it is first necessary to acquire continuous frame image data of the video to be processed. A video stream is essentially a collection of image frames arranged in a time sequence, with each frame recording the image information at that moment. Directly processing all frames would consume a large amount of computing resources and storage space, and there is a lot of redundant information between adjacent frames, so frame-by-frame processing has limited effect on improving the accuracy of hard caption location. Therefore, this embodiment of the invention extracts keyframes from continuous frame images, using keyframes as the basic unit for subsequent processing. Keyframes are image frames that are representative of the video content, which can reduce the amount of data processing while retaining the information required for hard caption recognition.

[0018] In some embodiments of the present invention, multiple frames are extracted from consecutive frame images at preset time intervals as keyframes. The preset time interval can be flexibly set according to the total frame rate of the video and the actual application scenario. For example, one frame is extracted every 0.5 seconds, that is, if the video frame rate is 30 frames per second, then one frame is extracted every 15 frames as a keyframe. This interval value is only an illustrative example, and in actual applications it can be adjusted according to the video duration, hard caption switching frequency and computing power requirements. The present invention does not limit this.

[0019] Meanwhile, keyframes may contain not only hard-coded text added later, but also text content inherent in the scene, such as road sign text, product labels, building logos, and other interfering text. To initially filter out all regions that may contain hard-coded text from the keyframes, character text detection is required. Therefore, in this embodiment of the invention, OCR technology is used to recognize character text in the keyframes. A detection box is used to select each character text region, and the selected region is taken as a suspected hard-coded text region. The position coordinates of the suspected hard-coded text region in the corresponding keyframe are recorded. That is, OCR technology is used to recognize character text in the image and return the smallest bounding rectangle of each text region. The position coordinates of this rectangle represent the spatial position of the text in the image. By processing keyframes with OCR technology, all regions containing character text in the image can be extracted as candidates, avoiding the omission of hard-coded text located in atypical positions due to preset area limitations.

[0020] S2: Based on the difference in feature point displacement between the suspected hard caption region and the background scene in consecutive frames, and the abrupt change characteristics of the edge pixels of the suspected hard caption region in time, the visual independence evaluation index of the suspected hard caption region is determined.

[0021] Because hard-coded text is an additional layer of information manually superimposed onto the video, its motion in the video frame differs significantly from the background scene. Specifically, background text within the video scene is an integral part of the overall scene. When the camera moves, rotates, or zooms, the background text moves along with the surrounding scenery, maintaining its relative position to the background. Its displacement direction and magnitude between adjacent frames are highly consistent with the background scene. In contrast, hard-coded text, as an information layer independent of the scene, has its motion pattern determined by the post-editing method. Fixed text maintains its absolute position within the video frame and does not move with the scene, while scrolling text or floating watermarks move according to preset rules, lacking correlation with the movement trend of the background scene. Based on this difference in motion patterns, a preliminary measurement and assessment of suspected hard-coded text areas can be performed visually.

[0022] Preferably, in some embodiments of the present invention, please refer to Figure 2 The diagram illustrates a flowchart of a visual independence assessment index method according to an embodiment of the present invention, which includes the following steps: S21: For any suspected hard caption region, take all other frames between the key frame where the suspected hard caption region is located and the adjacent key frames before and after it as the neighboring image set; for the key frame where the suspected hard caption region is located, determine the motion consistency metric value of the suspected hard caption region relative to the background based on the displacement difference of the feature points in the neighboring image set.

[0023] The positional changes of the same text block in multiple time-series images are compared. Since only keyframes are used for text recognition, the transition frames between keyframes can provide a complete motion trajectory change process. Therefore, all transition frames between adjacent keyframes are selected to form an adjacent image set, which fully captures the dynamic displacement rules of text and background, thereby quantifying the degree of synchronization between the two movements.

[0024] If the overall image displacement of the current keyframe and the previous adjacent keyframe is extremely small, it is determined that the current shot is a static scene with no obvious background displacement change. The displacement difference between the background and the text is not of distinguishing reference value. Therefore, in this embodiment of the invention, when the average displacement magnitude of the global feature points of the keyframe where the suspected hard subtitle area is located is less than 2 pixels compared with the previous keyframe, the motion consistency metric calculation process is skipped directly, and the visual index calculation is completed solely by relying on the edge mutation features.

[0025] For a set of adjacent images corresponding to the same suspected hard caption region, image feature points are extracted frame by frame for displacement trajectory representation. These feature points accurately reflect the direction and distance of movement in a local area. In some embodiments of this invention, a uniform grid is established in the image containing the suspected hard caption region, and feature points are extracted from each grid. The uniform grid can be divided into grid cells on the image plane according to a preset number of rows and columns, ensuring that each grid cell covers a local area of ​​the image. In one specific embodiment of this invention, the Harris corner detection algorithm can be used to extract corner points as feature points within each grid. Harris corner detection has good robustness to image translation, rotation, and illumination changes, providing a stable and reliable feature point foundation. The size of the uniform grid can be adjusted according to the image resolution and computational accuracy requirements, for example, a 30×30 grid size can be used. This size is only an example and is not intended to be limiting.

[0026] The non-suspected hard caption areas are designated as the background area. The background area encompasses all visual elements inherent to the scene within the video frame, and the motion trends of its feature points reflect the scene's motion patterns. A position tracking algorithm is used to obtain the background displacement vectors of feature points within the background area between adjacent frames, and the suspected region displacement vectors of feature points within the suspected hard caption areas. The position tracking algorithm can employ optical flow or feature point matching methods. By tracking the same feature point between adjacent frames, the coordinate offset of that feature point between the two frames is calculated, and this offset constitutes the feature point's displacement vector. Specifically, for each feature point in the background area, its coordinate difference in adjacent frames is used as its background displacement vector; similarly, for each feature point in the suspected hard caption area, its suspected region displacement vector is obtained. Both the background displacement vector and the suspected region displacement vector are two-dimensional vectors, representing the direction and magnitude of movement of the corresponding feature point on the image plane, respectively.

[0027] The cosine similarity between the mean background displacement vector and the mean displacement vector of the suspected region is further calculated, and then fused with the cosine similarity between all adjacent frames in the adjacent image set to obtain the motion consistency metric. The mean background displacement vector is the arithmetic mean of the displacement vectors of all feature points in the background region, and this mean vector represents the overall motion direction and amplitude of the background scene. The mean displacement vector of the suspected region is the arithmetic mean of the displacement vectors of all feature points in the suspected hard caption region, and represents the overall motion direction and amplitude of the suspected hard caption region. If there are cases where the mean background displacement vector or the mean displacement vector of the suspected region is zero, the motion consistency metric calculation process is skipped, and the visual index is calculated solely based on edge abrupt change features.

[0028] Cosine similarity measures the directional similarity between two vectors by calculating the cosine of the angle between them. Its value ranges from -1 to 1, with values ​​closer to 1 indicating more consistent directions. If the movement direction of a suspected hard-subtitle region is highly consistent with the movement direction of the background scene, the cosine similarity is close to 1, indicating that the region moves along with the background scene and is more likely to be interfering text within the scene. If the two directions differ significantly or are opposite, the cosine similarity is low or even negative, indicating that the movement of the region is independent of the background scene and is more likely to be hard-subtitle.

[0029] In one specific embodiment of the present invention, the mean cosine similarity among all adjacent frames in an adjacent image set is used as the motion consistency metric. As an example, the expression for the motion consistency metric of the suspected hard caption region is: In the formula, This is represented as the motion consistency metric for the suspected hard-subtitle area. It represents the total number of image frames in a set of adjacent images. This represents the frame number excluding the first frame in the adjacent image set. Represented as the first The mean of the background displacement vector of a frame relative to the previous frame. Represented as the first The mean of the suspected region displacement vectors relative to the previous frame. This is expressed as cosine similarity. When... When the value is close to 1, it indicates that the movement trend of the suspected hard-subtitle area is highly consistent with the background scene. When the value is close to -1 or 0, it indicates that there is a significant difference in the movement trends of the two.

[0030] S22: Analyze the distribution characteristics of the number of edge pixels in each frame of the suspected hard caption region in the adjacent image set, and determine the clear abrupt change measure of text switching in the suspected hard caption region.

[0031] Hard-coded text and background noise also exhibit significant differences in their appearance and disappearance patterns. Background noise is inherent to the scene itself during video recording. When the scene changes or camera parameters change, the clarity of the background noise typically changes gradually with camera movement, focal length changes, or lighting changes, with the edge sharpness transitioning slowly and continuously from blurry to sharp or from sharp to blurry. Conversely, hard-coded text is an information layer superimposed onto video frames in post-production. To meet the requirements of smooth reading for the human eye, hard-coded text usually appears suddenly in the frame in a complete and clear form, maintaining a stable display with high edge sharpness. It then disappears abruptly when switching to the next subtitle, exhibiting significant abrupt changes in its appearance and disappearance. Based on this difference, the degree of abrupt change is quantified by analyzing the distribution characteristics of the number of edge pixels in suspected hard-coded text areas across frames.

[0032] Preferably, in some embodiments of the present invention, within the suspected hard caption area, edge detection is performed on each frame of the adjacent image set, and the number of edge pixels is counted. Specifically, within the rectangular area defined by the suspected hard caption area, Sobel operator edge detection is performed on each frame of the adjacent image set, the total number of all edge pixels in a single frame area is counted, the total number of edge pixels corresponding to each transition frame is stored frame by frame, and a temporal sequence of edge pixel counts is generated.

[0033] After obtaining the number of edge pixels in each frame, the variance of the number of edge pixels in all frames of the adjacent image set is first calculated. If this variance is less than a preset fluctuation threshold, it indicates that the video image is in a stable, no-switching period. In this case, the subsequent Otsu's method is skipped, and all frames in the adjacent image set are directly classified into the sharp-edge class set. The preset fluctuation threshold can be set to 2, and the specific value can be adjusted by the implementer and is not restricted here.

[0034] The Otsu's method (maximum inter-class variance) is used to divide the data sequence into two classes based on the number of edge pixels: a sharp edge class and a blurry edge class. Otsu's method is an adaptive threshold determination method that divides the data sequence into two classes according to a certain threshold, maximizing the inter-class variance between the two classes to achieve the optimal classification effect. In one specific embodiment of this invention, for the edge pixel number sequence, an optimal segmentation threshold is automatically determined using Otsu's method. Frames with an edge pixel number greater than or equal to the threshold are assigned to the sharp edge class, while frames with an edge pixel number less than the threshold are assigned to the blurry edge class.

[0035] Finally, based on the proportion of frames in the sharp edge category, a sharpness abruptness metric is determined. This metric represents the proportion of frames with sharp text edges in adjacent image sets. If hard caption switching exhibits abrupt changes, sharp frames will constitute the vast majority, while blurry frames will only appear in a very small number of frames at the moment of switching. The proportion of sharp frames will approach 1. The higher the proportion of sharp frames, the stronger the abruptness of the switching in the suspected hard caption region, and the greater the likelihood that it belongs to hard captions.

[0036] As an example, the expression for the clarity abruptness metric of the suspected hard caption region is: ; This is represented by a measure of the clarity abruptness of the suspected hard-subtitle area. Represented as the number of frames in the set of frames with sharp edges. This represents the number of frames in the set of edge-blurred classes.

[0037] S23: By fusing motion consistency and clarity abruptness metrics, a visual independence assessment index is obtained for the suspected hard caption region.

[0038] The visual independence of suspected hard-subtitle regions was characterized from different dimensions using motion consistency and sharpness abruptness metrics. Motion consistency metric reflects the degree of consistency between the suspected hard-subtitle region and the background scene in terms of motion trends; a smaller metric indicates stronger motion independence relative to the background. Sharpness abruptness metric reflects the degree of abruptness in text switching within the suspected hard-subtitle region; a larger metric indicates more significant abrupt changes in text switching characteristics. By combining these two metrics, the likelihood of a suspected hard-subtitle region visually constituting hard subtitles can be comprehensively assessed from both motion independence and abruptness characteristics.

[0039] In one specific embodiment of the present invention, since a higher motion consistency metric indicates that the text moves synchronously with the background and is more likely to belong to the scene text, the motion consistency metric is subtracted from 1 and divided by 2 to eliminate interference caused by synchronization. This result is then multiplied by the clarity abruptness metric to obtain the visual independence evaluation index. As an example, the expression for the visual independence evaluation index of the suspected hard-subtitle area is: In the formula, This is represented by the visual independence assessment index for the suspected hard-subtitle area. This is represented by a measure of the clarity abruptness of the suspected hard-subtitle area. This is represented as the motion consistency metric for the suspected hard-subtitle area.

[0040] because The range of its value is -1 to 1. The range of its value is from 0 to 1, and its value varies. The factor increases as the degree of motion independence increases, meaning the value of this factor increases as the degree of motion independence increases. The value range is 0 to 1. When the motion consistency is low and the abruptness is high... A value close to 1 indicates that the suspected hard-subtitle area is highly likely to be a hard-subtitle at the visual level; when the motion consistency is high or the abruptness is low, A value close to 0 indicates that the area is more likely to be visually distracting text.

[0041] If only a clear mutation measure is obtained, then the clear mutation measure is directly used as the visual independence assessment metric.

[0042] S3: Based on the text content within the suspected hard caption region in consecutive frame images, perform contextual semantic association analysis on the suspected hard caption region to obtain the semantic continuity evaluation index of the suspected hard caption region.

[0043] On the other hand, hard-subtitles and scene interference text differ significantly in semantics. Hard-subtitles are an information layer added artificially during post-production, typically containing dialogue, narration, or explanatory text to serve narrative needs. Within a video, the information expressed by hard-subtitles is continuous and logically related; the text in hard-subtitles between consecutive frames or adjacent frames often semantically revolves around the same theme or dialogue, and this semantic connection persists throughout the video's playback. Conversely, scene interference text consists of text that appears incidentally in the frame, its content only related to the specific scene being filmed in that frame. When the video cuts or the scene changes, the text content in different scenes lacks semantic connection. Based on this difference, we conduct contextual semantic association analysis on the text content of suspected hard-subtitle areas to further assess the likelihood of hard-subtitle regions from a semantic perspective.

[0044] Preferably, in some embodiments of the present invention, the method for obtaining the semantic continuity evaluation index includes: For any suspected hard caption region, multiple adjacent keyframes within a preset adjacent window are obtained from the keyframe containing the suspected hard caption region. Semantic analysis requires a sufficient number of context samples to determine the degree of correlation between texts. In one specific embodiment of this invention, the preset adjacent window contains no less than 30 adjacent keyframes before and after the keyframe. Its size is a preferred choice that strikes a balance between ensuring the statistical effectiveness of semantic analysis and controlling computational overhead. If the number of keyframes obtainable within the adjacent window is less than 30, for example, if it is located at the beginning or end of the video, the actual number of keyframes is used for processing, and the implementer can adjust it as needed.

[0045] Furthermore, the word frequency-inverse text frequency (IF-IMF) algorithm can be used to convert the text content of the keyframe containing the suspected hard caption region, as well as the suspected hard caption regions in adjacent keyframes, into corresponding word vectors. The IF-IMF algorithm is a text feature representation method. The word frequency component counts the frequency of each word in a single frame of text, reflecting the importance of the word in the current frame. The inverse text frequency component counts the reciprocal of the frequency of each word in all frames of text, reflecting the word's discriminative power across the entire text set. The higher the frequency of a word in a frame and the lower its frequency across all frames, the greater its contribution to the semantic features of that frame. Through this algorithm, the text content of the suspected hard caption region in each frame is converted into a word vector in a high-dimensional space. Each dimension of this vector corresponds to a weight value for a word, and the position of the word vector in space represents the semantic features of the text in that frame.

[0046] The similarity between the word vectors of the keyframe containing the suspected hard caption region and the word vectors of each adjacent keyframe is calculated. In one specific embodiment of the invention, the similarity between word vectors can be measured using cosine similarity, i.e., the cosine of the angle between two word vectors is calculated. A higher value indicates a higher degree of semantic coherence between the two text segments. The maximum value reflects the degree of semantic association between the keyframe and the most relevant frame in the context. If the maximum value is large, it indicates that the text content of the frame has a semantically related corresponding content in the context, and the region is more likely to be a hard caption. If the maximum value is small, it indicates that the text content of the frame has no significant semantic association with the context, and the region is more likely to be interfering text that appears accidentally in the scene.

[0047] It should be noted that if no text content is recognized in a frame, its word vector is a zero vector. In this case, the cosine similarity is undefined, and the similarity of the frame is directly assigned to 0.

[0048] S4: Determine the hard caption confidence probability of the suspected hard caption region based on the visual independence evaluation index and the semantic continuity evaluation index; determine the target hard caption position of the keyframe based on the hard caption confidence probability.

[0049] This study integrates visual independence and semantic continuity assessment metrics, which independently evaluate the likelihood of suspected hard-subtitle regions belonging to hard-subtitles from the perspectives of visual motion and semantic association, respectively. The visual independence assessment metric, based on inter-frame motion features and edge abrupt changes, effectively identifies text regions with inconsistent motion and abrupt transitions compared to the background. The semantic continuity assessment metric, based on cross-frame semantic association, effectively identifies text regions with semantic coherence within the context. Both metrics provide discriminative information from different dimensions, and their respective criteria are independent; their fusion yields the final evaluation result.

[0050] Considering the varying contributions of visual and semantic features to hard caption discrimination across different video content—for example, when captions consist of meaningless symbols or short onomatopoeic words, the discriminative power of semantic features is relatively limited, thus visual independence features should have a higher discriminative weight; conversely, when captions are long, coherent dialogues, the discriminative power of semantic features is significantly enhanced, thus the contribution of semantic continuity features to discrimination should be correspondingly increased—this invention uses the weighted sum of the visual independence evaluation index and the semantic continuity evaluation index as the hard caption confidence probability. By assigning weight coefficients to each evaluation index, the contribution ratio of the two in the final confidence probability is adjusted to adapt to the discrimination requirements of different video types and application scenarios.

[0051] As an example, the confidence probability of hard captions in this suspected hard caption region. The expression is: ; This is represented by the visual independence assessment index for the suspected hard-subtitle area. This is represented as a semantic continuity assessment index for the suspected hard-subtitle region. The weighting coefficients of the visual independence assessment index are represented. The weighting coefficients represent the semantic continuity evaluation metrics. and All are real numbers greater than 0 and less than 1, and and The sum equals 1. The specific values ​​of the weighting coefficients can be flexibly configured based on the credibility of visual and semantic features in the actual application scenario. For example, in a typical film and television material configuration, the following settings can be used: It is 0.6. The value is 0.4; in videos with drastic scene changes and significant differences between the hard subtitles and background motion, the value can be appropriately increased. The value of can be adjusted to increase the contribution of the visual independence assessment index to the confidence probability. For example, in short videos with many dynamic shots, a value of can be set. It is 0.7. The value is 0.3; in videos with relatively stable scenes and hard-subtitles appearing continuously in the form of long dialogue sentences, the value can be appropriately increased. The value of can be adjusted to increase the contribution of semantic continuity assessment metrics to confidence probability. For example, in static interview videos of dialogue types, the value can be set to . It is 0.4. The value is 0.6, and the implementer can adjust the specific value according to the specific implementation scenario.

[0052] When finally determining the location of hard-subtitles, the true hard-subtitle regions are selected based on confidence probability. In some embodiments of the present invention, when the confidence probability of hard-subtitles is greater than a preset confidence threshold, the corresponding suspected hard-subtitle region is determined to be a hard-subtitle region. In this embodiment of the present invention, the preset confidence threshold is set to 0.3. The preset confidence threshold is an optimal balance point determined through statistical analysis based on a large amount of experimental data. In practical applications, it can be adaptively adjusted according to the recognition needs of specific scenarios. For example, the threshold can be appropriately increased in scenarios that pursue high accuracy, and appropriately decreased in scenarios that pursue high recall. There are no restrictions on this.

[0053] Finally, the position coordinates of the hard subtitle region are extracted as the target hard subtitle location. That is, when a suspected hard subtitle region is identified as a hard subtitle region, the position coordinates of the detection box in the keyframe of that region are directly obtained. These coordinates are the target location of the hard subtitle in the video frame. The target location can be used for subsequent pixel-level subtitle extraction, subtitle content recognition, or subtitle erasure applications, providing accurate hard subtitle region positioning results for intelligent video processing.

[0054] In summary, this invention first captures subtitle text at any location in a video by extracting keyframes and identifying all suspected hard subtitle regions. Secondly, it determines a visual independence evaluation index based on the difference in feature point displacement and temporal abrupt changes in edge pixels between the suspected hard subtitle region and the background scene, effectively eliminating interfering text that moves with the scene. Simultaneously, it obtains a semantic continuity evaluation index by performing contextual semantic association analysis on the text content within the suspected region. Utilizing the difference between hard subtitles and scene text—which are semantically coherent in time—it improves the accuracy of hard subtitle region recognition. Finally, it determines the confidence probability of hard subtitles and performs localization by combining the visual independence evaluation index and the semantic continuity evaluation index. This invention effectively distinguishes and eliminates interference from native scene text by multidimensionally fusing the visual motion abrupt change features and contextual semantic coherence features of text in the video's spatiotemporal sequence, achieving automatic and accurate localization of video hard subtitles and improving the practicality and generalization ability of automatic video hard subtitle localization.

[0055] Example 2 This invention provides an automatic video hard caption positioning system. Please refer to [link / reference]. Figure 3 The diagram shows a structural diagram of an automatic video hard caption location system provided by an embodiment of the present invention. The system includes: a suspected region extraction module 201, a motion correlation analysis module 202, a semantic correlation analysis module 203, and a hard caption location module 204.

[0056] The suspected region extraction module is used to acquire consecutive frame images of a video, extract keyframes from the consecutive frame images, and identify suspected hard caption regions containing character text in the keyframes. The motion correlation analysis module is used to determine the visual independence evaluation index of the suspected hard caption region based on the difference in feature point displacement between the suspected hard caption region and the background scene in consecutive frames, as well as the abrupt change characteristics of the edge pixels of the suspected hard caption region in time. The semantic association analysis module is used to perform contextual semantic association analysis on the suspected hard caption regions based on the text content in the suspected hard caption regions in consecutive frame images, and obtain the semantic continuity evaluation index of the suspected hard caption regions. The hard caption localization module is used to determine the hard caption confidence probability of suspected hard caption regions based on visual independence evaluation index and semantic continuity evaluation index; and to determine the target hard caption position of keyframes based on the hard caption confidence probability.

[0057] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. Since the specific implementation process of the automatic video hard caption positioning system in this embodiment is the same as the specific implementation process of the automatic video hard caption positioning method described above, it will not be described in detail here.

Claims

1. A method for automatically locating hard subtitles in a video, characterized in that, The method includes: Acquire consecutive frame images of the video, extract keyframes from the consecutive frame images, and identify suspected hard caption areas containing character text in the keyframes; Based on the difference in feature point displacement between the suspected hard caption region and the background scene in consecutive frames, and the abrupt change characteristics of the edge pixels of the suspected hard caption region in time, the visual independence evaluation index of the suspected hard caption region is determined. Based on the text content within the suspected hard caption region in consecutive frame images, a contextual semantic association analysis is performed on the suspected hard caption region to obtain a semantic continuity evaluation index for the suspected hard caption region. Based on the visual independence assessment index and the semantic continuity assessment index, the confidence probability of hard subtitles in suspected hard subtitle regions is determined; and the target hard subtitle position in the keyframe is determined based on the hard subtitle confidence probability.

2. The method for automatically locating hard subtitles in a video according to claim 1, characterized in that, The method for obtaining the suspected hard-subtitle region includes: OCR technology is used to recognize the character text in the keyframe. Each character text region is selected by using a detection box. The selected region is taken as the suspected hard text region, and the position coordinate information of the suspected hard text region in the corresponding keyframe is recorded.

3. The method for automatic positioning of video hard subtitles according to claim 1, characterized in that, The methods for obtaining the visual independence assessment index include: For any suspected hard caption region, the keyframe containing the suspected hard caption region and all other frames between the keyframes before and after it are taken as the neighboring image set. For the keyframe where the suspected hard caption region is located, the motion consistency metric between the suspected hard caption region and the background is determined based on the displacement difference of feature points in the adjacent image set; the distribution characteristics of the number of edge pixels of the suspected hard caption region in each frame of the adjacent image set are analyzed to determine the clarity abruptness metric of text switching in the suspected hard caption region. By combining motion consistency and clarity abruptness metrics, a visual independence assessment index for the suspected hard caption region is obtained.

4. The method for automatically locating video hard subtitles according to claim 3, characterized in that, The method for obtaining the motion consistency metric includes: In images containing suspected hard captions, a uniform grid is created, and feature points within each grid are extracted; areas that are not suspected hard captions are treated as background areas. By combining position tracking algorithms, background displacement vectors of feature points in the background region between adjacent frames and suspected region displacement vectors of feature points in the suspected hard subtitle region are obtained. Calculate the cosine similarity between the mean of the background displacement vector and the mean of the displacement vector of the suspected region between adjacent frames, and fuse the cosine similarity between all adjacent frames in the adjacent image set to obtain the motion consistency metric.

5. The method for automatically locating video hard subtitles according to claim 3, characterized in that, The method for obtaining the clear mutation metric includes: Within the suspected hard caption area, edge detection is performed on each frame of the adjacent image set, and the number of edge pixels is counted. The number of edge pixels in each frame is obtained, and the maximum inter-class variance method is used to divide the frame into a set of clear edges and a set of blurry edges based on the number of edge pixels. The sharpness abruptness metric is determined based on the proportion of frames in the edge-sharp class set.

6. The method for automatically locating video hard subtitles according to claim 1, characterized in that, The method for obtaining the semantic continuity evaluation index includes: For any suspected hard caption region, obtain multiple adjacent keyframes of the keyframe where the suspected hard caption region is located within a preset adjacent window, and use word frequency statistics to convert the text content of the keyframe where the suspected hard caption region is located and the suspected hard caption region in the adjacent keyframes into corresponding word vectors respectively. Calculate the similarity between the word vector of the keyframe containing the suspected hard caption region and the word vectors of each adjacent keyframe, and use the maximum similarity as the semantic continuity evaluation index of the suspected hard caption region.

7. The method for automatically locating hard subtitles in a video according to claim 1, characterized in that, The confidence probability of the hard caption is the weighted sum of the visual independence assessment index and the semantic continuity assessment index.

8. The method for automatically locating video hard subtitles according to claim 1, characterized in that, Determining the target hard caption position of a keyframe based on hard caption confidence probability includes: When the confidence probability of a hard subtitle is greater than a preset confidence threshold, the corresponding suspected hard subtitle area is determined to be a hard subtitle area, and the position coordinates of the hard subtitle area are extracted as the target hard subtitle position.

9. The method for automatically locating hard subtitles in a video according to claim 1, characterized in that, The method for obtaining keyframes includes: extracting multiple frames from consecutive frame images at preset time intervals as keyframes.

10. An automatic positioning system for video hard subtitles, characterized in that, This system is used to implement the automatic positioning method for video hard captions according to any one of claims 1-9, and the system includes: The suspected region extraction module is used to acquire consecutive frame images of a video, extract keyframes from the consecutive frame images, and identify suspected hard caption regions containing character text in the keyframes. The motion correlation analysis module is used to determine the visual independence evaluation index of the suspected hard caption region based on the difference in feature point displacement between the suspected hard caption region and the background scene in consecutive frames, as well as the abrupt change characteristics of the edge pixels of the suspected hard caption region in time. The semantic association analysis module is used to perform contextual semantic association analysis on the suspected hard caption regions based on the text content in the suspected hard caption regions in consecutive frame images, and obtain the semantic continuity evaluation index of the suspected hard caption regions. The hard caption localization module is used to determine the hard caption confidence probability of suspected hard caption regions based on visual independence evaluation index and semantic continuity evaluation index; and to determine the target hard caption position of keyframes based on the hard caption confidence probability.

Images