A subtitle elimination method and device, electronic equipment and storage medium

Abstract

The application provides a subtitle elimination method and device, electronic equipment and storage medium, and relates to the field of video processing. The subtitle area in the video image frame can be determined, the subtitle area features are obtained by performing feature extraction on the subtitle area image, and the shot switching identifier corresponding to the video image frame is determined according to the image difference between adjacent subtitle area images. Then, the fusion weight of each video image frame can be dynamically determined according to the shot switching identifier, the fusion features are obtained by fusing the subtitle area features and the motion compensation features extracted from the video image frame according to the fusion weight, and the attention mechanism processing is performed on the fusion features of adjacent video image frames to obtain the time sequence fusion features. Furthermore, the initial repair features can be obtained by decoding the time sequence fusion features, the subtitle area in the initial repair features is corrected by using the subtitle area features, and the repaired video image frame with the eliminated subtitles is generated by using the corrected initial repair features, so that the subtitle elimination quality can be improved.

Description

Technical Field

[0001] This invention relates to the field of video processing, and in particular to a method, apparatus, electronic device, and storage medium for removing subtitles. Background Technology

[0002] With the popularization of video content creation, high-quality subtitle erasure has become a necessity in industries such as film and television post-production, online education, and short video production. However, in related technologies, after text erasure is prone to ghosting and flickering, which are particularly prominent during scene transitions. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, electronic device, and storage medium for removing subtitles in video images, which can reduce ghosting and flickering caused by scene switching and improve the subtitle removal effect.

[0004] To solve the above-mentioned technical problems, the present invention provides a subtitle removal method, comprising:

[0005] Obtain video image frames and their corresponding caption masks;

[0006] Motion-compensated features are obtained by performing feature extraction, optical flow estimation, and motion compensation on the video image frames and the subtitle mask;

[0007] The subtitle region in the video image frame is determined using the subtitle mask, and the subtitle region features are extracted from the subtitle region image. The subtitle region features are then determined based on the image differences between adjacent subtitle region images. The subtitle region feature is used to determine the scene switching identifier corresponding to the video image frame. The scene switching identifier indicates whether a scene switching occurs or not.

[0008] The fusion weight of the video image frame is determined according to the scene switching identifier. The motion compensation feature and the subtitle region feature are fused according to the fusion weight to obtain the fusion feature. Attention mechanism processing is applied to the fusion features of adjacent video image frames to obtain the temporal fusion feature.

[0009] The temporal fusion features are decoded to obtain initial repair features. The subtitle region features are used to correct the subtitle region in the initial repair features. The corrected initial repair features are then used to generate repaired video image frames with subtitles removed.

[0010] Optionally, determining the subtitle region in the video image frame using the subtitle mask includes:

[0011] Determine the minimum bounding matrix corresponding to the subtitle mask, and determine the position coordinates of the minimum bounding matrix in the video image frame; wherein, the position coordinates include the center coordinates, the top left corner coordinates, and the top right corner coordinates;

[0012] In a preset number of consecutive video image frames, the standard deviation of the center coordinates is calculated. If the standard deviation is determined to be less than a preset value, the average value, average width, and average height of the center coordinates in the consecutive video image frames are determined using the position coordinates in the consecutive video image frames.

[0013] The subtitle area is determined based on the average center coordinates, the average width, and the average height.

[0014] Optionally, the step of extracting features from the subtitle region image to obtain subtitle region features includes:

[0015] The subtitle region image is used to extract features using a convolutional layer to obtain a global feature vector;

[0016] The average color value of each color channel in the subtitle region image is determined, and the average color value is concatenated with the global feature vector to obtain the subtitle region feature.

[0017] Optionally, determining the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images includes:

[0018] Determine the structural similarity and color histogram difference values ​​between adjacent subtitle region images, and use the motion compensation features to determine the average optical flow amplitude in the subtitle region;

[0019] Determine whether the structural similarity value, the color histogram difference value, and the average optical flow amplitude meet preset segmentation conditions; wherein, the preset segmentation conditions include the structural similarity value being less than a first preset value, the color histogram difference value being greater than a second preset value, and the average optical flow amplitude being less than a third preset value.

[0020] If the conditions are met, then set a scene switching flag indicating the presence of scene switching for the current video image frame;

[0021] If the condition is not met, then a scene switching flag indicating that there is no scene switching is set for the current video image frame.

[0022] Optionally, determining the fusion weight of the video image frame based on the scene switching identifier includes:

[0023] Obtain the current scene switching identifier and current frame number of the current video frame, and determine the historical frame number of the most recent scene switching before the current video frame based on the scene switching identifier of each video image frame, and determine the interval value between the current frame number and the historical frame number.

[0024] If it is determined that the current scene switching indicator indicates that there is no scene switching and the interval value is not within the preset interval range, then the fusion weight is set to the first weight value; wherein, the first weight value is a preset value;

[0025] If it is determined that the current scene switching flag indicates that a scene switching has occurred, then the fusion weight is set to a second weight value; wherein the second weight value is a preset value and is less than the first weight value;

[0026] If it is determined that the current scene switching identifier indicates that there is no scene switching and the interval value is within a preset interval range, then the fusion weight is set to a third weight value between the second weight value and the first weight value using a preset linear function and the interval value.

[0027] Optionally, fusing the motion compensation features and the subtitle region features according to the fusion weights to obtain the fused features includes:

[0028] Determine the average subtitle region features corresponding to a preset number of consecutive video image frames;

[0029] Based on the fusion weight, the average subtitle region features are fused into the subtitle region in the motion compensation features to obtain the fused features;

[0030] Gain processing is applied to the subtitle region in the fusion feature using a gain coefficient.

[0031] Optionally, the step of correcting the subtitle region in the initial repair features using the subtitle region features includes:

[0032] Determine the average subtitle region features corresponding to a preset number of consecutive video image frames, and extract the features to be compared from the initial repair features located in the subtitle region;

[0033] Determine the feature difference value between the average subtitle region feature and the feature to be compared, and determine whether the feature difference value is greater than a preset difference threshold;

[0034] If it is greater than the average subtitle region feature, the features to be compared are weighted and fused to correct the subtitle region in the initial repair features.

[0035] If it is not greater than, then the feature to be compared is retained.

[0036] Optionally, the step of extracting features and estimating optical flow from the video image frame and the subtitle mask to obtain motion compensation features includes:

[0037] Feature extraction is performed on the video image frames and the subtitle mask to obtain feature maps;

[0038] Optical flow estimation is performed on the feature maps of the current video image frame, the previous video image frame, and the next video image frame to obtain a bidirectional optical flow field.

[0039] The feature maps of the previous video image frame and the next video image frame are spatially aligned using the bidirectional optical flow field to obtain the motion compensation features.

[0040] The present invention also provides a subtitle removal device, comprising:

[0041] The acquisition module is used to acquire video image frames and their corresponding subtitle masks;

[0042] The motion compensation feature extraction module is used to extract features, estimate optical flow, and perform motion compensation on the video image frames and the subtitle mask to obtain motion compensation features.

[0043] A fixed-region timing enhancement module is used to determine the subtitle region in the video image frame using the subtitle mask, extract features from the subtitle region image to obtain subtitle region features, and determine the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images; wherein, the scene switching identifier indicates whether a scene switching exists or not.

[0044] The feature propagation module is used to determine the fusion weight of the video image frame according to the scene switching identifier, fuse the motion compensation feature and the subtitle region feature according to the fusion weight to obtain the fusion feature, and perform attention mechanism processing on the fusion features of adjacent video image frames to obtain the temporal fusion feature;

[0045] The repair generation module is used to decode the temporal fusion features to obtain initial repair features, use the subtitle region features to correct the subtitle region in the initial repair features, and use the corrected initial repair features to generate repaired video image frames with subtitles removed.

[0046] The present invention also provides an electronic device, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor for executing the computer program to implement the subtitle removal method as described above.

[0049] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the subtitle removal method described above.

[0050] This invention provides a subtitle removal method, comprising: acquiring a video image frame and a corresponding subtitle mask; performing feature extraction, optical flow estimation, and motion compensation on the video image frame and the subtitle mask to obtain motion compensation features; determining the subtitle region in the video image frame using the subtitle mask, extracting features from the subtitle region image to obtain subtitle region features, and determining a scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images; wherein the scene switching identifier indicates whether a scene switching exists or not; determining a fusion weight for the video image frame based on the scene switching identifier, fusing the motion compensation features and the subtitle region features based on the fusion weight to obtain fusion features, and performing attention mechanism processing on the fusion features of adjacent video image frames to obtain temporal fusion features; decoding the temporal fusion features to obtain initial repair features, correcting the subtitle region in the initial repair features using the subtitle region features, and generating a repaired video image frame with removed subtitles using the corrected initial repair features.

[0051] The beneficial effects of this invention are as follows: First, this invention can acquire video image frames and corresponding subtitle masks, and then perform feature extraction, optical flow estimation, and motion compensation on the video image frames and subtitle masks to obtain motion compensation features. Subsequently, to explicitly utilize the prior knowledge that "subtitles are located in a fixed position," this invention can use the subtitle mask to determine the subtitle region in the video image frame, extract features from the subtitle region image to obtain subtitle region features, and determine the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images; wherein, the scene switching identifier indicates whether a scene switching exists or not. That is, this invention can determine the fixed region where the subtitles are located, extract subtitle region features, and determine whether a scene switching exists in the video image frame. Subsequently, the fusion weight of the video image frame can be determined based on the scene switching identifier, and the motion compensation features and subtitle region features can be fused according to the fusion weight to obtain fused features. Furthermore, the fusion features of adjacent video image frames are processed using an attention mechanism to obtain temporal fusion features, i.e., the fusion weight can be dynamically determined according to the scene switching situation to improve the feature fusion effect. Subsequently, the temporal fusion features can be decoded to obtain the initial repair features. The subtitle region features are used to correct the subtitle region in the initial repair features. The corrected initial repair features are then used to generate repaired video image frames with removed subtitles. This can reduce ghosting and flickering in the subtitle region when there are scene transitions in the video image frames, thereby improving the quality of subtitle removal.

[0052] The present invention also provides a subtitle removal device, an electronic device, and a storage medium, which have the above-mentioned beneficial effects. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 A flowchart of a subtitle removal method provided in an embodiment of the present invention;

[0055] Figure 2 A structural block diagram of a subtitle removal system framework provided in an embodiment of the present invention;

[0056] Figure 3 This is a structural block diagram of a subtitle removal device provided in an embodiment of the present invention;

[0057] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] With the popularization of video content creation, high-quality subtitle erasure has become a necessity in industries such as film and television post-production, online education, and short video production. However, in related technologies, "ghosting" and "flickering" are prone to occur after subtitle erasure, and these problems are particularly prominent during scene transitions. This is because the design of related subtitle erasure algorithms (such as Propainter) is intended to handle the erasure of moving objects in dynamic scenes. Their temporal consistency relies on motion estimation and feature matching of adjacent frames, which cannot adapt to the static characteristics of fixed areas. After a scene transition, the background content of the fixed area where the subtitle is located will suddenly change (such as from a static ground to the sky). At this time, there is no effective motion trajectory between adjacent frames, the traditional motion estimation mechanism fails, and the algorithm cannot correlate the background information of the fixed area before and after the scene transition.

[0060] In view of this, in order to improve the effect of subtitle removal and avoid the ghosting and flickering caused by scene switching, the present invention can provide a subtitle removal method. It can explicitly utilize the prior knowledge that "subtitles are located in a fixed position", specifically extract the subtitle area features corresponding to the subtitle area, and dynamically fuse the subtitle area features with the complete features of the video image frame according to the scene switching between video image frames, effectively solving the problems of "ghosting" and "flickering" caused by scene switching, thereby improving the subtitle removal effect.

[0061] For easier understanding, please refer to Figure 1 , Figure 1 A flowchart of a subtitle removal method provided in an embodiment of the present invention may include:

[0062] S10. Obtain video image frames and corresponding subtitle masks.

[0063] In this step, video frames are extracted from the video file containing the subtitles to be removed, and the subtitle mask within each frame is extracted. The subtitle mask is a region mask corresponding to the subtitle area in each video frame, indicating the subtitle content that needs to be erased in each frame. The subtitle mask can be extracted using a subtitle detection algorithm.

[0064] S20. Perform feature extraction, optical flow estimation, and motion compensation on video image frames and subtitle masks to obtain motion-compensated features.

[0065] In this step, feature extraction, optical flow estimation, and motion compensation can be performed sequentially on the video image frame and the subtitle mask to obtain motion compensation features, which can be used as one of the basic image features required for subtitle removal.

[0066] Specifically, motion compensation features are obtained by extracting features and estimating optical flow from video image frames and caption masks, which may include:

[0067] Step S21: Extract features from video image frames and subtitle masks to obtain feature maps.

[0068] In this step, multi-scale feature maps of video image frames and caption masks can be extracted using an 8-layer convolutional neural network (CNN) (channel count 64→512), outputting intra-frame spatial texture and semantic features, providing basic features for subsequent processing.

[0069] Step S22: Perform optical flow estimation on the feature maps of the current video image frame, the previous video image frame, and the next video image frame to obtain the bidirectional optical flow field.

[0070] In this step, the feature map F of the current image frame can be... t Feature map F from the previous frame t-1Feature map F of the next frame t+1 The input optical flow estimation module performs optical flow estimation and outputs a bidirectional optical flow field.

[0071] Step S23: Spatial alignment of the feature maps of the previous video image frame and the next video image frame is performed using a bidirectional optical flow field to obtain motion compensation features.

[0072] In this step, the motion compensation unit can be used to adjust F based on the bidirectional optical flow field. t-1 F t+1 Perform spatial alignment and output the motion compensation features (F) matched to the current video image frame. comp ).

[0073] S30. Use the subtitle mask to determine the subtitle region in the video image frame, extract features from the subtitle region image to obtain subtitle region features, and determine the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images; wherein, the scene switching identifier indicates whether scene switching exists or not.

[0074] In this step, in order to explicitly use the prior knowledge that "the subtitles are located in a fixed position", we focus on the characteristics of the fixed position and semantic stability of the region and avoid feature matching deviations that exist when repairing cross-scenes. We can specifically determine the subtitle region in the video image frame, extract the subtitle region features of the subtitle region, and determine the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images, thus making up for the lack of prior information in the original method.

[0075] The process of determining the subtitle region is described below. In one embodiment, determining the subtitle region in a video image frame using a subtitle mask may include:

[0076] Step S311: Determine the minimum bounding matrix corresponding to the subtitle mask, and determine the position coordinates of the minimum bounding matrix in the video image frame; wherein, the position coordinates include the center coordinates, the upper left corner coordinates, and the upper right corner coordinates.

[0077] In this step, the minimum bounding matrix corresponding to the subtitle mask can be determined, and the position coordinates of the minimum bounding matrix can be obtained. These position coordinates may include the center coordinates (cx, cy), the top-left corner coordinates (x1, y1), and the top-right corner coordinates (x2, y2). The minimum bounding matrix indicates the region where the subtitle is located in the current video image frame, which can be extracted using a contour detection algorithm.

[0078] It should be noted that since the subtitle content is different in different video image frames, the minimum bounding matrix determined based on the subtitle mask in different video image frames may be different.

[0079] Step S312: Calculate the standard deviation of the center coordinates in a preset number of consecutive video image frames. If the standard deviation is determined to be less than the preset value, the average value, average width, and average height of the center coordinates in the consecutive video image frames are determined using the position coordinates in the consecutive video image frames.

[0080] In this step, to ensure the stability and reliability of the final subtitle region, it is first necessary to determine whether the minimum bounding matrix obtained in each of the preset number of consecutive video image frames corresponds to approximately the center coordinates. Specifically, in this embodiment, the standard deviation of the center coordinates can be calculated in the preset number of consecutive video image frames. If the standard deviation is determined to be less than a preset value, then these minimum bounding matrices can be used to determine a stable subtitle region; otherwise, if the standard deviation is determined to be not less than the preset value, then the minimum bounding matrix is ​​detected and the standard deviation is calculated again in subsequent video image frames until the standard deviation is less than the preset value.

[0081] It should be noted that this embodiment does not limit the specific values ​​of the preset quantity and preset value, and can be set according to actual application needs. For example, the standard deviation (σ) of the center coordinates within 15 frames can be calculated. cx , σ cy If σ cx ≤3 pixels and σ cy A region with a value of ≤3 pixels can be classified as a "fixed region".

[0082] Furthermore, after determining that the aforementioned standard deviation is less than a preset value, the average center coordinate (cx) of consecutive video image frames can be determined. avg cy avg ) and average width and height (w avg =x2-x1, h avg =y2-y1).

[0083] Step S313: Determine the subtitle area based on the average center coordinates, average width, and average height.

[0084] In this step, the coordinates of the subtitle area (cx) can be obtained. avg cy avg , w avg , h avg Additionally, a subtitle region mask can be generated based on the subtitle region (only pixels within a fixed area are set to 1, and the rest to 0), which can be used as a region constraint for subsequent feature extraction.

[0085] The following describes the process of extracting features from the subtitle region. In one embodiment, extracting features from the subtitle region image to obtain subtitle region features may include:

[0086] Step S321: Use convolutional layers to extract features from the subtitle region image to obtain a global feature vector.

[0087] In this step, the subtitle region of the current frame is first cropped based on the previously obtained subtitle region coordinates to obtain the subtitle region image (size w). avg ×h avg ).

[0088] Subsequently, shallow features (256 channels) of the subtitle region image can be extracted using 4 layers of convolution (3×3 kernel, stride of 1, padding of 1, activation function ReLU, linear rectification function), and then a 256-dimensional global feature vector can be obtained through global average pooling.

[0089] Step S322: Determine the color mean of each color channel in the subtitle region image, and concatenate the color mean with the global feature vector to obtain the subtitle region features.

[0090] In this step, to incorporate color information from the subtitle region image into the subtitle region features, the average color values ​​(R, G, and B) of the three color channels of the subtitle region image can also be calculated. avg G avg B avg This is then concatenated with a 256-dimensional global feature vector to form a 259-dimensional subtitle region feature.

[0091] Furthermore, to facilitate the use of subtitle region features corresponding to historical video frames preceding the current video frame, and to introduce subtitle region feature information from historical video frames when removing subtitles from the current video frame, a FIFO (First In First Out) buffer queue can be constructed as a historical feature library. The capacity of this queue can be fixed at 30 frames (storing fixed region features of the most recent 30 frames). For each frame processed, the subtitle region features of the current frame are enqueued. If the queue is full, the features of the earliest frame are removed.

[0092] The following describes the detection method for scene transitions. In one implementation, the scene transition identifier corresponding to a video image frame is determined based on the image differences between adjacent subtitle region images, including:

[0093] Step S331: Determine the structural similarity value and color histogram difference value between adjacent subtitle region images, and use motion compensation features to determine the average amplitude of optical flow in the subtitle region.

[0094] In this step, to determine whether there is a scene transition between two adjacent video image frames, especially whether there is a scene transition between the subtitle regions of the two video image frames, the structural similarity index (SSIM) and color histogram difference (HSD) values ​​between adjacent subtitle region images can be determined. The average optical flow amplitude within the subtitle region of the current video image frame is then used, based on motion compensation features. The structural similarity index (SSIM) ranges from [0,1] and is used to measure image similarity. The color histogram difference (HSD) ranges from [0,1] and is used to measure color distribution differences. The average optical flow amplitude is used to determine whether there is valid motion information in the subtitle region.

[0095] Step S332: Determine whether the structural similarity value, color histogram difference value, and average optical flow amplitude meet the preset segmentation conditions; wherein, the preset segmentation conditions include the structural similarity value being less than the first preset value, the color histogram difference value being greater than the second preset value, and the average optical flow amplitude being less than the third preset value.

[0096] Step S333: If satisfied, set a scene switching flag to indicate the presence of scene switching for the current video image frame;

[0097] Step S334: If not satisfied, set a scene switching flag indicating that there is no scene switching for the current video image frame.

[0098] In steps S332 to S333, if it is determined that the structural similarity value in adjacent subtitle region images is less than a first preset value, the color histogram difference value is greater than a second preset value, and the average optical flow amplitude is less than a third preset value, it can be determined that there is no structural similarity, a large color difference, and no continuous effective motion information between adjacent subtitle region images. In this case, it can be determined that there is a scene transition. Conversely, if the above conditions are not met, it can be determined that there is no scene transition.

[0099] It should be noted that this embodiment does not limit the specific values ​​of the first, second, and third preset values, and can be set according to actual application requirements. For example, the first preset value can be 0.5, the second preset value can be 0.6, and the third preset value can be 0.8. This embodiment also does not limit the specific value of the scene switching flag when scene switching occurs and when scene switching does not occur. For example, if SSIM < 0.5 and HSD > 0.6, and the average amplitude of optical flow in the fixed area is < 0.8 (no effective movement), then scene switching is determined, and a switching flag S is generated. t =1; otherwise S t =0.

[0100] S40. Determine the fusion weight of video image frames based on the scene switching identifier, fuse motion compensation features and subtitle region features according to the fusion weight to obtain fusion features, and process the fusion features of adjacent video image frames using an attention mechanism to obtain temporal fusion features.

[0101] In this embodiment, the fusion weight of the video image frame can first be determined based on the scene switching identifier. This is because when scene switching occurs, poor feature fusion will cause ghosting and flickering issues in the subtitles after removal. Therefore, this embodiment can dynamically determine the fusion weight of the current video image frame based on the scene switching status of the current video image frame and the most recent scene switching status before the current video image frame.

[0102] Specifically, determining the fusion weights of video image frames based on the scene switching identifier can include:

[0103] Step S411: Obtain the current scene switching identifier and current frame number of the current video frame, and determine the historical frame number of the most recent scene switching before the current video frame based on the scene switching identifier of each video image frame, and determine the interval value between the current frame number and the historical frame number.

[0104] In this step, the first step is to obtain the current scene transition identifier (S) of the current video frame. t The current frame number (t) and the current frame number (t), along with the historical frame number t of the most recent scene switch before the current video frame, are determined based on the scene switch identifier of each video image frame. switch .

[0105] Subsequently, the interval value Δt between the current frame number and the historical frame number can be determined, Δt = tt switch .

[0106] Step S412: If it is determined that the current scene switching identifier indicates that there is no scene switching and the interval value is not within the preset interval range, then the fusion weight is set to the first weight value; wherein, the first weight value is a preset value.

[0107] Step S413: If it is determined that the current scene switching identifier indicates that a scene switching has occurred, then the fusion weight is set to the second weight value; wherein the second weight value is a preset value and is less than the first weight value.

[0108] Step S414: If it is determined that the current scene switching indicator indicates that there is no scene switching and the interval value is within the preset interval range, then the fusion weight is set to a third weight value between the second weight value and the first weight value using a preset linear function and the interval value.

[0109] The following specific example illustrates three scenarios for setting the fusion weights in steps S412-S414:

[0110] 1. If S t =0 (no switching) and Δt>10: fusion weight α=0.9 (preferential trust of the original motion compensation features).

[0111] 2. If S t =1 (just switched): α=0.3 (prioritize trusting historical features to avoid the impact of background mutations).

[0112] 3. If S t =0 and Δt∈[1,10] (transition phase after switching): α=0.3+0.06×Δt (linear transition, gradually restoring the weights of motion compensation features).

[0113] If Δt > 10: α = 0.9 (completely restore the original logic).

[0114] In this way, this embodiment can dynamically fuse motion compensation features and subtitle region features under different scene switching conditions, highlighting the importance of motion compensation features or subtitle region features under different scene switching conditions.

[0115] Furthermore, in this embodiment, motion compensation features and subtitle region features can be fused according to fusion weights to obtain fused features. To further incorporate subtitle region features from historical video image frames and improve the subtitle removal effect, this embodiment can also obtain subtitle region features corresponding to the current video image frame and a preset number of consecutive video image frames before it, and average these subtitle region features to obtain average subtitle region features. Subsequently, motion compensation features and average subtitle region features are fused according to fusion weights to obtain fused features.

[0116] In one implementation, fusing motion compensation features and subtitle region features according to fusion weights to obtain fused features may include:

[0117] Step S421: Determine the average subtitle region features corresponding to a preset number of consecutive video image frames.

[0118] For example, the subtitle region features of the most recent 5 frames can be obtained from the aforementioned historical feature database, and the average subtitle region feature F can be calculated. prior .

[0119] Step S422: Based on the fusion weight, fuse the average subtitle region features into the subtitle region in the motion compensation features to obtain the fused features.

[0120] It is worth noting that, when performing feature fusion, this embodiment may only fuse the average subtitle region features into the subtitle region in the motion compensation features. This can avoid interfering with other features in the non-subtitle region in the motion compensation features, thereby avoiding interference with the image content in the non-subtitle region.

[0121] Specifically, the fusion process can be represented as:

[0122] F fuse = α×F comp + (1-α)×F prior ;

[0123] Among them, F fuse F represents the fusion feature. comp F represents the motion compensation feature. prior Let α represent the average subtitle region features, and α represent the fusion weight. In this embodiment, a subtitle region mask can be used to fuse only the subtitle regions in the motion compensation features with the average subtitle region features.

[0124] Step S423: Apply gain processing to the subtitle region in the fusion feature using a gain coefficient.

[0125] In this step, the blending features within the subtitle area can also be multiplied by a gain factor of 1.2 to enhance the stability of historical features.

[0126] Furthermore, after obtaining the fusion features, attention mechanisms can be applied to the fusion features of adjacent video image frames to obtain temporal fusion features. This embodiment utilizes a bidirectional propagation Transformer structure to perform cross-frame attention interaction and output the enhanced temporal fusion feature F. prop .

[0127] S50. Decode the temporal fusion features to obtain the initial repair features, use the subtitle region features to correct the subtitle region in the initial repair features, and use the corrected initial repair features to generate repaired video image frames with subtitles removed.

[0128] In this step, the U-Net decoder is first used to generate the initial repaired features F by hopping connections with the encoder features through four upsampling operations. pre .

[0129] Furthermore, to further mitigate ghosting and flickering issues during subtitle removal, this embodiment can also utilize subtitle region features to correct the subtitle region in the initial restoration features. Specifically, this embodiment can determine the average subtitle region features corresponding to a preset number of consecutive video image frames and extract the comparison features located in the subtitle region from the initial restoration features. Subsequently, if it is determined that the feature difference between the average subtitle region features and the comparison features is large, the average subtitle region features can be used to perform weighted fusion of the comparison features to achieve the correction effect.

[0130] In one implementation, the subtitle region in the initial repair features is corrected using subtitle region features, including:

[0131] Step S51: Determine the average subtitle region features corresponding to a preset number of consecutive video image frames, and extract the features to be compared from the initial repair features located in the subtitle region.

[0132] Specifically, in this embodiment, the average subtitle region feature F of the most recent 5 frames can be obtained from the aforementioned historical feature database. prior_avg .

[0133] Step S52: Determine the feature difference value between the average subtitle region feature and the feature to be compared, and determine whether the feature difference value is greater than the preset difference threshold.

[0134] In this step, the band alignment feature F of the subtitle region can be extracted from the initial repair features. pre_fixed And calculate its relationship with F. prior_avg Characteristic difference values:

[0135] D=||F pre_fixed - F prior_avg ||2.

[0136] Where ||·||2 represents the L2 distance. Then, a preset difference threshold can be set, and it can be determined whether the feature difference value is greater than the preset difference threshold. If it is greater, it can be determined that the inter-frame difference is too large, which may cause flickering. In this case, the average subtitle region feature can be used to perform weighted fusion of the features to be compared, so as to correct the subtitle region in the initial repair features. If it is not greater, F can be directly retained. pre .

[0137] Step S53: If it is greater than, then use the average subtitle region features to perform weighted fusion of the features to be compared, so as to correct the subtitle region in the initial repair features;

[0138] Step S54: If it is not greater than, then retain the feature to be compared.

[0139] It should be noted that this embodiment does not limit the specific value of the preset difference threshold, and it can be set according to actual application needs. For example, the difference threshold T can be set to 0.2. If D > T (the inter-frame difference is too large, which may cause flickering), then correction is performed: F corr = 0.8×F pre + 0.2×F prior_avg (Only applies to fixed regions); If D≤T, keep F directly. pre F, as the final feature of the video image frame to be repaired corr .

[0140] Finally, F can be obtained through 3×3 convolution.corr The number of feature channels was reduced from 64 to 3 (RGB channels), and pixel-level reconstruction was performed, outputting the final repaired frame I with the subtitles removed. final .

[0141] Based on the above embodiments, the present invention first acquires video image frames and corresponding subtitle masks, and performs feature extraction, optical flow estimation, and motion compensation on the video image frames and subtitle masks to obtain motion compensation features. Subsequently, to explicitly utilize the prior knowledge that "subtitles are located in a fixed position," the present invention uses the subtitle mask to determine the subtitle region in the video image frame, extracts features from the subtitle region image to obtain subtitle region features, and determines the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images; wherein, the scene switching identifier indicates whether a scene switching exists or not. That is, the present invention can determine the fixed region where the subtitles are located, extract subtitle region features, and determine whether a scene switching exists in the video image frame. Subsequently, the fusion weight of the video image frame can be determined based on the scene switching identifier, and the motion compensation features and subtitle region features can be fused according to the fusion weight to obtain fused features. Attention mechanism processing is applied to the fused features of adjacent video image frames to obtain temporal fusion features, that is, the fusion weight can be dynamically determined according to the scene situation to improve the feature fusion effect. Subsequently, the temporal fusion features can be decoded to obtain the initial repair features. The subtitle region features are used to correct the subtitle region in the initial repair features. The corrected initial repair features are then used to generate repaired video image frames with removed subtitles. This can reduce ghosting and flickering in the subtitle region when there are scene transitions in the video image frames, thereby improving the quality of subtitle removal.

[0142] To facilitate understanding of the subtitle removal system framework provided in this embodiment, please refer to... Figure 2 , Figure 2 This invention provides a structural block diagram of a subtitle removal system framework, and the functions of each part are as follows:

[0143] Fixed-region temporal enhancement module (new module): Independent and parallel to the original link, it takes the original video frame and subtitle mask as input and outputs "fixed region coordinates, historical feature library, and scene switching flag", which are then passed to the feature propagation module and the repair generation module respectively.

[0144] Feature extraction module: The function remains unchanged, and the output features of the current frame are passed to the optical flow estimation module and the improved feature propagation module respectively.

[0145] Optical flow estimation and motion compensation module: The function remains unchanged, but the output motion compensation features are passed to the improved feature propagation module.

[0146] Feature propagation module (improved module): Added dynamic weight fusion logic, which takes current frame features, motion compensation features, fixed region historical features and scene switching flags as input, and outputs enhanced temporal fusion features.

[0147] Repair Generation Module (Improved Module): Adds a temporal consistency correction unit, which takes the enhanced temporal fusion features and fixed region historical features as inputs and outputs the final repair frame.

[0148] The beneficial effects of this invention are as follows:

[0149] Lightweight improvements with strong compatibility: Based on the original Propainter architecture, only one core module is added and the local logic of two modules is modified. There are no complex structural adjustments, making it easy to integrate and implement without damaging the original dynamic scene processing capabilities.

[0150] Targeted solutions to core pain points: Explicit modeling of prior information in fixed regions, and fusion of historical feature databases and dynamic weights, accurately solves the "ghosting" and "flickering" problems caused by scene switching, and the temporal consistency optimization is more targeted.

[0151] Adapted to core film and television requirements: Optimization is applied only to fixed subtitle areas without altering the details and texture of non-subtitle areas, avoiding the shortcomings of diffusion-type methods that indiscriminately modify the entire screen, thus meeting the demand for "seamless restoration" in post-production film and television.

[0152] Highly efficient balance between quality and speed: The computational load of adding new modules and modifying logic is small (time consumption ≤10%), and no additional complex network structure is required. While improving the repair quality, the real-time performance of video processing is guaranteed.

[0153] The following describes the subtitle removal device, electronic device, computer-readable storage medium, and computer program product provided in the embodiments of the present invention. The subtitle removal device, electronic device, computer-readable storage medium, and computer program product described below can be referred to in correspondence with the subtitle removal method described above.

[0154] Please refer to Figure 3 , Figure 3 This is a structural block diagram of a subtitle removal device provided in an embodiment of the present invention. The device may include:

[0155] The acquisition module 301 is used to acquire video image frames and corresponding subtitle masks;

[0156] The motion compensation feature extraction module 302 is used to extract features, estimate optical flow, and perform motion compensation on video image frames and subtitle masks to obtain motion compensation features.

[0157] The fixed region timing enhancement module 303 is used to determine the subtitle region in the video image frame using the subtitle mask, extract features from the subtitle region image to obtain subtitle region features, and determine the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images; wherein, the scene switching identifier indicates whether there is a scene switching or not.

[0158] The feature propagation module 304 is used to determine the fusion weight of video image frames according to the scene switching identifier, fuse motion compensation features and subtitle region features according to the fusion weight to obtain fusion features, and perform attention mechanism processing on the fusion features of adjacent video image frames to obtain temporal fusion features.

[0159] The repair generation module 305 is used to decode the temporal fusion features to obtain the initial repair features, use the subtitle region features to correct the subtitle region in the initial repair features, and use the corrected initial repair features to generate repaired video image frames with subtitles removed.

[0160] Optionally, the fixed-region timing enhancement module 303 may include:

[0161] The minimum bounding matrix detection submodule is used to determine the minimum bounding matrix corresponding to the subtitle mask and to determine the position coordinates of the minimum bounding matrix in the video image frame; wherein, the position coordinates include the center coordinates, the top left corner coordinates, and the top right corner coordinates;

[0162] The coordinate calculation submodule is used to calculate the standard deviation of the center coordinates in a preset number of consecutive video image frames. If the standard deviation is determined to be less than the preset value, the average value, average width, and average height of the center coordinates in the consecutive video image frames are determined using the position coordinates in the consecutive video image frames.

[0163] The subtitle region determination submodule is used to determine the subtitle region based on the average center coordinates, average width, and average height.

[0164] Optionally, the fixed-region timing enhancement module 303 may include:

[0165] The feature extraction submodule is used to extract features from the subtitle region image using convolutional layers to obtain a global feature vector;

[0166] The feature splicing submodule is used to determine the color mean of each color channel in the subtitle region image, and splice the color mean with the global feature vector to obtain the subtitle region features.

[0167] Optionally, the fixed-region timing enhancement module 303 may include:

[0168] The difference determination submodule is used to determine the structural similarity value and color histogram difference value between adjacent subtitle region images, and to determine the average optical flow amplitude in the subtitle region using motion compensation features;

[0169] The scene transition detection submodule is used to determine whether the structural similarity value, color histogram difference value, and average optical flow amplitude meet the preset scene transition conditions. The preset scene transition conditions include a structural similarity value less than a first preset value, a color histogram difference value greater than a second preset value, and an average optical flow amplitude less than a third preset value. If the conditions are met, a scene transition flag indicating the presence of a scene transition is set for the current video frame. If the conditions are not met, a scene transition flag indicating the absence of a scene transition is set for the current video frame.

[0170] Optionally, the feature propagation module 304 may include:

[0171] The parameter acquisition submodule is used to acquire the current scene switching identifier and the current frame number of the current video frame, and to determine the historical frame number of the most recent scene switching before the current video frame based on the scene switching identifier of each video image frame, and to determine the interval value between the current frame number and the historical frame number.

[0172] The first weight value setting submodule is used to set the fusion weight to the first weight value if it is determined that the current scene switching identifier indicates that there is no scene switching and the interval value is not within the preset interval range; wherein, the first weight value is a preset value;

[0173] The second weight value setting submodule is used to set the fusion weight to the second weight value if it is determined that the current scene switching identifier indicates that a scene switching has occurred; wherein, the second weight value is a preset value and is less than the first weight value;

[0174] The third weight value setting submodule is used to set the fusion weight to a third weight value between the second weight value and the first weight value if it is determined that the current scene switching identifier indicates that there is no scene switching and the interval value is within the preset interval range.

[0175] Optionally, the feature propagation module 304 may include:

[0176] The feature averaging submodule is used to determine the average subtitle region features corresponding to a preset number of consecutive video image frames.

[0177] The feature fusion submodule is used to fuse the average subtitle region features into the subtitle region in the motion compensation features according to the fusion weights, so as to obtain the fused features;

[0178] The feature gain submodule is used to perform gain processing on the subtitle region in the fused feature using gain coefficients.

[0179] Optionally, the repair generation module 305 may include:

[0180] The feature setting submodule is used to determine the average subtitle region features corresponding to a preset number of consecutive video image frames, and to extract the features to be compared in the subtitle region from the initial repair features;

[0181] The comparison and correction submodule is used to determine the feature difference value between the average subtitle region feature and the feature to be compared, and to determine whether the feature difference value is greater than the preset difference threshold. If it is greater, the average subtitle region feature is used to perform weighted fusion of the feature to be compared in order to correct the subtitle region in the initial repair feature. If it is not greater, the feature to be compared is retained.

[0182] Optionally, the motion compensation feature extraction module 302 may include:

[0183] The feature extraction submodule is used to extract features from video image frames and subtitle masks to obtain feature maps;

[0184] The optical flow estimation submodule is used to estimate the optical flow from the feature maps of the current video image frame, the previous video image frame, and the next video image frame to obtain the bidirectional optical flow field.

[0185] The motion compensation submodule is used to spatially align the feature maps of the previous and next video image frames using a bidirectional optical flow field to obtain motion compensation features.

[0186] Please refer to Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The present invention provides an electronic device 40, including a processor 41 and a memory 42; wherein, the memory 42 is used to store a computer program; the processor 41 is used to execute the subtitle removal method provided in the foregoing embodiment when executing the computer program.

[0187] For details regarding the specific process of the above-mentioned subtitle removal method, please refer to the corresponding content provided in the foregoing embodiments, which will not be repeated here.

[0188] Furthermore, the memory 42, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0189] In addition, the electronic device 40 also includes a power supply 43, a communication interface 44, an input / output interface 45, and a communication bus 46; wherein, the power supply 43 is used to provide operating voltage for the various hardware devices on the electronic device 40; the communication interface 44 can create a data transmission channel between the electronic device 40 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this invention, and is not specifically limited here; the input / output interface 45 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0190] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the subtitle removal method described in the above embodiments.

[0191] Since the embodiments of the computer program product section correspond to the embodiments of the subtitle removal method section, please refer to the description of the embodiments of the subtitle removal method section for the embodiments of the computer program product section, and will not be repeated here.

[0192] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the subtitle removal method described in the above embodiments.

[0193] Since the embodiments of the computer-readable storage medium portion correspond to the embodiments of the subtitle removal method portion, the embodiments of the storage medium portion are described in the description of the embodiments of the subtitle removal method portion, and will not be repeated here.

[0194] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0195] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0196] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0197] The foregoing has provided a detailed description of a subtitle removal method, apparatus, electronic device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for removing subtitles, characterized in that, include: Obtain video image frames and their corresponding caption masks; Motion-compensated features are obtained by performing feature extraction, optical flow estimation, and motion compensation on the video image frames and the subtitle mask; The subtitle region in the video image frame is determined using the subtitle mask, and the subtitle region features are extracted from the subtitle region image. The subtitle region features are then determined based on the image differences between adjacent subtitle region images. The subtitle region feature is used to determine the scene switching identifier corresponding to the video image frame. The scene switching identifier indicates whether a scene switching occurs or not. The fusion weight of the video image frame is determined according to the scene switching identifier. The motion compensation feature and the subtitle region feature are fused according to the fusion weight to obtain the fusion feature. Attention mechanism processing is applied to the fusion features of adjacent video image frames to obtain the temporal fusion feature. The temporal fusion features are decoded to obtain initial repair features. The subtitle region features are used to correct the subtitle region in the initial repair features. The corrected initial repair features are then used to generate repaired video image frames with subtitles removed.

2. The subtitle removal method according to claim 1, characterized in that, Determining the subtitle region in the video image frame using the subtitle mask includes: Determine the minimum bounding matrix corresponding to the subtitle mask, and determine the position coordinates of the minimum bounding matrix in the video image frame; wherein, the position coordinates include the center coordinates, the top left corner coordinates, and the top right corner coordinates; In a preset number of consecutive video image frames, the standard deviation of the center coordinates is calculated. If the standard deviation is determined to be less than a preset value, the average value, average width, and average height of the center coordinates in the consecutive video image frames are determined using the position coordinates in the consecutive video image frames. The subtitle area is determined based on the average center coordinates, the average width, and the average height.

3. The subtitle removal method according to claim 1, characterized in that, Feature extraction is performed on the subtitle region image to obtain subtitle region features, including: The subtitle region image is used to extract features using a convolutional layer to obtain a global feature vector; The average color value of each color channel in the subtitle region image is determined, and the average color value is concatenated with the global feature vector to obtain the subtitle region feature.

4. The subtitle removal method according to claim 1, characterized in that, Determining the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images includes: Determine the structural similarity and color histogram difference values ​​between adjacent subtitle region images, and use the motion compensation features to determine the average optical flow amplitude in the subtitle region; Determine whether the structural similarity value, the color histogram difference value, and the average optical flow amplitude meet preset segmentation conditions; wherein, the preset segmentation conditions include the structural similarity value being less than a first preset value, the color histogram difference value being greater than a second preset value, and the average optical flow amplitude being less than a third preset value. If the conditions are met, then set a scene switching flag indicating the presence of scene switching for the current video image frame; If the condition is not met, then a scene switching flag indicating that there is no scene switching is set for the current video image frame.

5. The subtitle removal method according to claim 1, characterized in that, Determining the fusion weights of the video image frames based on the scene switching identifier includes: Obtain the current scene switching identifier and current frame number of the current video frame, and determine the historical frame number of the most recent scene switching before the current video frame based on the scene switching identifier of each video image frame, and determine the interval value between the current frame number and the historical frame number. If it is determined that the current scene switching indicator indicates that there is no scene switching and the interval value is not within the preset interval range, then the fusion weight is set to the first weight value; wherein, the first weight value is a preset value; If it is determined that the current scene switching flag indicates that a scene switching has occurred, then the fusion weight is set to a second weight value; wherein the second weight value is a preset value and is less than the first weight value; If it is determined that the current scene switching identifier indicates that there is no scene switching and the interval value is within a preset interval range, then the fusion weight is set to a third weight value between the second weight value and the first weight value using a preset linear function and the interval value.

6. The subtitle removal method according to claim 1, characterized in that, The motion compensation features and the subtitle region features are fused according to the fusion weights to obtain fused features, including: Determine the average subtitle region features corresponding to a preset number of consecutive video image frames; Based on the fusion weight, the average subtitle region features are fused into the subtitle region in the motion compensation features to obtain the fused features; Gain processing is applied to the subtitle region in the fusion feature using a gain coefficient.

7. The subtitle removal method according to claim 1, characterized in that, The subtitle region in the initial repair features is corrected using the subtitle region features, including: Determine the average subtitle region features corresponding to a preset number of consecutive video image frames, and extract the features to be compared from the initial repair features located in the subtitle region; Determine the feature difference value between the average subtitle region feature and the feature to be compared, and determine whether the feature difference value is greater than a preset difference threshold; If it is greater than the average subtitle region feature, the features to be compared are weighted and fused to correct the subtitle region in the initial repair features. If it is not greater than, then the feature to be compared is retained.

8. The subtitle removal method according to claim 1, characterized in that, Motion compensation features are obtained by performing feature extraction and optical flow estimation on the video image frames and the subtitle mask, including: Feature extraction is performed on the video image frames and the subtitle mask to obtain feature maps; Optical flow estimation is performed on the feature maps of the current video image frame, the previous video image frame, and the next video image frame to obtain a bidirectional optical flow field. The feature maps of the previous video image frame and the next video image frame are spatially aligned using the bidirectional optical flow field to obtain the motion compensation features.

9. A subtitle removal device, characterized in that, include: The acquisition module is used to acquire video image frames and their corresponding subtitle masks; The motion compensation feature extraction module is used to extract features, estimate optical flow, and perform motion compensation on the video image frames and the subtitle mask to obtain motion compensation features. A fixed-region timing enhancement module is used to determine the subtitle region in the video image frame using the subtitle mask, extract features from the subtitle region image to obtain subtitle region features, and determine the scene switching identifier corresponding to the video image frame based on the image differences between adjacent subtitle region images; wherein, the scene switching identifier indicates whether a scene switching exists or not. The feature propagation module is used to determine the fusion weight of the video image frame according to the scene switching identifier, fuse the motion compensation feature and the subtitle region feature according to the fusion weight to obtain the fusion feature, and perform attention mechanism processing on the fusion features of adjacent video image frames to obtain the temporal fusion feature; The repair generation module is used to decode the temporal fusion features to obtain initial repair features, use the subtitle region features to correct the subtitle region in the initial repair features, and use the corrected initial repair features to generate repaired video image frames with subtitles removed.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the subtitle removal method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the subtitle removal method as described in any one of claims 1 to 8.

Images