Intelligent traceless erasing method and system for video screen text

Abstract

The intelligent traceless erasing method and system for text on video screens provided by the present application relate to the field of video stream processing technology, including: performing spatiotemporal dual-domain feature extraction on the pre-processed first video frame sequence to obtain the first spatiotemporal feature; establishing a spatiotemporal dual-domain attention model based on the first spatiotemporal feature, the spatiotemporal dual-domain attention model fuses the spatial domain and temporal domain features through cross-attention calculation, and dynamically allocates fusion weights based on the text motion speed, obtains the second video frame sequence of the video stream to be processed and pre-processes it, and extracts the second spatiotemporal feature. The spatiotemporal dual-domain attention model synchronously refers to the texture and motion features of the previous and next frames to improve the temporal consistency of the repaired area, and combines the SSIM-driven temporal consistency compensation mechanism to reduce the flicker frequency, and through a dynamic weight allocation strategy, the spatiotemporal domain weights are adaptively adjusted according to the text speed to achieve stable erasing of fast-moving text.

Description

Technical Field

[0001] The present application relates to the field of video stream processing technology, and in particular to a method and system for intelligent and seamless erasing of text on video screens. Background Art

[0002] The method for erasing text from video screens is to identify and remove the text area in the video by synchronously referencing the spatiotemporal information of the previous and next frames to reconstruct the background. However, the existing video text erasure technology has the following technical bottlenecks:

[0003] The single-frame image restoration method based on generative adversarial networks only relies on the texture information of the current frame and does not consider the temporal correlation between the previous and next frames. This causes the restored video to exhibit flickering phenomena such as background jitter and brightness mutation during continuous playback. Especially in the static text erasure scenario, the flicker frequency can be as high as 30%.

[0004] Moreover, although the optical flow estimation method based on ProPainter introduces time information, it is sensitive to the speed of text movement. When the text movement speed exceeds 30 pixels / frame, such as in bullet screens and scrolling subtitles, the optical flow estimation error increases significantly, with an error >10 pixels, resulting in smearing or residual artifacts in the repaired area. In addition, the public technology does not design a dynamic weight distribution mechanism, and cannot adaptively adjust the contribution ratio of spatiotemporal domain features to the repair results.

[0005] In addition, models that rely on optical flow calculation or 3D convolution consume a lot of computing resources, and the feature fusion method is not optimized for text erasure scenarios, which leads to low restoration efficiency and difficulty in meeting real-time requirements. Summary of the Invention

[0006] This application provides an intelligent and seamless erasing method and system for video screen text, aiming to solve the flickering problem caused by the lack of time consistency in related technologies, the lack of compatibility caused by fast-moving text, and the contradiction between computational efficiency and restoration quality.

[0007] In order to achieve the above objectives, this application provides an intelligent and traceless erasing method for video screen text, including the following:

[0008] Obtain the first video frame sequence of the video stream to be processed and perform preprocessing.

[0009] Performing spatiotemporal dual-domain feature extraction on the preprocessed first video frame sequence to obtain a first spatiotemporal feature.

[0010] A spatiotemporal dual-domain attention model is established based on the first spatiotemporal feature. The spatiotemporal dual-domain attention model fuses spatial domain and temporal domain features through cross-attention calculation, and dynamically allocates fusion weights based on the text movement speed.

[0011] A second video frame sequence of the video stream to be processed is obtained and preprocessed to extract a second spatiotemporal feature.

[0012] The second spatiotemporal feature is input into the spatiotemporal dual-domain attention model to perform the matching process.

[0013] The matching process includes main matching and auxiliary determination.

[0014] Main matching, according to the dynamic propagation weight matrix, matches the second spatiotemporal feature with the spatiotemporal dual-domain feature in the spatiotemporal dual-domain attention model. If the matching degree is higher than the matching threshold, the repaired second video frame is directly generated.

[0015] Auxiliary judgment: If the matching degree is lower than the matching threshold, the timing consistency compensation mechanism is triggered to calculate the SSIM similarity of the repaired areas of adjacent frames. When the SSIM is lower than the auxiliary judgment threshold, the sliding window averaging method is used to smooth the timing mutation and re-repair to generate a corrected second video frame.

[0016] Generating a seamless erasure video stream based on the repaired second video frame specifically includes:

[0017] The text erased area in the repaired second video frame is separated in the frequency domain to extract low-frequency structural information and high-frequency texture details.

[0018] The frequency spectrum of the repaired area is compared with the frequency spectrum of the corresponding area of ​​the original video. If the spectrum distribution error exceeds the distribution threshold, the frequency component of the repaired area is adjusted through the frequency domain compensation algorithm.

[0019] The second video frames after dynamic frequency constraint processing are arranged in sequence according to the time sequence of the original video stream, and the SSIM similarity verification is performed on the repaired areas of adjacent frames. If the SSIM values ​​of L consecutive frames are all higher than the similarity threshold, the synthesis result is retained; otherwise, the sliding window averaging method is triggered for local smoothing.

[0020] The video frame sequence synthesized with temporal coherence is encoded into a complete video stream, all text mask marks and temporary data are removed, and a text-free video stream is output that is consistent with the original video resolution and frame rate.

[0021] As a preferred solution of the present invention, obtaining a first video frame sequence of a video stream to be processed and performing preprocessing specifically includes:

[0022] The first video frame sequence includes the current frame and its adjacent previous and next frames. The preprocessing includes detecting the text area through a multi-scale feature pyramid network, generating an initial text mask and a motion trajectory prediction map, extracting N consecutive frames from the video stream to be processed as the first video frame sequence, which includes the current frame, the previous M frames and the next M frames, and eliminating the offset error between adjacent frames caused by lens motion through a time alignment module.

[0023] Input the first video frame sequence into the multi-scale feature pyramid network and perform the following operations:

[0024] The backbone network extracts hierarchical feature maps of each frame and generates feature pyramids at four scales, P2-P5. Deformable convolutional layers are deployed at the P3-P5 levels to adaptively match the geometric deformation features of the text area and output multi-scale feature maps based on the matching.

[0025] Based on the output multi-scale feature map, the initial text candidate box is generated at the P3 level, and the coordinates are adjusted using the candidate box regression network. The feature maps of the P4 and P5 levels are upsampled, fused with the P3 level features, and the final candidate box is screened using the non-maximum suppression algorithm.

[0026] The final candidate box after screening is refined: a mask branch network is used to predict the binary text mask. The mask branch network consists of three 3×3 convolutional layers and one bilinear interpolation upsampling layer.

[0027] Based on the temporal continuity of the first video frame sequence, optical flow estimation is performed on the candidate frames of adjacent frames, and the displacement vectors of the text area in the previous and next frames are calculated. The displacement vectors are fitted using a trajectory smoothing algorithm to generate a motion trajectory prediction map. The motion trajectory prediction map contains the coordinate sequence and velocity vector of the text center point. The velocity vector is normalized to obtain the text motion speed value for subsequent weight allocation.

[0028] As a preferred solution of the present invention, performing spatiotemporal dual-domain feature extraction on the preprocessed first video frame sequence to obtain the first spatiotemporal feature specifically includes:

[0029] Based on the generated initial text mask and multi-scale feature map, spatial domain feature extraction is performed, and a deformable convolution layer is deployed in the text area. The convolution kernel contains K offset sampling points, and the offset is calculated using the following formula:

[0030] Δp k =W k ·F text

[0031] Where, F text is the text area feature map, W k is the learnable weight matrix, Δp k is the coordinate offset of the kth sampling point.

[0032] The background edge features are extracted using the offset sampling points. The spatial features of the P3 level are fused with the contextual features of the P4 and P5 levels across scales through a three-layer residual convolutional network to generate a spatial domain feature map. Based on the generated motion trajectory prediction map, temporal domain feature extraction is performed.

[0033] Time series construction,arrange the text velocity vectors in the trajectory prediction graph and the displacement vectors of adjacent frames in time sequence to form a time series with a length of 2M+1.

[0034] Bidirectional LSTM modeling: The time series is input into the bidirectional LSTM network. The bidirectional LSTM network contains two hidden layers, each with a dimension of 128. The temporal dependency is calculated through the forget gate, input gate, and output gate.

[0035] Dynamic weight generation, splicing the forward and backward hidden states of LSTM output, mapping it into a dynamic propagation weight matrix through the fully connected layer, the matrix dimension and spatial domain features Figure 1 To.

[0036] Multiply the spatial domain feature map and the dynamic propagation weight matrix element by element to generate the fused first spatiotemporal feature. The calculation formula is as follows:

[0037] F ST =F spatial ⊙σ(W temporal )

[0038] Where σ is the Sigmoidi activation function, which is used to constrain the weight range to [0,1], and F ST represents the first spatiotemporal feature after fusion, W temporal represents the dynamic propagation weight matrix, F spatial Represents the spatial domain feature map.

[0039] As a preferred solution of the present invention, a spatiotemporal dual-domain attention model is established based on the first spatiotemporal feature. The spatiotemporal dual-domain attention model fuses spatial and temporal domain features through cross-attention calculation and dynamically allocates fusion weights based on the text motion speed. Specifically, it includes:

[0040] The spatial domain feature map is used as the query vector, and the temporal domain dynamic propagation weight matrix is ​​used as the key value and content vector. A multi-head attention mechanism is used to calculate the association weights of the spatial and temporal domains, where the dimension of each attention head is a preset value, and the weights are normalized by the Softmax function. The calculation results of multiple attention heads are merged to generate the initial spatiotemporal association weight matrix.

[0041] Based on the normalized text motion speed value, perform the following operations:

[0042] When the text motion speed is lower than the threshold α, the spatial domain weight ratio is set to be greater than 70%.

[0043] When the text motion speed is higher than or equal to the threshold α, the time domain weight ratio is set to be greater than 60%.

[0044] The velocity value is converted into a proportional coefficient of spatial domain and temporal domain weights through a mapping function, wherein the mapping function includes a trainable velocity-weight mapping coefficient and a bias term.

[0045] The dynamic weight allocation rule is weightedly fused with the initial spatiotemporal correlation weight matrix to generate the final weight parameters of the spatiotemporal dual-domain attention model and store them in the model database.

[0046] As a preferred solution of the present invention, obtaining a second video frame sequence of the video stream to be processed and performing preprocessing to extract the second spatiotemporal feature specifically includes:

[0047] Receive the video stream to be processed in real time, extract T consecutive frames as the second video frame sequence, including the current processing frame and its adjacent previous and next frames, eliminate pixel offsets between adjacent frames caused by lens motion or object displacement through time alignment, and output the aligned second video frame sequence.

[0048] The aligned second video frame sequence is input into a multi-scale feature pyramid network to generate candidate text region frames and initial text masks for each frame. Based on the motion trajectory prediction map generation method, the displacement vector and normalized velocity value of the text region in the second video frame sequence are calculated.

[0049] According to the initial text mask, the spatial domain feature extraction method is used to extract the spatial domain feature map of the second video frame sequence; based on the text displacement vector, the dynamic propagation weight matrix of the second video frame sequence is generated through the time domain feature extraction method; the spatial domain feature map and the dynamic propagation weight matrix are multiplied element by element according to the fusion rule to generate the second spatiotemporal feature.

[0050] This application also provides an intelligent traceless erasing system for video screen text, including the following:

[0051] The preprocessing unit obtains a first video frame sequence of the video stream to be processed and performs preprocessing.

[0052] The feature extraction unit performs spatiotemporal dual-domain feature extraction on the preprocessed first video frame sequence to obtain a first spatiotemporal feature.

[0053] A model construction unit is used to establish a spatiotemporal dual-domain attention model based on the first spatiotemporal feature. The spatiotemporal dual-domain attention model fuses spatial domain and temporal domain features through cross-attention calculation and dynamically allocates fusion weights based on the text movement speed.

[0054] The secondary extraction unit is used to obtain a second video frame sequence of the video stream to be processed and perform preprocessing to extract a second spatiotemporal feature.

[0055] The matching unit is used to input the second spatiotemporal feature into the spatiotemporal dual-domain attention model to perform a matching process.

[0056] The matching process includes main matching and auxiliary determination.

[0057] The main judgment unit is used for main matching. According to the dynamic propagation weight matrix, the second spatiotemporal feature is matched with the spatiotemporal dual-domain feature in the spatiotemporal dual-domain attention model. If the matching degree is higher than the matching threshold, the repaired second video frame is directly generated.

[0058] The auxiliary judgment unit is used to assist in judgment. If the matching degree is lower than the matching threshold, the timing consistency compensation mechanism is triggered to calculate the SSIM similarity of the repaired areas of adjacent frames. When the SSIM is lower than the auxiliary judgment threshold, the timing mutation is smoothed and re-repaired through the sliding window averaging method to generate a corrected second video frame.

[0059] The output unit is configured to generate a seamlessly erased video stream based on the repaired second video frame, specifically comprising:

[0060] The text erased area in the repaired second video frame is separated in the frequency domain to extract low-frequency structural information and high-frequency texture details.

[0061] The frequency spectrum of the repaired area is compared with the frequency spectrum of the corresponding area of ​​the original video. If the spectrum distribution error exceeds the distribution threshold, the frequency component of the repaired area is adjusted through the frequency domain compensation algorithm.

[0062] The second video frames after dynamic frequency constraint processing are arranged in sequence according to the time sequence of the original video stream, and the SSIM similarity verification is performed on the repaired areas of adjacent frames. If the SSIM values ​​of L consecutive frames are all higher than the similarity threshold, the synthesis result is retained; otherwise, the sliding window averaging method is triggered for local smoothing.

[0063] The video frame sequence synthesized with temporal coherence is encoded into a complete video stream, all text mask marks and temporary data are removed, and a text-free video stream is output that is consistent with the original video resolution and frame rate.

[0064] The beneficial effects of this application are:

[0065] 1. The spatiotemporal dual-domain attention model simultaneously references the texture and motion features of the previous and next frames to improve the temporal coherence of the repaired area. Combined with the SSIM-driven temporal consistency compensation mechanism, the flicker frequency is reduced. A dynamic weight allocation strategy adaptively adjusts the spatiotemporal domain weights according to the text speed, supporting stable erasure of fast-moving text. At the same time, the deformable convolution and trajectory smoothing algorithms capture text deformation and motion trajectory, reducing the residual rate of ghosting in the repaired area.

[0066] 2. Dynamic frequency constraints and sliding window averaging methods are used to reduce redundant calculations, reduce the number of model parameters, and speed up inference. The frequency domain compensation algorithm is used to control the spectrum distribution error of the repaired area to ≤3dB, thereby improving the quality of high-frequency detail reconstruction.

[0067] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 A flowchart of a method for intelligent and seamless erasing of text on video screens provided in an embodiment of the present application.

[0070] Figure 2 This is an overall framework diagram of the intelligent seamless erasing system for video screen text provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0072] See also Figure 1 , Figure 1 A flowchart of a method for intelligent and seamless erasing of text on video screens provided in an embodiment of the present application.

[0073] In this embodiment, the intelligent seamless erasing method for video screen text may include step S10, step S20, step S30, step S40, step S50 and step S60.

[0074] Step S10, obtaining a first video frame sequence of the video stream to be processed and performing preprocessing, specifically includes:

[0075] The first video frame sequence includes the current frame and its adjacent previous and next frames. The preprocessing includes detecting the text area through a multi-scale feature pyramid network, generating an initial text mask and a motion trajectory prediction map, extracting N consecutive frames from the video stream to be processed as the first video frame sequence, which includes the current frame, the previous M frames and the next M frames, and eliminating the offset error between adjacent frames caused by lens motion through a time alignment module.

[0076] It should be noted that N≥3, M≥1.

[0077] Input the first video frame sequence into the multi-scale feature pyramid network and perform the following operations:

[0078] The backbone network extracts hierarchical feature maps of each frame and generates feature pyramids at four scales, P2-P5. Deformable convolutional layers are deployed at the P3-P5 levels to adaptively match the geometric deformation features of the text area and output multi-scale feature maps based on the matching.

[0079] It should be noted that P2 has the highest resolution, followed by P3, and so on to P5.

[0080] Based on the output multi-scale feature map, the initial text candidate box is generated at the P3 level, and the coordinates are adjusted using the candidate box regression network. The feature maps of the P4 and P5 levels are upsampled, fused with the P3 level features, and the final candidate box is screened using the non-maximum suppression algorithm.

[0081] It should be noted that the confidence threshold here is set to θ (θ ≥ 0.7).

[0082] The final candidate box after screening is refined: a mask branch network is used to predict the binary text mask. The mask branch network consists of three 3×3 convolutional layers and one bilinear interpolation upsampling layer.

[0083] It should be noted that an L1 loss constraint is applied to the mask edge area to ensure that the text boundary positioning error is less than 2 pixels.

[0084] Based on the temporal continuity of the first video frame sequence, optical flow estimation is performed on the candidate frames of adjacent frames, and the displacement vectors of the text area in the previous and next frames are calculated. The displacement vectors are fitted using a trajectory smoothing algorithm to generate a motion trajectory prediction map. The motion trajectory prediction map contains the coordinate sequence and velocity vector of the text center point. The velocity vector is normalized to obtain the text motion speed value for subsequent weight allocation.

[0085] Step S20, performing spatiotemporal dual-domain feature extraction on the pre-processed first video frame sequence to obtain a first spatiotemporal feature, specifically comprising:

[0086] Based on the generated initial text mask and multi-scale feature map, spatial domain feature extraction is performed, and a deformable convolution layer is deployed in the text area. The convolution kernel contains K offset sampling points. It should be noted that K ≥ 9. The offset is calculated using the following formula:

[0087] Δp k =W k ·F text

[0088] Where, F text is the text area feature map, W kis the learnable weight matrix, Δp k is the coordinate offset of the kth sampling point.

[0089] The background edge features are extracted using the offset sampling points, and the spatial features of the P3 level are fused with the contextual features of the P4 and P5 levels across scales through a three-layer residual convolutional network to generate a spatial domain feature map.

[0090] It should be noted that each layer of the residual convolutional network contains 64 3×3 convolution kernels.

[0091] Based on the generated motion trajectory prediction map, time domain feature extraction is performed.

[0092] Time series construction,arrange the text velocity vectors in the trajectory prediction graph and the displacement vectors of adjacent frames in time sequence to form a time series with a length of 2M+1.

[0093] Bidirectional LSTM modeling: The time series is input into the bidirectional LSTM network. The bidirectional LSTM network contains two hidden layers, each with a dimension of 128. The temporal dependency is calculated through the forget gate, input gate, and output gate.

[0094] Dynamic weight generation, splicing the forward and backward hidden states of LSTM output, mapping it into a dynamic propagation weight matrix through the fully connected layer, the matrix dimension and spatial domain features Figure 1 To.

[0095] Multiply the spatial domain feature map and the dynamic propagation weight matrix element by element to generate the fused first spatiotemporal feature. The calculation formula is as follows:

[0096] F ST =F spatial ⊙σ(W temporal )

[0097] Where σ is the Sigmoidi activation function, which is used to constrain the weight range to [0,1], and F ST represents the first spatiotemporal feature after fusion, W temporal represents the dynamic propagation weight matrix, F spatial Represents the spatial domain feature map.

[0098] Step S30: Establishing a spatiotemporal dual-domain attention model based on the first spatiotemporal feature. The spatiotemporal dual-domain attention model fuses the spatial and temporal features through cross-attention calculation and dynamically assigns fusion weights based on the text motion speed. Specifically, the model includes:

[0099] The spatial domain feature map is used as the query vector, and the temporal domain dynamic propagation weight matrix is ​​used as the key value and content vector. A multi-head attention mechanism is used to calculate the association weights of the spatial and temporal domains, where the dimension of each attention head is a preset value, and the weights are normalized by the Softmax function. The calculation results of multiple attention heads are merged to generate the initial spatiotemporal association weight matrix.

[0100] Based on the normalized text motion speed value, perform the following operations:

[0101] When the text motion speed is lower than the threshold α, the spatial domain weight ratio is set to be greater than 70%.

[0102] When the text motion speed is higher than or equal to the threshold α, the time domain weight ratio is set to be greater than 60%.

[0103] The velocity value is converted into a proportional coefficient of spatial domain and temporal domain weights through a mapping function, wherein the mapping function includes a trainable velocity-weight mapping coefficient and a bias term.

[0104] The dynamic weight allocation rule is weightedly fused with the initial spatiotemporal correlation weight matrix to generate the final weight parameters of the spatiotemporal dual-domain attention model and store them in the model database.

[0105] It should be noted that the threshold α setting rules are as follows:

[0106] The distribution of motion speeds of all text areas in the training data set is statistically analyzed, and the maximum speed value covering 95% of the samples is selected as the initial threshold.

[0107] Adjust the initial threshold on the validation set and select the final threshold α that meets the following conditions:

[0108] The flicker frequency reduction rate exceeds the preset index.

[0109] The false erasure rate in non-text areas is less than 5%.

[0110] Set the value range of α to 0.3 to 0.5, and the unit is consistent with the normalized velocity value.

[0111] Step S40: obtaining a second video frame sequence of the video stream to be processed and performing preprocessing to extract a second spatiotemporal feature, specifically including:

[0112] Receive the video stream to be processed in real time, extract T consecutive frames as the second video frame sequence, including the current processing frame and its adjacent previous and next frames, eliminate pixel offsets between adjacent frames caused by lens motion or object displacement through time alignment, and output the aligned second video frame sequence.

[0113] It should be noted that T≥3.

[0114] The aligned second video frame sequence is input into a multi-scale feature pyramid network to generate candidate text region frames and initial text masks for each frame. Based on the motion trajectory prediction map generation method, the displacement vector and normalized velocity value of the text region in the second video frame sequence are calculated.

[0115] According to the initial text mask, the spatial domain feature extraction method is used to extract the spatial domain feature map of the second video frame sequence; based on the text displacement vector, the dynamic propagation weight matrix of the second video frame sequence is generated through the time domain feature extraction method; the spatial domain feature map and the dynamic propagation weight matrix are multiplied element by element according to the fusion rule to generate the second spatiotemporal feature.

[0116] Step S50: Input the second spatiotemporal feature into the spatiotemporal dual-domain attention model to perform a matching process.

[0117] The matching process includes main matching and auxiliary determination.

[0118] Main matching, according to the dynamic propagation weight matrix, matches the second spatiotemporal feature with the spatiotemporal dual-domain feature in the spatiotemporal dual-domain attention model. If the matching degree is higher than the matching threshold, the repaired second video frame is directly generated.

[0119] Auxiliary judgment: If the matching degree is lower than the matching threshold, the timing consistency compensation mechanism is triggered to calculate the SSIM similarity of the repaired areas of adjacent frames. When the SSIM is lower than the auxiliary judgment threshold, the sliding window averaging method is used to smooth the timing mutation and re-repair to generate a corrected second video frame.

[0120] Step S60, generating a seamlessly erased video stream based on the repaired second video frame, specifically includes:

[0121] The text erased area in the repaired second video frame is separated in the frequency domain to extract low-frequency structural information and high-frequency texture details.

[0122] The frequency spectrum of the repaired area is compared with the frequency spectrum of the corresponding area of ​​the original video. If the spectrum distribution error exceeds the distribution threshold, the frequency component of the repaired area is adjusted through the frequency domain compensation algorithm.

[0123] It should be noted that the distribution threshold is ≤3dB.

[0124] The second video frames after dynamic frequency constraint processing are arranged in sequence according to the time sequence of the original video stream, and the SSIM similarity verification is performed on the repaired areas of adjacent frames. If the SSIM values ​​of L consecutive frames are all higher than the similarity threshold, the synthesis result is retained; otherwise, the sliding window averaging method is triggered for local smoothing.

[0125] It should be noted that during the training phase, the SSIM similarity between all repaired frames and the original non-text background areas was calculated, and the 95% percentile value was selected as the initial similarity threshold. The effects of different similarity thresholds on flicker frequency and false erasure rate were tested on the validation set, and the final similarity threshold was selected to meet the following conditions:

[0126] The flicker frequency reduction rate is ≥85%; that is, the number of flicker frames after repair / the number of original flicker frames is ≤0.15.

[0127] The false erasure rate of non-text areas is ≤3%.

[0128] Value range limitation: Set γ∈[0.88,0.92], where γ represents the final similarity threshold and its unit is consistent with the SSIM indicator.

[0129] The video frame sequence synthesized with temporal coherence is encoded into a complete video stream, all text mask marks and temporary data are removed, and a text-free video stream is output that is consistent with the original video resolution and frame rate.

[0130] At this point, the intelligent and seamless erasing method for video screen text is completed.

[0131] See also Figure 2 , Figure 2 Provides an intelligent, traceless erasing system for video and text, including the following:

[0132] The preprocessing unit obtains a first video frame sequence of the video stream to be processed and performs preprocessing.

[0133] The feature extraction unit performs spatiotemporal dual-domain feature extraction on the preprocessed first video frame sequence to obtain a first spatiotemporal feature.

[0134] A model construction unit is used to establish a spatiotemporal dual-domain attention model based on the first spatiotemporal feature. The spatiotemporal dual-domain attention model fuses spatial domain and temporal domain features through cross-attention calculation and dynamically allocates fusion weights based on the text movement speed.

[0135] The secondary extraction unit is used to obtain a second video frame sequence of the video stream to be processed and perform preprocessing to extract a second spatiotemporal feature.

[0136] The matching unit is used to input the second spatiotemporal feature into the spatiotemporal dual-domain attention model to perform a matching process.

[0137] The matching process includes main matching and auxiliary determination.

[0138] The main judgment unit is used for main matching. According to the dynamic propagation weight matrix, the second spatiotemporal feature is matched with the spatiotemporal dual-domain feature in the spatiotemporal dual-domain attention model. If the matching degree is higher than the matching threshold, the repaired second video frame is directly generated.

[0139] The auxiliary judgment unit is used to assist in judgment. If the matching degree is lower than the matching threshold, the timing consistency compensation mechanism is triggered to calculate the SSIM similarity of the repaired areas of adjacent frames. When the SSIM is lower than the auxiliary judgment threshold, the timing mutation is smoothed and re-repaired through the sliding window averaging method to generate a corrected second video frame.

[0140] The output unit is configured to generate a seamlessly erased video stream based on the repaired second video frame, specifically comprising:

[0141] The text erased area in the repaired second video frame is separated in the frequency domain to extract low-frequency structural information and high-frequency texture details.

[0142] The frequency spectrum of the repaired area is compared with the frequency spectrum of the corresponding area of ​​the original video. If the spectrum distribution error exceeds the distribution threshold, the frequency component of the repaired area is adjusted through the frequency domain compensation algorithm.

[0143] The second video frames after dynamic frequency constraint processing are arranged in sequence according to the time sequence of the original video stream, and the SSIM similarity verification is performed on the repaired areas of adjacent frames. If the SSIM values ​​of L consecutive frames are all higher than the similarity threshold, the synthesis result is retained; otherwise, the sliding window averaging method is triggered for local smoothing.

[0144] The video frame sequence synthesized with temporal coherence is encoded into a complete video stream, all text mask marks and temporary data are removed, and a text-free video stream is output that is consistent with the original video resolution and frame rate.

[0145] At this point, the intelligent traceless erasing system for video screen text is completed.

[0146] The above description is merely a specific embodiment of the present invention, but the scope of protection of the embodiments of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included in the scope of protection of the embodiments of the present invention. Therefore, the scope of protection of the embodiments of the present invention should be based on the scope of protection of the claims.

Claims

1. An intelligent traceless erasing method for video screen text, characterized in that: These include: Obtaining a first video frame sequence of a video stream to be processed and performing preprocessing; Performing spatiotemporal dual-domain feature extraction on the preprocessed first video frame sequence to obtain a first spatiotemporal feature; establishing a spatiotemporal dual-domain attention model based on the first spatiotemporal feature, wherein the spatiotemporal dual-domain attention model fuses the spatial domain and temporal domain features through cross-attention calculation and dynamically assigns a fusion weight based on the text motion speed; Obtaining a second video frame sequence of the video stream to be processed and performing preprocessing to extract a second spatiotemporal feature; Input the second spatiotemporal feature into the spatiotemporal dual-domain attention model to perform the matching process; The matching process includes main matching and auxiliary determination; Main matching, according to the dynamic propagation weight matrix, matches the second spatiotemporal feature with the spatiotemporal dual-domain feature in the spatiotemporal dual-domain attention model. If the matching degree is higher than the matching threshold, the repaired second video frame is directly generated; Auxiliary judgment: If the matching degree is lower than the matching threshold, the temporal consistency compensation mechanism is triggered to calculate the SSIM similarity of the repaired areas of adjacent frames. When the SSIM is lower than the auxiliary judgment threshold, the sliding window averaging method is used to smooth the temporal mutation and re-repair to generate the corrected second video frame; Generating a seamless erasure video stream based on the repaired second video frame specifically includes: Perform frequency domain separation on the erased text area in the restored second video frame to extract low-frequency structural information and high-frequency texture details; The frequency spectrum of the repaired area is compared with the frequency spectrum of the corresponding area of ​​the original video. If the spectrum distribution error exceeds the distribution threshold, the frequency component of the repaired area is adjusted through the frequency domain compensation algorithm; Arrange the second video frames after dynamic frequency constraint processing in the chronological order of the original video stream, and perform SSIM similarity verification on the repaired areas of adjacent frames. If the SSIM values ​​of L consecutive frames are all higher than the similarity threshold, the synthesis result is retained; otherwise, the sliding window averaging method is triggered for local smoothing. The video frame sequence synthesized with temporal coherence is encoded into a complete video stream, all text mask marks and temporary data are removed, and a text-free video stream is output that is consistent with the original video resolution and frame rate.

2. The method for intelligent and traceless erasing of text on a video screen according to claim 1, wherein: Obtain the first video frame sequence of the video stream to be processed and perform preprocessing, specifically including: The first video frame sequence includes the current frame and its adjacent previous and next frames. Preprocessing includes detecting text regions using a multi-scale feature pyramid network, generating an initial text mask and a motion trajectory prediction map, extracting N consecutive frames from the video stream to be processed as the first video frame sequence, which includes the current frame, the previous M frames, and the next M frames, and eliminating offset errors between adjacent frames caused by lens motion using a time alignment module. Input the first video frame sequence into the multi-scale feature pyramid network and perform the following operations: The backbone network extracts hierarchical feature maps from each frame and generates feature pyramids at four scales, P2-P5. Deformable convolutional layers are deployed at the P3-P5 levels to adaptively match the geometric deformation features of the text area and output multi-scale feature maps based on the matching. Based on the output multi-scale feature map, the initial text candidate box is generated at the P3 level, and the coordinates are adjusted using the candidate box regression network. The feature maps of the P4 and P5 levels are upsampled and fused with the P3 level features. The final candidate box is selected using the non-maximum suppression algorithm. The final candidate boxes are refined by using a mask branch network consisting of three 3×3 convolutional layers and one bilinear interpolation upsampling layer to predict the binary text mask. Based on the temporal continuity of the first video frame sequence, optical flow estimation is performed on the candidate frames of adjacent frames, and the displacement vectors of the text area in the previous and next frames are calculated. The displacement vectors are fitted using a trajectory smoothing algorithm to generate a motion trajectory prediction map. The motion trajectory prediction map contains the coordinate sequence and velocity vector of the text center point. The velocity vector is normalized to obtain the text motion speed value for subsequent weight allocation.

3. The method for intelligent and traceless erasing of text on a video screen according to claim 2, characterized in that: Performing spatiotemporal dual-domain feature extraction on the preprocessed first video frame sequence to obtain a first spatiotemporal feature, specifically including: Based on the generated initial text mask and multi-scale feature map, spatial domain feature extraction is performed, and a deformable convolution layer is deployed in the text area. The convolution kernel contains K offset sampling points, and the offset is calculated using the following formula: Δp k =W k ·F text Where, F text is the text area feature map, W k is the learnable weight matrix, Δp k is the coordinate offset of the kth sampling point; The background edge features are extracted using the offset sampling points. The spatial features of the P3 level are fused with the context features of the P4 and P5 levels across scales through a three-layer residual convolutional network to generate a spatial domain feature map. Based on the generated motion trajectory prediction map, temporal domain feature extraction is performed: Time series construction: Arrange the text velocity vectors in the trajectory prediction graph and the displacement vectors of adjacent frames in chronological order to form a time series with a length of 2M+1; Bidirectional LSTM modeling: The time series is input into a bidirectional LSTM network. The bidirectional LSTM network contains two hidden layers, each with a dimension of 128. The temporal dependency is calculated through the forget gate, input gate, and output gate. Dynamic weight generation: concatenate the forward and backward hidden states of the LSTM output and map them into a dynamic propagation weight matrix through a fully connected layer. The matrix dimension is consistent with the spatial domain feature map. Multiply the spatial domain feature map and the dynamic propagation weight matrix element by element to generate the fused first spatiotemporal feature. The calculation formula is as follows: F ST =F spatial ⊙σ(W temporal ) Where σ is the Sigmoidi activation function, which is used to constrain the weight range to [0,1], and F ST represents the first spatiotemporal feature after fusion, W temporal represents the dynamic propagation weight matrix, F spatial Represents the spatial domain feature map.

4. The method for intelligent and traceless erasing of text on a video screen according to claim 1, wherein: A spatiotemporal dual-domain attention model is established based on the first spatiotemporal feature. The spatiotemporal dual-domain attention model fuses spatial and temporal features through cross-attention calculation and dynamically assigns fusion weights based on the text motion speed. Specifically, the model includes: The spatial domain feature map is used as the query vector, and the temporal domain dynamic propagation weight matrix is ​​used as the key value and content vector. A multi-head attention mechanism is used to calculate the association weights of the spatial and temporal domains, where the dimension of each attention head is a preset value and the weights are normalized by the Softmax function. The calculation results of multiple attention heads are combined to generate the initial spatiotemporal association weight matrix. Based on the normalized text motion speed value, perform the following operations: When the text motion speed is lower than the threshold α, the spatial domain weight ratio is set to be greater than 70%; When the text motion speed is higher than or equal to the threshold α, the time domain weight ratio is set to be greater than 60%; Converting the velocity value into a proportional coefficient of spatial domain and temporal domain weights through a mapping function, wherein the mapping function includes a trainable velocity-weight mapping coefficient and a bias term; The dynamic weight allocation rule is weightedly fused with the initial spatiotemporal correlation weight matrix to generate the final weight parameters of the spatiotemporal dual-domain attention model and store them in the model database.

5. The method for intelligent and traceless erasing of text on a video screen according to claim 1, wherein: Obtaining a second video frame sequence of the video stream to be processed and performing preprocessing to extract a second spatiotemporal feature, specifically including: Receive the video stream to be processed in real time, extract T consecutive frames as the second video frame sequence, including the current processing frame and its adjacent previous and next frames, eliminate pixel offsets between adjacent frames caused by lens motion or object displacement through time alignment, and output the aligned second video frame sequence; Input the aligned second video frame sequence into a multi-scale feature pyramid network to generate candidate text regions and initial text masks for each frame. Based on a motion trajectory prediction map generation method, the displacement vector and normalized velocity value of the text region in the second video frame sequence are calculated. According to the initial text mask, the spatial domain feature extraction method is used to extract the spatial domain feature map of the second video frame sequence; based on the text displacement vector, the dynamic propagation weight matrix of the second video frame sequence is generated through the time domain feature extraction method; the spatial domain feature map and the dynamic propagation weight matrix are multiplied element by element according to the fusion rule to generate the second spatiotemporal feature.

6. Intelligent traceless erasing system for video screen text, characterized by: Specifically include: A preprocessing unit, which obtains a first video frame sequence of a video stream to be processed and performs preprocessing; A feature extraction unit performs spatiotemporal dual-domain feature extraction on the preprocessed first video frame sequence to obtain a first spatiotemporal feature; a model building unit, configured to establish a spatiotemporal dual-domain attention model based on the first spatiotemporal feature, wherein the spatiotemporal dual-domain attention model fuses the spatial domain and temporal domain features through cross-attention calculation and dynamically assigns a fusion weight based on the text motion speed; A secondary extraction unit, configured to obtain a second video frame sequence of the video stream to be processed and perform preprocessing to extract a second spatiotemporal feature; A matching unit, configured to input the second spatiotemporal feature into the spatiotemporal dual-domain attention model to perform a matching process; The matching process includes main matching and auxiliary determination; The main decision unit is used for main matching. It matches the second spatiotemporal feature with the spatiotemporal dual-domain feature in the spatiotemporal dual-domain attention model according to the dynamic propagation weight matrix. If the matching degree is higher than the matching threshold, the repaired second video frame is directly generated. An auxiliary judgment unit is used to assist in the judgment. If the matching degree is lower than the matching threshold, the timing consistency compensation mechanism is triggered to calculate the SSIM similarity of the repaired areas of adjacent frames. When the SSIM is lower than the auxiliary judgment threshold, the sliding window averaging method is used to smooth the timing mutation and re-repair to generate a corrected second video frame. The output unit is configured to generate a seamlessly erased video stream based on the repaired second video frame, specifically comprising: Perform frequency domain separation on the erased text area in the restored second video frame to extract low-frequency structural information and high-frequency texture details; The frequency spectrum of the repaired area is compared with the frequency spectrum of the corresponding area of ​​the original video. If the spectrum distribution error exceeds the distribution threshold, the frequency component of the repaired area is adjusted through the frequency domain compensation algorithm; Arrange the second video frames after dynamic frequency constraint processing in the chronological order of the original video stream, and perform SSIM similarity verification on the repaired areas of adjacent frames. If the SSIM values ​​of L consecutive frames are all higher than the similarity threshold, the synthesis result is retained; otherwise, the sliding window averaging method is triggered for local smoothing. The video frame sequence synthesized with temporal coherence is encoded into a complete video stream, all text mask marks and temporary data are removed, and a text-free video stream is output that is consistent with the original video resolution and frame rate.

Images