Video processing method and device, electronic equipment and storage medium

Abstract

The application relates to a video processing method and device, electronic equipment and storage medium, the method comprising: obtaining an initial video frame of an original video, and extracting text information of the initial video frame; converting the text information into a structured vector image, performing layer separation on the initial video frame to obtain a text layer and a background layer; performing frame interpolation compensation processing on the text layer to obtain a compensated vector text frame; performing motion compensation processing on the background layer to obtain a compensated background frame; synthesizing the compensated vector text frame and the compensated background frame to obtain a target video frame; and performing frame sequence recombination on the target video frame to generate and output a target video picture. Through layer separation on the video frame, the motion vector of the background no longer interferes with the text, thereby completely solving the fragmentation problem of static subtitles and rolling subtitles, ensuring that the vector text always maintains clarity during scaling and motion, and being free from negative influence of a resolution and an interpolation algorithm.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, motion compensation (MEMC) is widely used in mid-to-high-end display devices. This function effectively improves the smoothness of motion pictures by inserting new frames between consecutive frames, greatly enhancing the user's viewing experience.

[0003] However, the side effects of exercise compensation are also significant and difficult to completely avoid, specifically manifested in the following ways: 1. When a static font is superimposed on a moving background, the correct motion vector of the font should be 0, while the background is moving; because the background and font cannot be processed separately, a fragmented appearance occurs around the font.

[0004] 2. When scrolling text is superimposed on a background, if there is a lot of background content in the motion vector block, the algorithm may not be able to accurately converge to the correct scrolling text motion vector, but instead incorrectly calculates the background motion vector, resulting in some text fragmentation or white spots.

[0005] 3. To prevent font corruption, existing solutions detect the font and assign it a value of 0mv. However, since the detection algorithm is block-level, it wraps around the font, causing the wrapped portion to use static pixels, thus creating a halo around the text edge. Furthermore, the temporal preservation strategy used to protect the font may also result in halo remnants after the OSD disappears.

[0006] Therefore, there is an urgent need to develop a video processing method, device, electronic equipment, and storage medium to solve one or more of the aforementioned problems. Summary of the Invention

[0007] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the embodiments of this application provide a video processing method, apparatus, electronic device and storage medium. The method can effectively separate the text layer and the background layer in the video, and adopt differentiated compensation processing strategies for different layers, thereby improving the smoothness of the video while avoiding motion compensation side effects such as fragmentation, white spots or halos in the text area, and significantly improving the quality of video images containing text information.

[0008] In a first aspect, this application provides a video processing method, the method comprising: The initial video frames of the original video are obtained, and the text information of the initial video frames is extracted using optical character recognition technology. The text information is converted into a structured vector image, and the initial video frame is separated into layers based on the vector image to obtain a text layer and a background layer; The text layer is subjected to frame interpolation compensation to obtain compensated vector text frames; Motion compensation processing is performed on the background layer to obtain the compensated background frame; The compensated vector text frame is combined with the compensated background frame to obtain the processed target video frame. The target video frames are reassembled according to the timestamp sequence of the original video to generate and output the target video frame.

[0009] In one possible implementation, the step of separating the initial video frame based on the vector image to obtain a text layer and a background layer includes: Based on the spatial position features of each basic vector graphic unit in the vector image, a text mask is generated, and the text mask marks the pixel range corresponding to all text in the initial video frame. Based on the text mask, pixel region segmentation is performed on the initial video frame to obtain a first pixel region marked by the mask and a second pixel region not marked by the mask. Pixel cleanup processing is performed on the second pixel area to remove text pixel interference, resulting in the background layer; A pixel-level spatial alignment algorithm is invoked, and the coordinates of the first pixel area are calibrated based on the spatial coordinates of the background layer to obtain the text layer.

[0010] In one possible implementation, the step of performing frame interpolation compensation on the text layer to obtain compensated vector text frames includes: Extract the motion attribute labels of each vector text unit in the text layer; Based on the motion attribute labels, perform inter-frame position comparison analysis on adjacent frame text layers to determine the motion state and motion parameters of each vector text unit, including motion speed and motion direction; Based on the motion state and motion parameters of each vector text unit, a matching motion vector is assigned to each vector text unit; The vector interpolation engine is invoked to perform interpolation operations between two adjacent original text frames based on the motion vector, generating an intermediate vector text frame that conforms to the motion trajectory. The intermediate vector text frame and the original text frame are time-synchronized and checked, and the deviation of the interpolation operation is corrected by a pixel-level position comparison algorithm. The original text frame that passes the verification is integrated with the intermediate vector text to form a compensated vector text frame.

[0011] In one possible implementation, assigning matching motion vectors to each vector text unit based on its motion state and motion parameters includes: For each vector text unit whose motion state is stationary, assign a zero motion vector; For each vector text unit in a rolling state, a matching uniform motion vector is assigned according to the motion speed and direction of each vector text unit.

[0012] In one possible implementation, performing motion compensation processing on the background layer to obtain a compensated background frame includes: Feature points are extracted from the background layer and the background layer of the adjacent previous frame to obtain the feature points of the background layer and the background layer of the adjacent previous frame. The feature points include background corner points and edge points. Based on the feature point matching algorithm, the feature points of the background layer are matched with the background layer of the adjacent previous frame to obtain the initial motion vector field. The initial motion vector field is smoothed to obtain the optimized motion vector field; The motion compensation interpolation engine is invoked, and pixel interpolation is performed on the missing frame positions between the background layer and the background layer of the adjacent previous frame based on the optimized motion vector field to generate an intermediate compensated background frame that conforms to the background motion trajectory. The intermediate compensated background frame and the original background frame are checked for consistency to correct the deviation of pixel interpolation; The original background frame that passes the verification is combined with the intermediate compensated background frame to form the compensated background frame.

[0013] In one possible implementation, the extraction of text information from the initial video frame using optical character recognition technology includes: Image optimization processing is performed on the initial video frame to improve the pixel distinction between text and background in the initial video frame; A deep learning object detection model is used to perform a full-domain scan of the optimized initial video frame to determine the text region of the initial video frame. The full-dimensional features of the text region are extracted to obtain the text information of the initial video frame. The full-dimensional features of the text include character semantic features, color attribute features, spatial position features, and motion state features.

[0014] In one possible implementation, converting the text information into a structured vector image includes: Based on the semantic features of the characters in the text information, each character is mapped to a vector path containing contour node coordinates, curve parameters, and filling rules; The color attribute features in the text information are bound to the vector path corresponding to each character to generate a basic vector graphic unit with attached color attributes; Based on the spatial location features in the text information, the basic vector graphic units of each character are laid out and arranged. Based on the motion state features in the text information, motion attribute labels are added to each basic vector graphic unit to obtain a structured vector image. The motion attribute labels include motion state identifiers and preset motion parameter fields.

[0015] Secondly, this application provides a video processing apparatus, the apparatus comprising: The acquisition module is used to acquire the initial video frames of the original video and extract the text information of the initial video frames using optical character recognition technology. The layer separation module is used to convert the text information into a structured vector image, and perform layer separation on the initial video frame based on the vector image to obtain a text layer and a background layer; The first compensation module is used to perform frame interpolation compensation processing on the text layer to obtain the compensated vector text frame; The second compensation module is used to perform motion compensation processing on the background layer to obtain a compensated background frame. The compositing module is used to combine the compensated vector text frame with the compensated background frame to obtain the processed target video frame. The output module is used to reassemble the target video frames according to the timestamp sequence of the original video to generate and output the target video frame.

[0016] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video processing method described in any embodiment of the first aspect.

[0017] Fourthly, this application also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the video processing method described in any embodiment of the first aspect.

[0018] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application, by separating video frames into layers, ensures that the motion vectors of the background no longer interfere with the text, thereby completely solving the problem of fragmented static and scrolling subtitles. Furthermore, rendering and motion processing the text layer as an independent vector object makes its edges clearer and sharper, completely eliminating the Halo halo and its residual effects caused by block-level detection and incorrect motion vector application. Simultaneously, by compensating the text layer and background layer separately before compositing, it ensures that the vector text maintains its clarity during scaling and motion, unaffected by the negative impact of resolution and interpolation algorithms. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0022] Figure 1 A flowchart illustrating a video processing method provided in an embodiment of this application; Figure 2 A schematic diagram illustrating a text layer compensation process provided in an embodiment of this application; Figure 3 A schematic diagram of motion vector estimation provided for an embodiment of this application; Figure 4 A schematic diagram illustrating a background layer compensation process provided in an embodiment of this application; Figure 5 A schematic diagram illustrating a text information conversion process provided in this application embodiment; Figure 6 This is a schematic diagram illustrating the steps of a video processing method provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0025] In order to solve the problems in existing technologies Because the background and text cannot be processed separately, fragmentation occurs around the text. When scrolling text is overlaid on the background, if there is a lot of background content in the motion vector block, the algorithm may not converge to the correct scrolling text motion vector, but instead calculates the background motion vector, resulting in some text fragmentation or white spots. Furthermore, because the detection algorithm is block-level, it wraps around the text, causing the wrapped portion to use static pixels, thus creating a halo around the text edges. This application provides a video processing method, apparatus, electronic device, and storage medium that can completely solve the fragmentation problem of static and scrolling text by separating video frames into layers, ensuring that the background motion vector no longer interferes with the text. This ensures that vector text maintains clarity during scaling and motion, unaffected by resolution and interpolation algorithms. By rendering and motion processing the text layer as an independent vector object, its edges are made sharper and clearer, completely eliminating the halo and its residual effects caused by block-level detection and incorrect motion vector application.

[0026] Figure 1 This is a flowchart illustrating a video processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method specifically includes: S101. Obtain the initial video frame of the original video and extract the text information of the initial video frame using optical character recognition technology.

[0027] Optical character recognition (OCR) technology refers to the technology of converting printed or handwritten text in paper documents, images, or video frames into computer-recognizable text characters using optical devices (such as scanners and cameras). Its core principle is to utilize image processing, pattern recognition, and artificial intelligence algorithms to preprocess the input image (such as grayscale conversion, binarization, noise reduction, and tilt correction), then perform character segmentation, decomposing the text region in the image into individual characters or character units. Next, feature extraction (such as extracting character contours, strokes, and structural features) is compared and matched with a pre-set character template library, ultimately converting the text symbols in the image into corresponding text codes (such as ASCII or Unicode), thereby achieving the digitization and editability of text information.

[0028] In this embodiment, optical character recognition technology is used to extract text information from the initial video frame, aiming to accurately locate and obtain the content, position and related attribute features of all text in the video frame, laying the foundation for subsequent layer separation and differential compensation processing.

[0029] S102. Convert the text information into a structured vector image, and perform layer separation on the initial video frame based on the vector image to obtain a text layer and a background layer.

[0030] Layer separation refers to dividing video frame images into independent image layers according to the attribute characteristics of different visual elements, so as to perform differentiated processing operations on the content of different layers.

[0031] In this embodiment, the text information to be processed is transformed into a structured vector image using specific technical means, ensuring that the text information is presented in a more accurate and easier-to-operate form. After obtaining the vector image, the pixel areas belonging to the text part are separated from the pixel areas of the non-text part based on the vector image. Through this layer separation method, the text layer and the background layer can be treated as two independent processing objects, creating conditions for subsequent targeted frame interpolation compensation and motion compensation processing, thereby avoiding the image quality problems caused by mutual interference between text and background in the traditional overall processing method.

[0032] The pixel areas containing text undergo further coordinate calibration and optimization to form a separate text layer. This layer contains only the text information from the video frames, and the text exists in the form of vector graphic units, maintaining extremely high clarity and editability. Meanwhile, the pixel areas containing non-text, after removing any potential text pixel interference, form a background layer. This layer contains all background content from the video frames except for the text, such as scene visuals and moving objects.

[0033] S103. Perform frame interpolation compensation on the text layer to obtain the compensated vector text frame.

[0034] Frame interpolation compensation refers to inserting intermediate transition frames that conform to the movement patterns of text between two adjacent original text frames to improve the smoothness of the text layer during video playback and avoid text jumps or stuttering.

[0035] In this embodiment, the text layer is subjected to frame interpolation compensation processing. By inserting additional frame data on the basis of the original text layer, possible gaps or discontinuities between frames are filled, thereby achieving a smoother and more natural visual effect. The final compensated vector text frame not only retains the clarity and editability of the original text, but also further optimizes the smoothness and coherence of dynamic display, providing higher quality basic materials for subsequent text animation or video rendering.

[0036] S104. Perform motion compensation processing on the background layer to obtain the compensated background frame.

[0037] Motion compensation processing refers to predicting and supplementing the background content between adjacent frames by analyzing the motion trajectory and motion patterns of objects in the background layer, so as to reduce the blurring, ghosting or jumping of the background during dynamic changes and improve the stability and clarity of the background image.

[0038] In this embodiment, motion compensation processing is performed on the background layer. Specifically, for the displacement of moving objects (such as moving vehicles, pedestrians, shaky camera shots, etc.) or the overall scene that may exist in the background layer, the intermediate compensated background frame is generated by extracting background feature points, matching motion vectors, and performing interpolation operations. This intermediate compensated background frame can accurately reflect the continuous motion state of the background. Then, it is integrated with the original background frame to obtain the compensated background frame. The compensated background frame can effectively eliminate the background distortion problem caused by insufficient frame rate or inaccurate motion estimation, providing a high-quality background foundation for subsequent synthesis with the text layer.

[0039] S105. Combine the compensated vector text frame with the compensated background frame to obtain the processed target video frame.

[0040] In this embodiment, according to the spatial correspondence between text and background in the original video frame, the text elements in the vector text frame are accurately superimposed onto the corresponding positions in the background frame. The text layer and the background layer are then merged at the pixel level using a layer fusion algorithm to generate a processed target video frame that retains the clarity and sharpness of the vector text while also possessing dynamic continuity of the background. This fully reproduces the content of the original video frame and effectively solves problems such as easily broken text and halo effects at the edges in traditional processing methods, significantly improving the overall quality of the video image.

[0041] S106. Reassemble the target video frames according to the timestamp sequence of the original video to generate and output the target video frame.

[0042] In this embodiment, all processed target video frames are arranged sequentially according to their corresponding original video timestamps to construct a target video frame sequence that is strictly synchronized with the original video timeline. The recombined frame sequence is compressed and encoded into a video data stream conforming to a preset format (such as MP4, AVI, etc.) through video encoding technology, and finally a smooth and clear target video picture is generated and output, ensuring that the temporal continuity and content integrity of the output video are consistent with the original video during playback.

[0043] The video processing method provided in this application, by separating video frames into text and background layers and performing differential compensation processing, can completely solve the problem of fragmented static and scrolling subtitles caused by mutual interference between text and background in traditional video processing. Furthermore, after converting the text information into structured vector images, the text exists in the form of vector graphic units, maintaining extremely high clarity throughout scaling and motion, unaffected by the negative impact of resolution and interpolation algorithms. Simultaneously, rendering and motion processing the text layer as an independent vector object makes its edges sharper and clearer, completely eliminating Halo halos and their residue caused by block-level detection and erroneous motion vector applications, thereby significantly improving the presentation quality of text information and the overall visual effect in the video.

[0044] In an optional embodiment of the present invention, the step of performing layer separation on the initial video frame based on the vector image to obtain a text layer and a background layer includes: Based on the spatial position features of each basic vector graphic unit in the vector image, a text mask is generated, and the text mask marks the pixel range corresponding to all text in the initial video frame. A text mask is a binary image used to mark a specific region (in this embodiment, a text region) in an image, wherein the pixel values ​​of the text region are set to preset foreground values, and the pixel values ​​of the non-text region are set to preset background values.

[0045] In this embodiment, based on the precise coordinate information of each basic vector graphic unit in the vector image under the coordinate system of the initial video frame, a closed region that completely overlaps with the text area is drawn on a blank canvas with the same size as the initial video frame using a graphics drawing algorithm. Pixels within the closed region are marked as foreground values, and pixels outside the region are marked as background values, thereby generating the corresponding text mask.

[0046] Based on the text mask, pixel region segmentation is performed on the initial video frame to obtain a first pixel region marked by the mask and a second pixel region not marked by the mask. In this embodiment, the generated text mask is used as the basis for region division to traverse and check the pixel matrix of the initial video frame. For each pixel, if the pixel value at the position corresponding to it in the text mask is a foreground value, then the pixel is determined to belong to the text region and is assigned to the first pixel region; if the pixel value at the position corresponding to it in the text mask is a background value, then the pixel is determined to belong to the background region and is assigned to the second pixel region.

[0047] Pixel cleanup processing is performed on the second pixel area to remove text pixel interference, resulting in the background layer; Pixel cleanup is performed to completely eliminate any remaining text pixels or text-related noise in the second pixel area, ensuring the purity of the background layer.

[0048] In this embodiment, edge detection is first performed on the second pixel region to identify any isolated high-brightness pixels, fine lines, or residual pixel clusters similar to the strokes of text that may exist within the region. Then, by combining the position coordinates and contour features of the extracted text information, a screening model for text interference pixels is constructed. By comparing the grayscale values, texture features, and spatial distribution patterns of pixels, pixels belonging to text interference are accurately located and marked. Finally, the neighboring pixel interpolation replacement method is used to replace the marked text interference pixels with the mean of the surrounding background pixels or to smooth them using Gaussian filtering. This ensures that the processed second pixel region visually blends naturally with the surrounding background without obvious traces of text residue, ultimately forming a clean and complete background layer.

[0049] A pixel-level spatial alignment algorithm is invoked, and the coordinates of the first pixel area are calibrated based on the spatial coordinates of the background layer to obtain the text layer.

[0050] Pixel-level spatial alignment algorithms are algorithms that can precisely adjust the spatial relationship between different image regions at the pixel scale. Their core lies in establishing a mapping relationship between a reference coordinate system and the target region to achieve high-precision matching of their geometric positions.

[0051] In this embodiment, when calling the pixel-level spatial alignment algorithm, the key feature points of the background layer are first extracted and its spatial coordinate system is established. Then, the text region feature points corresponding to the feature points of the background layer are extracted in the first pixel region. By calculating the spatial transformation parameters between the two sets of feature points, a mapping function from the first pixel region to the coordinate system of the background layer is constructed. Finally, based on the mapping function, the coordinates of each pixel in the first pixel region are transformed and repositioned to ensure that each pixel in the text region can be accurately aligned to its original spatial position in the background layer, thereby obtaining a text layer that is completely matched with the background layer in space and without misalignment.

[0052] The initial video frame layer separation method provided in this application first generates a text mask for the marked text region, then segments the initial video frame into pixel regions based on the mask, subsequently performs targeted pixel purification processing on the segmented non-text regions to completely eliminate text pixel interference, and finally performs pixel-level coordinate calibration on the text region using the clean background layer as a spatial reference, achieving precise separation of the text layer and the background layer. This not only ensures the integrity and independence of the text information in the text layer, but also guarantees the purity of the background layer, providing a clear and interference-free processing object for subsequent vector frame interpolation compensation of the text layer and motion compensation of the background layer, fundamentally avoiding various image quality problems caused by the mutual influence between text and background in traditional block-level processing.

[0053] Figure 2 This is a schematic diagram of a text layer compensation process provided in an embodiment of this application, as shown below. Figure 2 As shown, the step of performing frame interpolation compensation on the text layer to obtain compensated vector text frames includes: S201. Extract the motion attribute labels of each vector text unit in the text layer.

[0054] Motion attribute labels are structured data labels used to describe the motion state of vector text units in a video frame sequence. Specifically, they include key parameters such as the motion type, motion speed, motion direction, motion start timestamp, and motion end timestamp of the text unit.

[0055] In this embodiment, by tracking and analyzing the position coordinates, size parameters, and angle information of the same vector text unit in multiple consecutive frames of text layers, a motion state classification algorithm is used to identify its motion mode, and its motion parameters are recorded in combination with time axis information, thereby generating a unique corresponding motion attribute label for each vector text unit, providing accurate motion basis for subsequent frame interpolation compensation.

[0056] S202. Based on the motion attribute label, perform inter-frame position comparison analysis on adjacent frame text layers to determine the motion state and motion parameters of each vector text unit, wherein the motion parameters include motion speed and motion direction.

[0057] In this embodiment, for two adjacent frames of text layers, firstly, based on the motion start timestamp and end timestamp in the motion attribute label, vector text units in active motion state are selected; then, position feature data such as center coordinates, bounding box vertex coordinates and rotation angles of these vector text units in the two adjacent frames are extracted; by calculating the difference between the corresponding position feature data of the same vector text unit in the two frames, and combining the time interval between the two frames, its motion speed and motion direction are determined.

[0058] For stationary vector text units, their motion state is marked as "static" and the motion parameters are set to zero by default. For vector text units that are scaled or deformed, their size change rate and deformation coefficient need to be calculated as supplementary motion parameters.

[0059] S203. Based on the motion state and motion parameters of each vector text unit, assign a matching motion vector to each vector text unit.

[0060] In this embodiment, for vector text units in a "static" motion state, a zero motion vector is assigned to ensure that their position remains unchanged during frame interpolation. For vector text units in a translational motion state, a linear motion vector is generated based on the calculated motion speed and direction. The magnitude of this vector is proportional to the motion speed, and its direction is consistent with the motion direction. For vector text units with scaling motion, a scaling factor determined by the size change rate is superimposed on the translational motion vector to form a composite motion vector containing scaling information. For vector text units with rotational motion, a three-dimensional motion vector containing rotation parameters is further generated by combining the rotation angle change rate to accurately describe their rotation trajectory.

[0061] S204. Call the vector interpolation engine to perform interpolation operations between two adjacent original text frames based on the motion vector, and generate an intermediate vector text frame that conforms to the motion trajectory.

[0062] The vector interpolation engine is the core processing module used to perform interpolation calculations on vector graphic units. It can generate intermediate transition vector graphics with continuous motion characteristics based on the geometric properties and motion vector parameters of vector graphics.

[0063] In this embodiment, when the vector interpolation engine is invoked, each vector text unit and its corresponding motion vector in two adjacent original text frames are first input to the engine. The engine selects a matching vector interpolation algorithm according to the type of motion vector: for translational motion, linear interpolation is used to calculate the coordinate position of the vector text unit in the intermediate frame; for scaling motion, proportional interpolation is performed between the original size and the target size based on the scaling factor; for rotational motion, angle interpolation is performed based on the rate of change of the rotation angle; for composite motion vectors containing multiple motion forms, the engine fuses the motion parameters of different dimensions through matrix transformation and performs multi-dimensional collaborative interpolation operations.

[0064] It should be noted that during the interpolation process, the vector interpolation engine strictly follows the geometric characteristics of vector graphics to ensure that the text outline, stroke thickness, and fill attributes in the intermediate vector text frame are consistent with the original vector text unit, thus avoiding text deformation or loss of details caused by interpolation.

[0065] S205. Perform time synchronization verification on the intermediate vector text frame and the original text frame, and correct the deviation of the interpolation operation through a pixel-level position comparison algorithm.

[0066] Timing synchronization verification is to ensure that the position of the intermediate vector text frame on the time axis strictly matches the timestamp sequence of the original text frame, avoiding time misalignment caused by interpolation calculation errors.

[0067] In this embodiment, the timestamp information of the original text frame and the time interval between adjacent frames are first obtained. The theoretical timestamp corresponding to each intermediate vector text frame is calculated according to the preset number of interpolation frames. Then, the actual timestamp of the generated intermediate vector text frame is compared with the theoretical timestamp. If the deviation between the two exceeds the preset threshold, it is corrected by adjusting the interpolation step size or the time weight parameter of the motion vector to ensure the accuracy of the intermediate frame in the time dimension.

[0068] The pixel-level position comparison algorithm compares the pixel coordinates of key feature points between the intermediate vector text frame and the original text frame to quantitatively evaluate the positional deviation after interpolation and correct any possible errors.

[0069] In this embodiment, key feature pixels such as vertices and turning points of text units are selected from the original text frame, and their absolute coordinates in the background layer coordinate system are recorded. Corresponding feature pixels of the same text units are extracted in the intermediate vector text frame, and the coordinate deviation between the two is calculated. If the deviation exceeds the set pixel-level precision threshold, the motion vector parameters of the vector text units in the intermediate vector text frame are fine-tuned based on the deviation direction and magnitude. Through iterative calculation, the coordinate deviation of the feature pixels is converged to the threshold range, thereby achieving sub-pixel-level alignment between the intermediate vector text frame and the original text frame in spatial position.

[0070] S206. Integrate the verified original text frame with the intermediate vector text to form a compensated vector text frame.

[0071] In this embodiment, intermediate vector text frames that have undergone time synchronization verification and pixel-level position deviation correction are inserted into the original text frame sequence according to their corresponding interpolation timestamps to construct a complete text frame sequence containing the original keyframes and interpolated intermediate frames. Then, the integrated text frame sequence is subjected to inter-frame consistency checks to ensure that the movement transition of text units between adjacent frames is smooth without jumps and that stroke details are continuous without breaks. Finally, a compensated vector text frame that is continuous in the time dimension and accurately aligned in the spatial position is formed, providing high-quality text layer data for subsequent pixel-level fusion with the background layer.

[0072] The text layer compensation method provided in this application extracts the motion attribute labels of vector text units and performs inter-frame position comparison analysis to accurately determine the motion state and parameters of each text unit, and then assigns a matching motion vector to it. On this basis, a vector interpolation engine is called to perform targeted interpolation operations to generate intermediate vector text frames, and the interpolation deviation is strictly corrected through temporal synchronization verification and pixel-level position comparison algorithms. Finally, the original text frames that have passed the verification are integrated with the intermediate vector text frames. This not only ensures the consistency of the geometric features of the vector text units during the motion process and effectively avoids text deformation or loss of details caused by interpolation, but also achieves continuous smoothness of the text layer in the time dimension and accurate alignment in the spatial position.

[0073] In an optional embodiment of the present invention, the step of assigning matching motion vectors to each vector text unit based on the motion state and motion parameters of each vector text unit includes: For each vector text unit in a stationary state, a zero motion vector is assigned; for each vector text unit in a rolling state, a matching uniform motion vector is assigned based on the motion speed and direction of each vector text unit.

[0074] In this embodiment, for vector text units whose motion state is stationary, their position coordinates, size, and angle information in the video frame sequence remain unchanged. Therefore, zero motion vectors are assigned to ensure that the position of such text units in the intermediate vector text frames is completely consistent with the original text frames during the frame interpolation compensation process, thus avoiding meaningless positional shifts.

[0075] For a vector text unit in a scrolling state, first obtain its scroll start position coordinates, scroll end position coordinates, and scroll cycle parameters in adjacent original text frames. Based on the straight-line distance between the scroll start position and the end position and the scroll cycle, calculate the scroll displacement per unit time, which serves as a quantitative indicator of the motion speed. At the same time, determine the motion direction through the direction vector pointing from the start position to the end position.

[0076] For example, such as Figure 3 As shown, for vector text layers: precise motion vectors are assigned to them based on the text's properties (static or scrolling). Static text is assigned a zero motion vector (0MV), while scrolling text is assigned a uniform motion vector based on its scrolling speed. Subsequently, frame interpolation compensation is performed on the text layer based on this precise vector to generate motion-compensated vector text frames.

[0077] The small rectangle Pr in the figure corresponds to the local text layer space where the vector text unit is located. The coordinate point (x, y) in it can be used as the "scrolling start position coordinates" of the scrolling vector text unit, and the reference point (x_r, y_r) corresponds to the "scrolling end position coordinates". The parent space P corresponds to the global space of the original video frame, and the area B in the parent space is the mapping area of ​​the vector text unit in the global space.

[0078] Specifically, in the local space Pr, the direction vector from the starting position (x,y) to the ending position (x_r,y_r) can be determined by connecting the starting position (x,y) and the ending position (x_r,y_r) (combined with the direction vectors i and j), which is the direction of movement in the scrolling state. At the same time, using the straight-line distance between (x,y) and (x_r,y_r) and the scrolling period parameter, the amount of scrolling displacement per unit time (i.e., the movement speed) can be calculated. These motion parameters (speed and direction) in the local space can be synchronized to the global space through the mapping relationship between Pr and the parent space P (such as the correspondence between region B* and B), ensuring that the trajectory of the scrolling motion of the vector text unit in the global video frame is consistent with the motion law in the local text layer.

[0079] Figure 4 This is a schematic diagram of a background layer compensation process provided in an embodiment of this application, as shown below. Figure 4 As shown, the motion compensation processing of the background layer to obtain the compensated background frame includes: S401. Extract feature points from the background layer and the background layer of the adjacent previous frame to obtain feature points of the background layer and the background layer of the adjacent previous frame. The feature points include background corner points and edge points.

[0080] In this embodiment, an improved SIFT algorithm is used to extract feature points from the background layer. Gaussian blurring and differencing are performed on background layers at different scales to generate multi-scale candidate regions for feature points. Then, the stability of the candidate points is determined by the Hessian matrix, and background corner points with scale invariance are selected. Simultaneously, the Canny edge detection operator is used to extract edges from the background layer, obtaining edge segments in the image, and edge feature points are uniformly sampled on the edge segments. The extracted background corner points and edge points are combined to form the feature point set of the background layer, ensuring that the feature points can represent both the local details of the background image and reflect the overall structural features. For the background layer of the adjacent previous frame, the same feature point extraction process is performed to obtain its corresponding feature point set, providing a matching feature basis for subsequent inter-frame motion estimation.

[0081] S402. Based on the feature point matching algorithm, the feature points of the background layer are matched with the background layer of the adjacent previous frame to obtain the initial motion vector field.

[0082] In this embodiment, a feature point matching strategy based on the nearest neighbor distance ratio is first adopted to perform preliminary matching on the feature point set of the current background layer and the adjacent previous frame background layer. The Euclidean distance between the descriptor of the feature point of the current frame and all feature point descriptors of the previous frame is calculated. The two candidate matching points with the closest distance are selected. If the ratio of the nearest distance to the second nearest distance is less than a preset threshold, it is determined to be a valid matching pair, thereby obtaining the initial feature point matching pair set.

[0083] Then, based on the coordinate information of the matching pairs, an initial motion vector field representing the overall motion trend of the background layer is generated by calculating the pixel displacement difference between the current frame and the previous frame for each pair of matching points. Each feature point matching pair corresponds to an initial motion vector, where the direction of the vector points from the feature point in the previous frame to the corresponding feature point in the current frame, and the magnitude of the vector is the pixel distance between the two feature points.

[0084] S403. Perform smoothing on the initial motion vector field to obtain the optimized motion vector field.

[0085] In this embodiment, an outlier removal algorithm based on median filtering is first used to identify and filter isolated outlier motion vectors in the initial motion vector field. A neighborhood window of a preset size is constructed with each motion vector as the center, and the median magnitude and median direction angle of all motion vectors within the window are calculated. If the deviation between the magnitude and the median magnitude of the current motion vector exceeds a preset magnitude threshold, or the deviation between the direction angle and the median direction angle exceeds a preset direction threshold, then the vector is determined to be an outlier and replaced with the median neighborhood vector.

[0086] Then, the motion vector field is optimized by combining global motion estimation and local motion compensation: the global motion model is estimated from the remaining motion vectors using the random sampling consensus algorithm to separate the overall motion trend of the background; for local areas with large deviations from the global motion model, the block matching optimization algorithm is used to divide the area into overlapping sub-blocks, and local smoothing is performed based on the statistical characteristics of the motion vectors within the sub-blocks to control the gradient of motion vector changes in adjacent sub-blocks within a preset range, and finally, the optimized motion vector field with continuous spatial distribution and consistent motion trend is obtained.

[0087] S404. Call the motion compensation interpolation engine and, based on the optimized motion vector field, perform pixel interpolation on the missing frame positions between the background layer and the background layer of the adjacent previous frame to generate an intermediate compensated background frame that conforms to the background motion trajectory.

[0088] The motion compensation interpolation engine is a dedicated processing module that integrates multiple pixel-level interpolation algorithms. It can generate intermediate background frames with spatiotemporal continuity between adjacent background frames based on the motion information of each pixel in the optimized motion vector field.

[0089] In this embodiment, the adjacent previous frame background layer, the current background layer, and the optimized motion vector field are first input into the engine. The engine divides the motion vector field into a grid, dividing the background layer into multiple regular interpolation units, each corresponding to a set of motion vector parameters. For background regions dominated by global motion, the engine uses a reverse mapping interpolation algorithm based on the motion vector field. According to the optimized motion vectors, the pixels of the previous frame background layer are mapped to the corresponding positions in the intermediate compensated background frame, and the pixel values ​​are filled by bilinear interpolation. For background regions with local motion, a layered interpolation strategy is enabled to separate the region from the global background and perform independent forward interpolation calculations based on its local motion vectors to ensure the accuracy of local motion trajectories.

[0090] It should be noted that during the interpolation process, the engine also feathers the boundary areas of the interpolation units and eliminates interpolation traces between adjacent units by weighted averaging of pixel values, so that the generated intermediate compensated background frame maintains a smooth visual transition and avoids block effects or blurring.

[0091] S405. Perform a consistency check between the intermediate compensated background frame and the original background frame, and correct the deviation of pixel interpolation.

[0092] In this embodiment, consistency verification is carried out from two dimensions: spatial domain and temporal domain.

[0093] Specifically, during spatial domain verification, uniformly distributed background feature regions are first selected from the original background frame, and statistical features such as the mean gray level, gradient magnitude, and edge direction histogram of these regions are extracted. Then, the same feature parameters of the corresponding regions in the intermediate compensation background frame are calculated, and the feature similarity is quantified by Bach distance or structural similarity index. If the similarity is lower than the preset spatial threshold, it is determined that there is an interpolation deviation in the region, triggering the region-level pixel correction process—the motion vector field of the deviation region is optimized a second time, and the pixel interpolation weight is recalculated using a back projection algorithm based on regional feature constraints to adjust the gray level distribution of pixels in the region.

[0094] During temporal verification, the optical flow changes between adjacent intermediate compensated background frames are analyzed. The TV-L1 optical flow algorithm is used to calculate the optical flow field of consecutive intermediate frames, and the directional consistency and modulus stability of the optical flow vectors are statistically analyzed. If the frequency of optical flow abrupt changes in a local area exceeds the temporal threshold, it indicates a motion discontinuity problem in the temporal dimension. In this case, the temporal sampling step size of the interpolation engine is dynamically adjusted, and multi-frame joint interpolation optimization is performed on the region to ensure a smooth transition of background motion on the time axis. For isolated pixel-level deviations found during verification, point-to-point correction is performed using neighborhood median filtering, ultimately generating intermediate compensated background frames that maintain a high degree of consistency with the original background frames in both spatial structure and temporal motion.

[0095] S406. Integrate the original background frame that has passed the verification with the intermediate compensated background frame to form the compensated background frame.

[0096] In this embodiment, the interpolated timestamps corresponding to the intermediate compensated background frames are sequentially inserted into the timeline of the original background frame sequence to construct a complete background frame sequence containing the original key background frames and the interpolated intermediate background frames. Then, the integrated background frame sequence undergoes inter-frame pixel value continuity detection, calculating the grayscale difference between corresponding pixels in adjacent background frames to ensure the absolute value of the difference is within a preset pixel fluctuation threshold, avoiding significant brightness jumps. Next, edge structure consistency verification is performed, using the Sobel operator to extract edge contour maps of adjacent frames and checking the positional offset and length change rate of edge line segments using a morphological similarity comparison algorithm to ensure that the object contours in the background remain intact and unbroken during movement. Finally, a global motion trend review is conducted, inputting the integrated background frame sequence into an optical flow analysis model to generate an overall motion trajectory heatmap. This confirms that the direction and magnitude distribution of the motion vectors in the heatmap are consistent with the motion feature labels of the original background layer, ultimately forming a compensated background frame that is continuous and smooth in the time dimension and clear and complete in spatial detail, providing stable and reliable background image data support for subsequent pixel-level fusion with the text layer.

[0097] For example, for textless background layers: since there is no text interference in the background, traditional MEMC algorithms can focus on motion estimation and compensation of the background without having to consider protecting the text, thus generating smoother and more accurate motion-compensated background frames.

[0098] The background layer compensation method provided in this application constructs basic data for background motion analysis by extracting multi-dimensional feature points, generates a high-precision motion vector field by combining outlier removal and global-local collaborative optimization strategies, performs layered pixel interpolation using a professional motion compensation interpolation engine, and corrects deviations through spatiotemporal consistency verification, ultimately integrating to form a high-quality compensated background frame. This method not only accurately captures the global motion trend and local detail changes of the background, effectively avoiding blurring, ghosting, or block effects caused by traditional interpolation algorithms, but also ensures the motion continuity of the background layer in the temporal dimension and the structural integrity in the spatial dimension, laying a solid foundation for the natural integration of dynamic text and complex backgrounds in videos.

[0099] In an optional embodiment of the present invention, the step of extracting the text information of the initial video frame using optical character recognition technology includes: Image optimization processing is performed on the initial video frame to improve the pixel distinction between text and background in the initial video frame; a full-domain scan is performed on the image-optimized initial video frame based on a deep learning object detection model to determine the text region of the initial video frame; the full-dimensional features of the text region are extracted to obtain the text information of the initial video frame, wherein the full-dimensional features of the text include character semantic features, color attribute features, spatial position features, and motion state features.

[0100] In this embodiment, the image optimization process adopts a multi-stage enhancement strategy. First, the local contrast of the text region is improved by an adaptive histogram equalization algorithm. Then, the denoising process based on edge-preserving filtering is performed. Guided filtering is used to suppress Gaussian noise while preserving the sharp edges of the text strokes. Then, the grayscale difference between the text and the background is enhanced by morphological top-hat transformation, so that the difference between the average pixel grayscale value of the text region and the background region is increased to more than 1.8 times that of the original image.

[0101] In addition, for video frames containing colored text, an additional color space conversion module is activated to convert the RGB image to the YCrCb space. The above optimization process is performed separately on the luminance channel to avoid interference from color component fluctuations in text extraction.

[0102] Then, based on a deep learning object detection model, a full-domain scan is performed on the optimized initial video frame to determine the text region of the initial video frame.

[0103] Figure 5This is a schematic diagram of a text information conversion process provided in an embodiment of this application, such as... Figure 5 As shown, converting the text information into a structured vector image includes: S501. Based on the semantic features of the characters in the text information, map each character into a vector path containing contour node coordinates, curve parameters, and filling rules.

[0104] Vector paths include various elements such as contour node coordinates, curve parameters, and fill rules.

[0105] In this embodiment, based on the semantic features of the characters contained in the text information, each character is converted into a vector path. The contour node coordinates can accurately determine the position of the character's outer boundary. The curve parameters are used to depict the curvature, direction, and other subtle morphological features of the character lines. The filling rules determine the filling method of the internal area of ​​the character to ensure that the character has the correct color, texture, or other filling effects in visual presentation, thereby completely and accurately expressing all the visual information that the original character should have based on its semantic features.

[0106] S502. Bind the color attribute features in the text information to the vector path corresponding to each character to generate a basic vector graphic unit with attached color attributes.

[0107] In this embodiment, the color attribute features of each character are parsed from the text information, including RGB color values, transparency parameters, and special color effects. Then, these color attribute features are deeply bound to the vector path of the corresponding character to establish a mapping relationship table between vector path nodes and color parameters. Through this binding method, the basic vector graphic unit not only contains the geometric outline information of the character, but also carries complete color rendering instructions, providing data support for accurate rendering in subsequent video frames.

[0108] For example, for characters with gradient fill, the coordinates of the gradient start point and end point, as well as the corresponding color values, are embedded into the fill rules of the vector path; for characters with shadow effects, parameters such as the shadow offset direction, blur radius, and color overlay mode are recorded and associated with the outline extension attribute of the vector path.

[0109] S503. Based on the spatial position characteristics in the text information, the basic vector graphic units of each character are laid out.

[0110] In this embodiment, the layout information, including character coordinate parameters, character spacing, line spacing, and text alignment, contained in the spatial location features are first analyzed. Then, a local coordinate system matching the background layer is established with the top left corner vertex of the text area in the original video frame as the origin. The basic vector graphic unit of each character is mapped to this coordinate system according to the original spatial location parameters. For multi-line text, the vertical offset of each line vector graphic unit is calculated according to the line spacing parameters to ensure that the relative position between lines is consistent with the original text area.

[0111] Meanwhile, the horizontal spacing between adjacent character vector units is adjusted by the character spacing parameter, and a greedy algorithm is used to optimize the compactness of the character arrangement, avoiding text layout distortion caused by vector conversion.

[0112] S504. Based on the motion state features in the text information, add motion attribute labels to each basic vector graphic unit to obtain a structured vector image. The motion attribute labels include motion state identifiers and preset motion parameter fields.

[0113] In this embodiment, motion state feature parameters are first extracted from the text information, including motion type, motion trajectory equation, motion velocity vector, motion start timestamp, and duration. Then, an independent motion attribute label data structure is created for each basic vector graphic unit. The motion state identifier is used to mark whether the character is in motion (e.g., static is marked as "0", dynamic is marked as "1"). The preset motion parameter field stores the corresponding quantization parameters according to the motion type. By associating these motion attribute labels with the unique identifier of the basic vector graphic unit, the structured vector image not only contains the static visual information of the character, but also fully records its dynamic change pattern on the video timeline, providing a precise motion control basis for the subsequent dynamic synthesis with the compensated background frame.

[0114] Among them, the preset motion parameter field records the velocity components and acceleration coefficients in the X and Y axes for translational motion; records the rotation center coordinates, angular velocity, and rotation direction for rotational motion; records the scaling center coordinates, scaling factor change rate, and scaling direction for scaling motion; and stores the transparency change curve parameters for fade-in and fade-out effects.

[0115] For example, text recognition and vectorization are separated: Optical Character Recognition (OCR) technology is used to accurately locate and recognize all text content in the input raw video frame. The recognized text information (covering elements such as characters, font, size, color, and position) is then converted into structured vector graphic data. In this process, the original video frame is split into two independent layers: one is a vector text layer constructed from vector data, and the other is a textless background layer with the text removed.

[0116] The text information conversion method provided in this application constructs a vector image data model containing multi-dimensional attributes by structurally integrating the character semantics, color attributes, spatial position, and motion state features of text information. This model not only fully preserves the visual presentation details and layout relationships of the original text, but also achieves a precise description of the dynamic behavior of the text through motion attribute labels, providing a standardized data interface for subsequent dynamic rendering in the compensated background frame. It has significant advantages such as infinite scaling without distortion, precise and controllable color rendering, and editable motion parameters, and can perfectly adapt to the motion changes of the video background, ensuring that the text always maintains a clear outline, accurate color, and stable spatial position relationship in complex dynamic backgrounds, laying a high-quality vector graphics foundation for achieving natural integration of text and background.

[0117] Figure 6 This is a schematic diagram illustrating the steps of a video processing method provided in an embodiment of this application, as shown below. Figure 6 As shown, the video processing method includes the following steps: Step 1: The process is initiated, and the system receives the input sequence of raw video frames.

[0118] Step Two: Perform text recognition and vectorization separation. The system's built-in OCR module analyzes the current frame and identifies all on-screen text (OSD) areas. It extracts attributes such as character type, font, font size, color, and position coordinates, and uses the font rendering engine to convert these attributes into vector graphic paths. At this point, two data streams are generated in system memory: a vector text layer and a background layer with the corresponding pixel areas removed.

[0119] Step 3: Perform independent motion compensation for each layer: Text layer processing: The system determines text attributes. For static titles, logos, etc., they are marked as static and assigned 0MV. For scrolling news, subtitles, etc., a unified motion vector is calculated based on their scrolling direction and speed. Based on this vector, the system interpolates between the vector text of two consecutive frames to generate the vector text of the intermediate frame.

[0120] Background layer processing: The system uses a traditional MEMC algorithm for background layers without text. The image is divided into macroblocks, motion estimation and motion compensation are performed, and the background image of the intermediate frame is generated. Since there is no text interference, the calculation of motion vectors is more accurate.

[0121] Step 4: Perform the final compositing. At the time synchronization point, render the generated intermediate frame vector text as a bitmap and perform alpha blending or direct overlay with the intermediate frame background image to synthesize a complete intermediate frame.

[0122] Step 5: Output. Output the synthesized high-quality intermediate frames and combine them with the original frames to form a higher frame rate video stream for playback.

[0123] Figure 7 This is a schematic diagram of the structure of a video processing device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device specifically includes: The acquisition module 701 is used to acquire the initial video frame of the original video and extract the text information of the initial video frame using optical character recognition technology; The layer separation module 702 is used to convert the text information into a structured vector image, and perform layer separation on the initial video frame based on the vector image to obtain a text layer and a background layer; The first compensation module 703 is used to perform frame interpolation compensation processing on the text layer to obtain a compensated vector text frame. The second compensation module 704 is used to perform motion compensation processing on the background layer to obtain a compensated background frame. The compositing module 705 is used to combine the compensated vector text frame with the compensated background frame to obtain the processed target video frame. The output module 706 is used to reassemble the target video frames according to the timestamp sequence of the original video to generate and output the target video frame.

[0124] In one possible implementation, the layer separation module 702 is further configured to generate a text mask based on the spatial position characteristics of each basic vector graphic unit in the vector image, wherein the text mask marks the pixel range corresponding to all text in the initial video frame; perform pixel region segmentation on the initial video frame based on the text mask to obtain a first pixel region marked by the mask and a second pixel region not marked by the mask; perform pixel purification processing on the second pixel region to remove text pixel interference in the second pixel region to obtain a background layer; and call a pixel-level spatial alignment algorithm to perform coordinate calibration on the first pixel region based on the spatial coordinates of the background layer to obtain a text layer.

[0125] In one possible implementation, the first compensation module 703 is further configured to extract motion attribute labels of each vector text unit in the text layer; perform inter-frame position comparison analysis on adjacent frame text layers based on the motion attribute labels to determine the motion state and motion parameters of each vector text unit, the motion parameters including motion speed and motion direction; assign matching motion vectors to each vector text unit based on the motion state and motion parameters; call a vector interpolation engine to perform interpolation operations between two adjacent original text frames based on the motion vectors to generate intermediate vector text frames that conform to the motion trajectory; perform time synchronization verification between the intermediate vector text frames and the original text frames, and correct the deviation of the interpolation operation through a pixel-level position comparison algorithm; integrate the verified original text frames with the intermediate vector text frames to form a compensated vector text frame.

[0126] In one possible implementation, the first compensation module 703 is further configured to assign zero motion vectors to each vector text unit in a stationary state; and to assign matching uniform motion vectors to each vector text unit in a rolling state according to the motion speed and direction of each vector text unit.

[0127] In one possible implementation, the second compensation module 704 is further configured to extract feature points from the background layer and the background layer of the adjacent previous frame, respectively, to obtain feature points of the background layer and the background layer of the adjacent previous frame, the feature points including background corner points and edge points; based on a feature point matching algorithm, perform matching processing on the feature points of the background layer and the background layer of the adjacent previous frame to obtain an initial motion vector field; perform smoothing processing on the initial motion vector field to obtain an optimized motion vector field; call a motion compensation interpolation engine, and based on the optimized motion vector field, perform pixel interpolation on the empty frame positions between the background layer and the background layer of the adjacent previous frame to generate an intermediate compensated background frame that conforms to the background motion trajectory; perform consistency verification between the intermediate compensated background frame and the original background frame to correct the deviation of pixel interpolation; and integrate the original background frame that passes the verification with the intermediate compensated background frame to form a compensated background frame.

[0128] In one possible implementation, the acquisition module 701 is further configured to perform image optimization processing on the initial video frame to improve the pixel distinction between text and background in the initial video frame; perform a full-domain scan on the image-optimized initial video frame based on a deep learning object detection model to determine the text region of the initial video frame; extract the full-dimensional features of the text region to obtain the text information of the initial video frame, wherein the full-dimensional features of the text include character semantic features, color attribute features, spatial position features, and motion state features.

[0129] In one possible implementation, the layer separation module 702 is further configured to: map each character to a vector path containing contour node coordinates, curve parameters, and fill rules based on the character semantic features in the text information; bind the color attribute features in the text information to the vector path corresponding to each character to generate a basic vector graphic unit with attached color attributes; arrange the basic vector graphic units of each character according to the spatial position features in the text information; and add motion attribute labels to each basic vector graphic unit based on the motion state features in the text information to obtain a structured vector image, wherein the motion attribute labels include motion state identifiers and preset motion parameter fields.

[0130] The video processing device provided in this embodiment can be as follows: Figure 7 The video processing apparatus shown can perform, for example Figure 1-6 All steps of mid-video processing, thereby achieving Figure 1-6 For details on the technical effects of the video processing shown, please refer to [link / reference]. Figure 1-6 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, this application embodiment provides an electronic device, including a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804. The memory 803 stores computer programs. When the processor 801 executes the program stored in the memory 803, it implements the video processing steps provided in any of the aforementioned method embodiments. The process involves acquiring the initial video frames of the original video and extracting text information from them using optical character recognition (OCR) technology. The text information is then converted into a structured vector image, and the initial video frames are layered based on this vector image to obtain a text layer and a background layer. Frame interpolation compensation is applied to the text layer to obtain compensated vector text frames. Motion compensation is applied to the background layer to obtain compensated background frames. The compensated vector text frames and the compensated background frames are then combined to obtain the processed target video frame. Finally, the target video frame is reassembled according to the timestamp sequence of the original video to generate and output the target video image.

[0133] In one possible implementation, a text mask is generated based on the spatial position characteristics of each basic vector graphic unit in the vector image, and the text mask marks the pixel range corresponding to all text in the initial video frame; pixel region segmentation is performed on the initial video frame based on the text mask to obtain a first pixel region marked by the mask and a second pixel region not marked by the mask; pixel cleaning processing is performed on the second pixel region to remove text pixel interference in the second pixel region to obtain a background layer; a pixel-level spatial alignment algorithm is called to perform coordinate calibration on the first pixel region based on the spatial coordinates of the background layer to obtain a text layer.

[0134] In one possible implementation, motion attribute labels are extracted from each vector text unit in the text layer; based on the motion attribute labels, inter-frame position comparison analysis is performed on adjacent frame text layers to determine the motion state and motion parameters of each vector text unit, including motion speed and motion direction; based on the motion state and motion parameters of each vector text unit, a matching motion vector is assigned to each vector text unit; a vector interpolation engine is invoked to perform interpolation operations between two adjacent original text frames based on the motion vectors, generating an intermediate vector text frame that conforms to the motion trajectory; the intermediate vector text frame and the original text frame are time-synchronized and verified, and the deviation of the interpolation operation is corrected by a pixel-level position comparison algorithm; the verified original text frame is integrated with the intermediate vector text frame to form a compensated vector text frame.

[0135] In one possible implementation, a zero motion vector is assigned to each vector text unit in a stationary state; and a matching uniform motion vector is assigned to each vector text unit in a rolling state based on the speed and direction of motion of each vector text unit.

[0136] In one possible implementation, feature points are extracted from the background layer and the background layer of the adjacent previous frame to obtain feature points of the background layer and the background layer of the adjacent previous frame, the feature points including background corner points and edge points; based on a feature point matching algorithm, the feature points of the background layer and the background layer of the adjacent previous frame are matched to obtain an initial motion vector field; the initial motion vector field is smoothed to obtain an optimized motion vector field; a motion compensation interpolation engine is invoked, and based on the optimized motion vector field, pixel interpolation is performed on the missing frame positions between the background layer and the background layer of the adjacent previous frame to generate an intermediate compensated background frame that conforms to the background motion trajectory; the consistency of the intermediate compensated background frame and the original background frame is checked to correct the deviation of pixel interpolation; the original background frame that passes the check is integrated with the intermediate compensated background frame to form a compensated background frame.

[0137] In one possible implementation, image optimization processing is performed on the initial video frame to improve the pixel distinction between text and background in the initial video frame; a full-domain scan is performed on the image-optimized initial video frame based on a deep learning object detection model to determine the text region of the initial video frame; the full-dimensional features of the text region are extracted to obtain the text information of the initial video frame, wherein the full-dimensional features of the text include character semantic features, color attribute features, spatial position features, and motion state features.

[0138] In one possible implementation, based on the semantic features of the characters in the text information, each character is mapped to a vector path containing contour node coordinates, curve parameters, and fill rules; the color attribute features in the text information are bound to the vector path corresponding to each character to generate a basic vector graphic unit with attached color attributes; the basic vector graphic units of each character are laid out according to the spatial position features in the text information; and motion attribute labels are added to each basic vector graphic unit based on the motion state features in the text information to obtain a structured vector image, wherein the motion attribute labels include motion state identifiers and preset motion parameter fields.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0141] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A video processing method, characterized in that, include: The initial video frames of the original video are obtained, and the text information of the initial video frames is extracted using optical character recognition technology. The text information is converted into a structured vector image, and the initial video frame is separated into layers based on the vector image to obtain a text layer and a background layer; The text layer is subjected to frame interpolation compensation to obtain compensated vector text frames; Motion compensation processing is performed on the background layer to obtain the compensated background frame; The compensated vector text frame is combined with the compensated background frame to obtain the processed target video frame. The target video frames are reassembled according to the timestamp sequence of the original video to generate and output the target video frame.

2. The method according to claim 1, characterized in that, The step of separating the initial video frame based on the vector image to obtain a text layer and a background layer includes: Based on the spatial position features of each basic vector graphic unit in the vector image, a text mask is generated, and the text mask marks the pixel range corresponding to all text in the initial video frame. Based on the text mask, pixel region segmentation is performed on the initial video frame to obtain a first pixel region marked by the mask and a second pixel region not marked by the mask. Pixel cleanup processing is performed on the second pixel area to remove text pixel interference, resulting in the background layer; A pixel-level spatial alignment algorithm is invoked, and the coordinates of the first pixel area are calibrated based on the spatial coordinates of the background layer to obtain the text layer.

3. The method according to claim 1, characterized in that, The step of performing frame interpolation compensation on the text layer to obtain compensated vector text frames includes: Extract the motion attribute labels of each vector text unit in the text layer; Based on the motion attribute labels, perform inter-frame position comparison analysis on adjacent frame text layers to determine the motion state and motion parameters of each vector text unit, including motion speed and motion direction; Based on the motion state and motion parameters of each vector text unit, a matching motion vector is assigned to each vector text unit; The vector interpolation engine is invoked to perform interpolation operations between two adjacent original text frames based on the motion vector, generating an intermediate vector text frame that conforms to the motion trajectory. The intermediate vector text frame and the original text frame are time-synchronized and checked, and the deviation of the interpolation operation is corrected by a pixel-level position comparison algorithm. The original text frame that passes the verification is integrated with the intermediate vector text to form a compensated vector text frame.

4. The method according to claim 3, characterized in that, The process of assigning matching motion vectors to each vector text unit based on its motion state and motion parameters includes: For each vector text unit whose motion state is stationary, assign a zero motion vector; For each vector text unit in a rolling state, a matching uniform motion vector is assigned according to the motion speed and direction of each vector text unit.

5. The method according to claim 1, characterized in that, The step of performing motion compensation processing on the background layer to obtain the compensated background frame includes: Feature points are extracted from the background layer and the background layer of the adjacent previous frame to obtain the feature points of the background layer and the background layer of the adjacent previous frame. The feature points include background corner points and edge points. Based on the feature point matching algorithm, the feature points of the background layer are matched with the background layer of the adjacent previous frame to obtain the initial motion vector field. The initial motion vector field is smoothed to obtain the optimized motion vector field; The motion compensation interpolation engine is invoked, and pixel interpolation is performed on the missing frame positions between the background layer and the background layer of the adjacent previous frame based on the optimized motion vector field to generate an intermediate compensated background frame that conforms to the background motion trajectory. The intermediate compensated background frame and the original background frame are checked for consistency to correct the deviation of pixel interpolation; The original background frame that passes the verification is combined with the intermediate compensated background frame to form the compensated background frame.

6. The method according to claim 1, characterized in that, The extraction of text information from the initial video frame using optical character recognition technology includes: Image optimization processing is performed on the initial video frame to improve the pixel distinction between text and background in the initial video frame; A deep learning object detection model is used to perform a full-domain scan of the optimized initial video frame to determine the text region of the initial video frame. The full-dimensional features of the text region are extracted to obtain the text information of the initial video frame. The full-dimensional features of the text include character semantic features, color attribute features, spatial position features, and motion state features.

7. The method according to claim 1, characterized in that, The step of converting the text information into a structured vector image includes: Based on the semantic features of the characters in the text information, each character is mapped to a vector path containing contour node coordinates, curve parameters, and filling rules; The color attribute features in the text information are bound to the vector path corresponding to each character to generate a basic vector graphic unit with attached color attributes; Based on the spatial location features in the text information, the basic vector graphic units of each character are laid out and arranged. Based on the motion state features in the text information, motion attribute labels are added to each basic vector graphic unit to obtain a structured vector image. The motion attribute labels include motion state identifiers and preset motion parameter fields.

8. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire the initial video frames of the original video and extract the text information of the initial video frames using optical character recognition technology. The layer separation module is used to convert the text information into a structured vector image, and perform layer separation on the initial video frame based on the vector image to obtain a text layer and a background layer; The first compensation module is used to perform frame interpolation compensation processing on the text layer to obtain the compensated vector text frame; The second compensation module is used to perform motion compensation processing on the background layer to obtain a compensated background frame. The compositing module is used to combine the compensated vector text frame with the compensated background frame to obtain the processed target video frame. The output module is used to reassemble the target video frames according to the timestamp sequence of the original video to generate and output the target video frame.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video processing method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the video processing method according to any one of claims 1 to 7.

Images