Adaptive video restoration method based on digital video technology

Abstract

The invention discloses an adaptive video restoration method based on a digital video technology, and relates to the technical field of image communication. The method comprises the following steps: preprocessing an original video containing hard subtitles and automatically removing the hard subtitles to obtain a video to be repaired; screening a reference frame and a to-be-repaired frame by comparing the original video with the to-be-repaired video; determining a to-be-repaired area according to the pixel gray scale difference; extracting matched feature points, calculating a reference motion vector, estimating possible positions and motion differences of points in the to-be-repaired region in combination with object structure segmentation and region prediction, and generating a corrected motion vector; performing sub-pixel-level interpolation by using the vector, and constructing a corrected reference image; and finally, fusing the corrected image and the to-be-repaired area to generate a repaired video frame. The method can effectively improve the restoration quality of the subtitle shielded area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image communication, in particular to an adaptive video repair method based on digital video technology. BACKGROUND

[0002] With the wide application of digital video technology, in order to ensure the compatibility and stability of hard subtitles on different platforms and terminals, hard subtitles are often used in scenes such as film and television works, educational videos, and conference recordings. However, in the process of video re-creation and secondary editing, removing hard subtitles often produces holes in the original hard subtitle area, seriously damaging the details of the area.

[0003] The existing hard subtitle removal video repair method usually relies on spatial interpolation of surrounding pixels or optical flow hole filling technology, but since the hard subtitle area pixels are covered in a large range, the available pixel points are limited, resulting in a decrease in the repaired picture quality, which is manifested as edge detail blur, color distortion, and increased noise. SUMMARY

[0004] Therefore, the embodiments of the present application provide an adaptive video repair method based on digital video technology, which at least partially solves the problems in the prior art.

[0005] The embodiments of the present application provide an adaptive video repair method based on digital video technology, which comprises: Pretreating an original video containing hard subtitles and automatically removing the hard subtitles to obtain a video to be repaired; Based on the original video and the video to be repaired, comparing and screening reference frames and frames to be repaired; According to the pixel gray difference analysis of the frame to be repaired and the corresponding frame of the original video, the area to be repaired is determined; Constructing a modified motion vector of the area to be repaired, the construction comprising extracting matching feature points of the reference frame and the frame to be repaired, calculating the reference motion vector of the matching feature points; performing object segmentation to obtain a plurality of object regions and identifying an analysis area intersecting with the area to be repaired; performing structure prediction on the analysis area to predict the possible coordinates of the points in the area to be repaired; performing motion mode analysis to predict the movement difference of the area to be repaired; and combining the reference motion vector and the movement difference to generate the modified motion vector; According to the modified motion vector, determining the source position of the pixel sub-pixel corresponding to the area to be repaired, and performing sub-pixel interpolation to obtain the pixel value, to construct a modified reference image; Fusing the modified reference image and the area to be repaired to generate a repaired video frame.

[0006] Optionally, the automatic removal of the hard subtitles comprises: locating the area where the hard subtitles are located by an image recognition algorithm, and replacing or covering the pixels in the area to remove the hard subtitles.

[0007] Optionally, the comparison comprises: calculating hash values of video frames at the same time point in the original video and the video to be repaired respectively, and judging whether the frames are reference frames by bit-by-bit accurate comparison, wherein the hash values are perceptual hash values or other hash types capable of reflecting the content features of the frames.

[0008] Optionally, the pixel gray value difference analysis of the frame to be repaired and the corresponding frame of the original video to determine the repair area comprises: performing gray processing on the frame to be repaired and the corresponding frame of the original video; calculating the difference value of the pixel gray values at the same position, and marking the pixel as a repair pixel point when the difference value exceeds a preset threshold; based on the repair pixel point, identifying and determining the repair area by a region merging algorithm.

[0009] Optionally, the structure prediction of the analysis area to predict the possible coordinates of the points in the repair area comprises: based on the spatial node relationship of the objects to which each feature point in the reference frame belongs and the positions of the corresponding feature points in the frame to be repaired, a structure prediction model is used to predict the possible coordinates of the points in the repair area, and the possible coordinates are obtained by averaging the prediction results of multiple analysis nodes.

[0010] Optionally, the motion pattern analysis to predict the movement difference of the repair area comprises: The motion pattern analysis adopts a difference prediction strategy, and the difference prediction strategy adopts a difference autoregressive moving average model to predict the movement difference.

[0011] Optionally, the construction of the corrected reference image comprises: dividing the repair area into a plurality of neighborhoods centered on feature points; mapping the pixel values of the neighborhoods in the reference frame to the corresponding areas in the frame to be repaired according to the corresponding correction motion vectors; and constructing a corrected reference image corresponding to the repair area.

[0012] Optionally, the fusion of the corrected reference image and the repair area comprises: applying mean filtering to the corrected reference image to obtain a filtered image; and performing pixel-level fusion of the filtered image and the corresponding repair area in the frame to be repaired.

[0013] Optionally, the pre-processing of the original video containing hard subtitles comprises: simultaneously performing spatial domain mean filtering and time domain mean filtering on the original video frames.

[0014] Optionally, the object segmentation to obtain a plurality of object regions comprises: performing edge detection on the reference frame and connecting broken edges through morphological processing to divide the reference frame into a plurality of object regions; and matching the feature points only within the object regions.

[0015] The embodiments of the present application only relate to a self-adaptive video repair method based on digital video technology, which can be executed by a general electronic device with a processor and a memory, or can be deployed in a server, a workstation or a terminal device as a computer program product. When the processor of the device executes the program code stored in the memory, the steps of the present application can be completed.

[0016] The self-adaptive video repair scheme based on digital video technology provided by the embodiments of the present application effectively solves the problems of detail loss in the original region after hard subtitle removal and poor repair effect in the prior art, and has significant beneficial effects.

[0017] Specifically, the method pre-processes and removes the hard subtitles from the original video, and efficiently obtains clean data and reliable reference sources required for repair based on comparison and screening of reference frames and frames to be repaired. The method accurately locates the region to be repaired through pixel-level difference analysis on the frame to be repaired. In particular, the method can construct a corrected motion vector of the region to be repaired, and even in the case where the region damage cannot directly obtain feature points, the motion of the region can be accurately estimated based on the extracted image feature points, the calculated reference motion vector, and the object structure analysis and prediction model. This accurate motion estimation enables subsequent determination of sub-pixel level pixel corresponding source positions and sub-pixel interpolation according to the corrected motion vector, thereby high-precision reconstruction of the image content of the region to be repaired. Finally, the reconstructed image is fused with the original region, and filtering processing is applied to reduce the discontinuity of the boundary, thereby realizing natural connection of the repaired region and the surrounding picture. Through the synergistic effect of the above key stages, the method of the present application overcomes the limitations of traditional methods, realizes high-quality and high-fidelity repair of the region after hard subtitle removal, and significantly improves the video picture effect.

[0018] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the embodiments of the present application can be implemented in accordance with the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced as follows. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 A flowchart of a method for adaptive video inpainting based on digital video technology is schematically shown; Figure 2 A structural diagram of an apparatus for adaptive video inpainting based on digital video technology is schematically shown; Figure 3 A block diagram of a computing processing device for executing the method according to the embodiments of the present application is schematically shown; Figure 4 A storage unit for holding or carrying program codes for implementing the method according to the embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0021] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0022] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0023] It should be noted that the drawings provided in the following embodiments are only to schematically illustrate the basic concept of the present disclosure, and only the components related to the present disclosure are shown in the drawings, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change in shape, number and proportion, and the layout pattern of the components may be more complex.

[0024] In addition, in the following description, specific details are provided in order to facilitate a thorough understanding of examples. However, one skilled in the art will understand that the described aspects can be practiced without these specific details.

[0025] This disclosure provides a video restoration method. The video restoration method provided in this embodiment can be executed by a computing device, which can be implemented as software or a combination of software and hardware. This computing device can be integrated into a server, terminal device, or the like.

[0026] Please see Figure 1 The diagram illustrates a flowchart of an adaptive video restoration method based on digital video technology provided in an embodiment of this application, including the following steps: Step 101: Preprocess the original video containing hard subtitles and automatically remove the hard subtitles to obtain the video to be repaired.

[0027] It should be noted that the original video refers to the source video data to be repaired, which typically contains hard-coded subtitles. The original video can be acquired in various ways, usually by using various video capture devices to capture, digitize, and store video sources in analog or digital formats as digital video files. These video capture devices include, but are not limited to, digital cameras, video capture cards, and other devices capable of digitizing video signals. The acquisition process converts the video signal into a series of digital video frames, which are then stored as the original video file.

[0028] In this embodiment, preliminary image preprocessing is a necessary step after acquiring the original video file. This preprocessing aims to improve the image quality of the video and suppress noise that may affect subsequent processing. Specifically, the preprocessing focuses on applying noise reduction algorithms, including but not limited to filtering algorithms, in the spatial and / or temporal domains of the video data to effectively reduce the high-frequency noise commonly present in video frames. The noise reduction algorithm can achieve image smoothing and noise suppression by processing the pixel values ​​in the video frame and their spatiotemporal neighborhoods, obtaining the preprocessed original video.

[0029] It should be noted that hard-coded subtitles refer to fixed text or graphic information that is directly encoded and burned into the pixel data of each frame of a video, becoming part of the image itself. Unlike soft subtitles, which can be selectively turned off by the player, hard subtitles are fixed and cannot be removed by the player, and their presence directly covers and destroys the original image content below them. Such hard subtitles are commonly found in various video sources, including but not limited to historical footage, educational videos, and conference recordings.

[0030] In this embodiment, an artificial intelligence (AI) algorithm is used to automatically analyze and process the preprocessed original video to detect, locate, and remove the hard subtitles. The removal operation may include, but is not limited to, using image restoration techniques to reconstruct the background area covered by the subtitles, or using other image processing techniques to remove the subtitle information from the image. After the automatic hard subtitle removal process, a video file with the hard subtitles removed is obtained, denoted as the video to be repaired. The video to be repaired forms the basis for subsequent video repair processes.

[0031] Step 102: Based on the original video and the video to be repaired, compare and filter reference frames and frames to be repaired.

[0032] It should be noted that the video to be repaired is based on the preprocessed original video, obtained solely through automatic removal of hard captions. Therefore, for video frames that originally did not contain hard captions, their image content should maintain a high degree of consistency between the corresponding frames in the preprocessed original video and the video to be repaired; however, for video frames that originally contained hard captions and underwent removal processing, pixel differences will occur between corresponding frames in the two videos due to the modification or removal of the hard caption area. This step utilizes these pixel differences to identify frames affected by hard captions.

[0033] In this embodiment, based on the above principle, preferably, a one-to-one image-level hash comparison is performed between the preprocessed original video and the video to be repaired that are at the same time (e.g., have the same frame number or timestamp). Specifically, for each pair of video frames at the same time, a hash value of their image content is calculated. The hash value is a compact digital fingerprint that can represent the unique content features of the image. The hash algorithm can include, but is not limited to, various hash algorithms suitable for image content comparison. Then, the two calculated hash values ​​are compared, for example, by comparing them bit by bit to determine whether the two hash values ​​are completely identical.

[0034] Based on the comparison results of the hash values, the video frames in the video to be repaired are classified and filtered. If the hash value of a frame in the original video is exactly the same as the hash value of the corresponding frame in the video to be repaired at the same time, it is determined that the frame in the video to be repaired was not changed during the removal of hard captions. This indicates that the frame originally did not contain hard captions, and its image content is complete and undamaged. In this case, the corresponding frame in the video to be repaired is marked as a reference frame that does not need to be repaired. The reference frame can be used as the source of image information when repairing the frame to be repaired in subsequent steps. Conversely, if there is any difference between the two hash values, it is determined that the frame in the video to be repaired has been changed due to the modification or removal of hard captions during the removal of hard captions. This indicates that the frame originally contained hard captions and had been removed. In this case, the corresponding frame in the video to be repaired is marked as a frame that needs to be repaired subsequently. The corresponding frame in the original video can be regarded as a damaged frame containing the original hard captions, and its content can be used to assist in the analysis of the original image information of the damaged area.

[0035] Through the above comparison process, this step automatically completes the preliminary classification of all video frames in the video to be repaired, distinguishing them into the reference frame set and the frame set to be repaired. The reference frame set provides a large amount of image information that is undamaged or unaffected by hard captions, while the frame set to be repaired clearly identifies the specific video frames that require subsequent repair processing. This classification result provides a clear data foundation and processing objective for accurate damage area localization and efficient image repair workflow in subsequent steps.

[0036] Step 103: Perform pixel grayscale difference analysis between the frame to be repaired and the corresponding frame of the original video to determine the area to be repaired.

[0037] It should be noted that the frame to be repaired is the video frame whose content has changed compared to the original frame after the automatic hard caption removal operation is performed in step 101, and which is determined in step 102 to have changed content compared to the original frame. The automatic hard caption removal method typically first determines the screen area where the hard captions are located, and then performs pixel-level processing on that area to remove the caption information. During this process, pixels in non-caption areas usually remain unmodified. Therefore, by comparing the frame to be repaired with the damaged frame (i.e., the frame containing the original hard captions in the original video) at the same time in the preprocessed original video, the area whose pixel values ​​have changed solely due to the hard caption removal operation can be accurately isolated, thereby locating the area to be repaired.

[0038] In this embodiment, specifically, for each frame to be repaired, a pixel-level difference analysis is performed on the corresponding damaged frame at the same time in the preprocessed original video to identify changed pixels. First, the image data of the frame to be repaired and the image data of the damaged frame are converted to grayscale to obtain corresponding grayscale images. Then, each pixel position in the grayscale image is traversed, and the grayscale value of the pixel at that position in the frame to be repaired is compared with the grayscale value of the pixel at the same position in the damaged frame. If the absolute value of the difference between the grayscale values ​​of the two corresponding pixels is less than or equal to a preset grayscale difference threshold, it is determined that the pixel at that position has not changed significantly; conversely, if the absolute value of the difference between the grayscale values ​​is greater than the grayscale difference threshold, it is determined that the pixel at that position belongs to a pixel affected or changed by hard caption removal, and it is marked as a pixel to be repaired. The grayscale difference threshold is a preset parameter used to set the identification of pixel differences.

[0039] After identifying all the pixels to be repaired in the frame image to be repaired, these discretely distributed pixels need to be clustered or combined into one or more connected regions, i.e., preliminary damaged regions. (The preliminary damaged region refers to an unmerged cluster of pixels identified from the pixels to be repaired using preliminary local spatial criteria). The identification criteria for the preliminary damaged region can be based on criteria including, but not limited to, the spatial connectivity or local density of the pixels to be repaired. The preliminary damaged region can be generated using various spatial relationship-based methods, including but not limited to clustering or region growing. After generating all the preliminary damaged regions, a region merging process is required to obtain the final set of regions to be repaired. The merging process aims to merge spatially related, adjacent, or overlapping preliminary damaged regions into one or more larger, logically continuous regions to be repaired. The merging rules include, but are not limited to: if two preliminary damaged regions share common pixels or are adjacent to each other in space, they are merged into a unified region to be repaired. By iteratively applying the merging rules until no more regions meeting the conditions can be merged, the precisely defined set of regions to be repaired is finally obtained. The set of regions to be repaired is the target range for pixel reconstruction and texture synthesis using reference image information in subsequent steps.

[0040] Through the aforementioned processes of pixel difference detection, pixel marking to be repaired, region clustering, and region merging, this step accurately locates the specific spatial range within the frame to be repaired that requires subsequent image restoration, i.e., the region to be repaired. The precise definition of this region provides a foundation and guidance for subsequent efficient and accurate pixel reconstruction and texture synthesis of the damaged area using reference image information.

[0041] Step 104: Construct the corrected motion vector for the region to be repaired. The construction includes extracting matching feature points between the reference frame and the frame to be repaired, calculating the reference motion vector of the matching feature points; performing object segmentation to obtain multiple object regions and identifying regions to be analyzed that intersect with the region to be repaired; performing structure prediction on the region to be analyzed to predict the possible coordinates of points within the region to be repaired; performing motion pattern analysis to predict the movement differences of the region to be repaired; and combining the reference motion vector and the movement differences to generate the corrected motion vector.

[0042] It should be noted that within a single frame of a video, there is always a foreground and a background, and some frames may even contain multiple layers of scenery. The shapes of these objects are fixed and do not change, meaning that the same object has the same shape and structure at different times. Therefore, the positional information of the corresponding object in the image is determined based on the intersection points and feature points of numerous structures in the image. In a specific implementation, an algorithm processes the reference frame and the frame to be repaired to obtain the feature corner points in the two frame images.

[0043] In one specific implementation, the feature point extraction method may include, but is not limited to, the following process: first, convert the frame image into an integral image; then, generate multiple images at different scales using box filters of different sizes (e.g., preferably 9×9 and 7×7); subsequently, for each scale image, calculate the matrix determinant value or similar response value of all pixels to generate a response map; then, perform non-maximum suppression on the numerous response values, and may introduce methods including, but not limited to, two-dimensional entropy filtering mechanisms, to finally filter and obtain stable feature points. The threshold for filtering stable feature points can be adaptively obtained through methods including, but not limited to, calculating the global contrast of the image (e.g., the average gradient magnitude). The obtained feature points are recorded as contrast nodes.

[0044] It should be noted that as time changes in the video, the various objects in the frame are constantly changing and moving. That is, different objects occupy different positions in different video frames, but their movement does not change their shape. This makes the feature points of the same object in two different video frames correspond to each other. Therefore, the movement of the object is described based on the positional differences of two corresponding feature points in the video frame image. Reference motion vectors for each node are constructed accordingly. In a specific implementation, the two frame images are first matched using methods including but not limited to brute-force matching, which can be combined with spatial constraints, including but not limited to grid partitioning, to accelerate the matching speed. For example, all comparison nodes within the grid partitioned between the two frame images (e.g., the reference frame and the frame to be repaired) are traversed, and the distance between the corresponding descriptors of the comparison nodes (e.g., Euclidean distance) is calculated. The descriptors are obtained by converting the comparison nodes into high-dimensional floating-point vectors based on methods including but not limited to gradient or wavelet response. By finding methods that minimize the sum of the distances between numerous descriptors, the correspondence between numerous comparison nodes, i.e., the matching feature points, can be obtained.

[0045] In one specific implementation, after obtaining the set of comparison nodes for the reference frame and the frame to be repaired, matching is performed between the comparison nodes to establish a correspondence between the two frames. The feature point matching method aims to find the optimal corresponding point of each comparison node in the reference frame in the frame to be repaired. This method may include, but is not limited to, feature descriptor-based matching methods. By comparing the descriptor similarity of the comparison nodes, and combining methods including, but not limited to, spatial constraints, the correspondence between the comparison nodes in the reference frame and their corresponding matching nodes in the frame to be repaired is determined. Thus, numerous matching pairs are obtained.

[0046] Based on the correspondence (i.e., matching pairs) between the comparison nodes obtained above, the reference motion vector of the matching feature point is calculated. Each matching pair connects the position of the same feature point in the reference frame and the frame to be repaired, and the spatial displacement between these matching points estimates the motion of the region represented by the feature point between frames.

[0047] For example, in a preferred embodiment, the reference motion vector corresponding to each matching pair is obtained by calculating the coordinate difference between the matching points in a two-dimensional coordinate system. The coordinate system, for example, is preset with the geometric center of the frame image as the origin, the rightward direction as the positive X-axis, and the upward direction as the positive Y-axis. The reference motion vector... It can be calculated using the following formula: (1) In the formula, Indicates the comparison node in the reference frame exist Coordinate values ​​on the axis; Indicates the comparison node in the reference frame exist Coordinate values ​​on the axis; Indicates the node in the frame to be repaired compared with the node in the comparison. Matching nodes exist Coordinate values ​​on the axis; Indicates the node in the frame to be repaired compared with the node in the comparison. Matching nodes exist The coordinate values ​​on the axis.

[0048] The above-described coordinate system yields the reference motion vector, which represents the motion of the same location on the object from the time corresponding to the reference frame to the time corresponding to the frame to be repaired. These calculated reference motion vectors reflect the motion of their respective feature points between frames.

[0049] Furthermore, the purpose of object segmentation to obtain multiple object regions and identifying regions to be analyzed that intersect with the region to be repaired is to provide structural context and data sources for subsequent construction of corrected motion vectors. In a frame image, different objects form contacting or non-contact edges. These edges allow for the segmentation of numerous objects in the frame image, and each object has its own key feature nodes. Based on this, numerous contrasting nodes are segmented: for example, first through... The operator performs edge detection on the frame image, detecting the edges of each object in the frame image; then, morphological processing (such as closing operations) is used to connect partially broken edges; subsequently, a line scan algorithm is used to fill the interior of closed edges, dividing the frame image into different object regions. Finally, numerous comparison nodes within the same object region are recorded as comparison nodes of the same object. Furthermore, not all regions in the video need repair, so all object regions need to be compared. If an object region intersects with the region to be repaired, it is recorded as the region to be analyzed, and its internal comparison nodes are recorded as nodes to be analyzed; otherwise, the corresponding object region does not need to be analyzed.

[0050] To construct the corrected motion vector more accurately, particularly for the region to be repaired, the structural information of the image content itself can be used to constrain and refine the motion vector. In one specific approach, the frame to be repaired or the reference frame can undergo object or region segmentation to extract object structural information, for example, through edge detection and region filling.

[0051] Furthermore, regarding structural prediction of the region to be analyzed to predict the possible coordinates of points within the region to be repaired, this process utilizes the structural features of an object for prediction. Due to image blurring or missing data, the corresponding feature points in the region to be repaired are often difficult to detect accurately. However, undamaged and damaged areas on the same object form a whole, and the positional relationships between their feature points are approximately consistent. Therefore, more relative positions can be referenced to determine the possible coordinates of points in the damaged area. It should be noted that the "possible coordinates" mentioned here refer to the predicted position coordinates of missing or unstable points in the region to be repaired, estimated through spatial structure mapping. These coordinates are not actually detected image points, but rather inferred values ​​calculated based on the structural relationships of multiple known feature points, used for subsequent construction of motion difference sequences and correction vector prediction. These possible coordinates provide a method for approximately reconstructing the spatial evolution path of feature points across time frames within regions where they cannot be extracted, and are a core intermediate variable for correction vector generation throughout the repair process.

[0052] In a preferred embodiment, a method based on object structural features preliminarily predicts the possible coordinates of points within the region to be repaired. The prediction method utilizes the positional relationship between the comparison nodes in the reference frame and their corresponding matching nodes in the frame to be repaired, combined with object structural features, to make the inference. The calculation process is as follows: (2) In the formula, Indicates the comparison node in the reference frame The coordinates of the location; Indicates the comparison node in the reference frame The coordinates of the location; Indicates the node in the frame to be repaired compared with the node in the comparison. Matching nodes Location coordinates; Indicates the node in the frame to be repaired compared with the node in the comparison. The total number of nodes to be analyzed within the same region to be analyzed.

[0053] Simply put, its function is to utilize known structural relationships; it assumes that on the same object, points... and points Relative positional relationship in the reference frame , and point and points Relative positional relationships in the frame to be repaired - They are similar. Because the points are... Located in the area to be repaired (image blurred or missing), we don't know its exact location within the frame to be repaired. However, we do know points in other undamaged areas of the same object. The position in the frame to be repaired ( ), and points and points Position in the reference frame ( and For each known point and its corresponding point in the reference frame The formula attempts to estimate points based on structural similarity. The position within the frame to be repaired. The beginning of the formula. Indicates all 10 related known points Summing and averaging are performed. This means it does not depend on only one known point. Instead, it comprehensively considered multiple known points. The provided information is used to average the prediction results, resulting in a more stable and robust point. Possible coordinates in the frame to be repaired .

[0054] Based on the structural characteristics of the object itself and combined with known motion information, the above method preliminarily determines the possible coordinates of potential feature points in the area to be repaired.

[0055] It should be noted that motion pattern analysis is performed to predict the movement differences in the area to be repaired; and the corrected motion vector is generated by combining the reference motion vector and the movement differences. This process aims to predict the movement differences in the area to be repaired and ultimately generate the corrected motion vector. Different locations appear differently in the video. For example, some positions closer to the camera have smaller movement amplitudes, but positions farther from the camera have more obvious movement, and the movement difference (i.e., the reference motion vector) in two video frames will also exhibit different characteristics. Considering that the two nodes to be analyzed are located on the same object, since the movement of the object is certain, the movement of many nodes between these two nodes is also the same. From the perspective of the observing shot, the scale of movement increases with distance. Based on this, the possible motion reference vectors for each point in the area to be repaired are calculated.

[0056] In a preferred embodiment, a prediction method based on motion pattern analysis is employed. The motion within the area to be repaired can be predicted based on the motion patterns corresponding to undamaged areas within the area to be analyzed. This may include constructing a motion vector sequence and a motion difference sequence (for example, first arranging the motion reference vectors corresponding to numerous comparison nodes within the area to be analyzed according to the order of the comparison nodes to obtain a motion vector sequence corresponding to the area to be analyzed, where the node farthest from the node to be repaired is placed at the beginning of the sequence; then differencing the motion vector sequence to obtain a motion difference sequence, i.e., the difference in movement between numerous comparison nodes within the same object in these two video frames), and using a prediction model for prediction. The prediction model may employ time series prediction models, including but not limited to the Differential Autoregressive Moving Average (DARMA) model. Based on the prediction model, the movement differences of each location point within the area to be repaired relative to its previous neighboring node are predicted.

[0057] Finally, based on the movement differences obtained from the above motion pattern analysis, and combined with the original motion vector, a corrected motion vector is generated. This corrected motion vector is typically obtained based on the movement differences and the reference motion vector corresponding to the node to be analyzed closest to the node in the area to be repaired. The calculation process is as follows: (3) In the formula, Indicates the distance from the node to be repaired. The most recent node to be analyzed The corresponding reference motion vector; Indicates the node in the area to be repaired. Relative to the node to be analyzed The corresponding movement differences.

[0058] Thus, by analyzing the frame to be repaired and its corresponding object region, and combining the reference motion vector, object structure information, and motion pattern prediction, corrected motion vectors for each position in the region to be repaired are obtained. This set of corrected motion vectors constitutes motion guidance information describing how each pixel in the region to be repaired should be reconstructed from the reference frame based on motion information.

[0059] Step 105: Based on the corrected motion vector, determine the source position corresponding to the sub-pixel of the pixel in the area to be repaired, and perform sub-pixel interpolation to obtain the pixel value in order to construct a corrected reference image.

[0060] It should be noted that the corrected motion vector indicates the displacement of pixels in the region to be repaired between the frame to be repaired and the reference frame. Based on the corrected motion vector, the corresponding source position of each pixel in the region to be repaired in the reference frame can be determined.

[0061] In a preferred embodiment, constructing the corrected reference image includes the following process. First, the process of dividing the region to be repaired into multiple neighborhoods centered on feature points can be specifically implemented as follows: based on the positions of the nodes to be analyzed identified in step 104 within the region to be repaired and their corresponding corrected motion vectors. In a specific implementation, the division can be achieved, for example, by a clustering method, where the cluster center can be set as a comparison node associated with the region to be repaired, thereby dividing the region to be repaired into numerous small regions centered on the comparison nodes, denoted as the neighborhoods of each comparison node. The number of neighborhoods can be, for example, preset to the total number of comparison nodes associated with the region to be repaired.

[0062] Secondly, for each pixel within the region to be repaired, the correspondence between the pixel and the reference frame is determined based on its neighborhood and the corresponding corrected motion vector, thereby determining its corresponding source position in the reference frame. This correspondence defines the spatial mapping between the pixel position within the region to be repaired and the source position in the reference frame, and this mapping is determined by the corrected motion vector. In a specific implementation, the correspondence between pixels in the frame to be repaired and pixels in the reference frame can be expressed as: (4) In the formula, Indicates the comparison node in the reference frame Within the neighborhood of The coordinates of the pixel; Indicates the comparison node The corresponding corrected motion vector; Indicates the node in the frame to be repaired compared with the node in the comparison. Matching nodes Within the neighborhood of The coordinates of the pixel.

[0063] The formula describes the spatial mapping relationship between pixel positions in the reference frame and corresponding pixel positions in the frame to be repaired. After determining the corresponding source position in the reference frame, the pixel value at that position is obtained from the reference frame.

[0064] It should be noted that the source position is usually a non-integer coordinate. Therefore, in a preferred embodiment, the process of obtaining the pixel value requires sub-pixel interpolation. The sub-pixel interpolation method may include, but is not limited to, bilinear interpolation, bicubic interpolation, or a weighted average based on surrounding integer pixels, etc., to calculate a more accurate pixel value based on the pixel values ​​of integer pixels surrounding the source position coordinates. For example, in sub-pixel interpolation: an image is composed of discrete pixels, each located at an integer coordinate position. However, the corresponding source position (i.e., the sub-pixel corresponding source position) of the pixel in the reference frame calculated based on the modified motion vector is usually a non-integer coordinate. Since the pixel value at a non-integer position cannot be directly obtained, interpolation techniques are required. Sub-pixel interpolation refers to estimating and obtaining the pixel value at a non-integer position based on the pixel values ​​of integer pixels surrounding the non-integer coordinates (e.g., (100, 200), (101, 200), (100, 201), (101, 201)) using a specific algorithm. In step 105, subpixel interpolation is performed to obtain pixel values ​​from the reference frame at the precise location pointed to by the motion vector, in order to ensure the accuracy and quality of the reconstructed image.

[0065] In another embodiment, the acquired pixel values ​​are obtained directly from the reference frame and mapped, and the grayscale values ​​(brightness, etc.) of the pixels are kept unchanged.

[0066] Finally, by repeating the pixel mapping and pixel acquisition process described above, all pixel positions within the region to be repaired are traversed (or processed in units of the neighborhood of the comparison node), and the corresponding pixel values ​​are obtained from the reference frame according to the corrected motion vector, and then filled into the corresponding pixel positions in the corrected reference image. Thus, a corrected reference image with the same spatial range and pixel arrangement as the region to be repaired is constructed.

[0067] Through the above process, this step constructs a corrected reference image with the same spatial range and pixel arrangement as the region to be repaired. The corrected reference image contains image information precisely mapped from the reference frame according to the motion compensation principle, aiming to provide accurate and lossless reference image content for subsequent filling of the region to be repaired.

[0068] Step 106: Fuse the corrected reference image with the area to be repaired to generate a repaired video frame.

[0069] It should be noted that the corrected reference image is directly mapped or reconstructed from the pixels in the reference frame through motion compensation, and typically retains its original pixel value characteristics (e.g., grayscale value or brightness). However, due to differences in video shooting environment, lens parameters, or time points, there may be overall or local differences in brightness between the reference frame and the frame to be repaired. This difference may result in obvious visual discontinuities at the boundary of the area to be repaired when the corrected reference image is directly replaced.

[0070] To address the aforementioned issue of brightness discrepancies and ensure smooth fusion and visual consistency, in one embodiment, when fusing the corrected reference image with the region to be repaired, image processing methods, including but not limited to filtering, can be applied to adjust the pixel values ​​of the region to be repaired or the pixel values ​​of the corrected reference image. The adjustment aims to make the pixels at the boundary of the corrected reference image and the region to be repaired more closely match in terms of brightness characteristics. For example, in one specific implementation, the filtering method can apply mean filtering.

[0071] In this embodiment, the fusion process involves replacing or combining the pixel information of the adjusted correction reference image with the corresponding area to be repaired in the frame to be repaired, thereby filling the area covered by the original hard caption with effective image information from the reference frame.

[0072] The fusion process generates a repaired video frame, wherein the area to be repaired has been filled with pixel information from the reference frame, which has undergone motion compensation and possibly brightness adjustment. Thus, the repaired video frame is obtained.

[0073] Optionally, step 101 includes: locating the region where the hard subtitles are located using an image recognition algorithm, and replacing or covering the pixels within the region to remove the hard subtitles. It should be noted that when automatic hard subtitle removal is performed using methods including but not limited to analyzing the pre-processed original video frames using image recognition algorithms to locate the hard subtitle region and processing the pixels within the located region, the specific implementation of the processing can be varied. For example, an artificial intelligence (AI)-based algorithm model can be used to achieve automatic detection and precise location of the hard subtitle region. The removal operation includes processing the pixels within the located hard subtitle region. The pixel processing methods may include, but are not limited to: using image inpainting technology to reconstruct the background image covered by the subtitles, and filling the subtitle region with reconstructed or predicted background pixels through pixel replacement; or using image segmentation or foreground extraction technology to separate the hard subtitles from the video frame, and then extracting the background image as the result after subtitle removal; or using other pixel replacement, covering, or processing techniques to achieve the purpose of removing the hard subtitles. After automatic hard subtitle removal is achieved through the above methods, a video file with the hard subtitles removed is obtained, denoted as the video to be repaired. The video to be repaired forms the basis for subsequent video repair processes.

[0074] Optionally, step 102 includes: Hash values ​​are calculated for video frames at the same time point in the original video and the video to be repaired, and it is determined whether they are reference frames by comparing them bit by bit. The hash values ​​are perceptual hash values ​​or other hash types that can reflect the content characteristics of the frame image.

[0075] It should be noted that the principle behind using a comparison method to filter reference frames and frames to be repaired is based on the fact that the video to be repaired only changes the pixel content of some frames due to the removal of hard captions. Specifically, the original video and the video to be repaired at the same time (e.g., having the same frame number or timestamp) are compared one by one. For each pair of corresponding frames, a hash value of its image content is calculated. The hash value is a compact digital fingerprint that can represent the unique content features of the image. The hash algorithm can include, but is not limited to, perceptual hashing, difference hashing, and other hash algorithms suitable for image content comparison. Then, the two calculated hash values ​​are compared. The comparison is performed bit by bit to determine whether the two hash values ​​are completely identical. Based on the result of the bit by bit comparison, the video frames in the video to be repaired are classified. If the two hash values ​​are completely identical, it is determined that the frame in the video to be repaired has not been changed by the removal of hard captions, and its original content does not contain hard captions. This frame is marked as a reference frame that does not need to be repaired. Conversely, if there is any difference between the two hash values, it is determined that the frame in the video to be repaired has been altered due to hard caption removal, and that it originally contained hard captions. This frame is then marked as a frame requiring further repair. The corresponding frame in the original video can be considered a corrupted frame containing the original hard captions.

[0076] Optionally, step 103 includes: converting the frame to be repaired and its corresponding original frame to grayscale; calculating the difference between the grayscale values ​​of pixels at the same location, and marking the pixel as a pixel to be repaired when the difference exceeds a preset threshold; and identifying and determining the region to be repaired based on the pixel to be repaired using a region merging algorithm.

[0077] It should be noted that this section details the specific implementation method for locating the region to be repaired based on the frame to be repaired. The principle is based on the fact that the pixel content of some frames in the video to be repaired has changed only due to hard caption removal. By comparing the frame to be repaired with the corresponding damaged frame (i.e., the frame containing the original hard captions in the original video) at the same moment in the preprocessed original video, the region whose pixel values ​​have changed only due to the hard caption removal operation can be accurately isolated, thereby locating the region to be repaired.

[0078] In this embodiment, specifically, for each frame to be repaired, a pixel-level difference analysis is performed on the corresponding damaged frame at the same time in the preprocessed original video to identify the changed pixels. First, the image data of the frame to be repaired and the image data of the damaged frame are converted to grayscale to obtain corresponding grayscale images. Then, each pixel position in the grayscale image is traversed, and the grayscale value of the pixel at that position in the frame to be repaired is compared with the grayscale value of the pixel at the same position in the damaged frame. If the absolute value of the difference between the grayscale values ​​of the two corresponding pixels is less than or equal to a preset grayscale difference threshold... If the difference in grayscale values ​​is greater than the grayscale difference threshold, then the pixel at that location is determined to have not changed significantly; otherwise, if the absolute value of the difference in grayscale values ​​is greater than the grayscale difference threshold, then the pixel at that location is determined to have not changed significantly. If the pixel at that location is determined to be affected or altered due to hard subtitle removal, it is marked as a pixel to be repaired. The grayscale difference threshold... It is a preset parameter used to set the recognition of pixel differences (e.g., preferably, =3).

[0079] After identifying all the pixels to be repaired in the frame image to be repaired, these discretely distributed pixels need to be clustered or combined into one or more connected regions, i.e., preliminary damaged regions. The preliminary damaged regions refer to unmerged pixel clusters identified from the pixels to be repaired using preliminary local spatial criteria (such as spatial connectivity or local density of pixels). The identification criteria for the preliminary damaged regions can be based on criteria including, but not limited to, the spatial connectivity or local density of the pixels to be repaired. For example, when several consecutive pixels to be repaired (e.g., a preset number of consecutive pixels, preferably 6) are detected along a horizontal or vertical direction, the line segment formed by these consecutive pixels can be identified as a preliminary damaged region. Alternatively, when the number of pixels to be repaired within a local neighborhood of a preset size (e.g., preferably a 3×3 pixel area) reaches or exceeds a preset threshold (e.g., preferably 6 / 9 of the total number of pixels in the area), the local neighborhood can be identified as a preliminary damaged region. The preliminary damaged regions can be generated using various spatial relationship-based methods, including but not limited to clustering or region growing. After generating all the initially damaged regions, a region merging process is required to obtain the final set of regions to be repaired. This set of regions to be repaired refers to the final connected region obtained after the merging process, which is the target area for actual pixel reconstruction. The merging process aims to merge spatially related, adjacent, or overlapping initially damaged regions into one or more larger, logically connected regions to be repaired. The merging rules may include, but are not limited to: if two initially damaged regions share a common pixel or are adjacent to each other, they are merged into a unified region to be repaired. By iteratively applying the merging rules until no more regions meeting the conditions can be merged, the precisely defined set of regions to be repaired is finally obtained. This set of regions to be repaired is the target area for pixel reconstruction and texture synthesis using reference image information in subsequent steps.

[0080] Through the above-described process of pixel difference detection, pixel marking to be repaired, region clustering, and region merging, the precise definition of the region to be repaired provides a foundation and guidance for subsequent efficient and accurate pixel reconstruction and texture synthesis of the damaged region using reference image information.

[0081] Optionally, step 104 includes: based on the spatial node relationship of the object to which each feature point belongs in the reference frame and the position of the corresponding feature point in the frame to be repaired, using a structural prediction model to predict the possible coordinates of points in the area to be repaired, wherein the possible coordinates are obtained by averaging the prediction results of multiple analysis nodes.

[0082] In a preferred embodiment, the structural prediction model uses a method based on object structural features to initially predict the possible coordinates of potential points within the area to be repaired. The prediction method can utilize the positional relationship between the comparison nodes in the reference frame and their corresponding matching nodes in the frame to be repaired, combined with the object structural features, to make inferences.

[0083] The area to be analyzed contains both undamaged areas that do not require repair and damaged areas that need repair. Because the image in the damaged area is blurry, the corresponding feature points cannot be accurately detected, or the positions of the feature points are deviated. However, since the damaged area and the undamaged area are on the same object, the positional relationship of the feature points in the damaged area relative to the feature points in the undamaged area is nearly consistent with the positional relationship of the two feature points of the object in the corresponding reference frame. However, considering that the object may have moved slightly during video recording, causing some deformation of the feature structure, it is necessary to refer to more relative positions to determine the possible coordinates of the feature points in the damaged area.

[0084] The calculation process of the possible coordinates, as well as the related formula (Formula 2) and its variable definitions, can be found in the detailed description of structural prediction in the section describing the main body of step 104 above. The possible coordinates are obtained by averaging the prediction results of multiple analysis nodes, which helps to improve the robustness and accuracy of the prediction.

[0085] Optionally, step 104 includes: The motion pattern analysis employs a differential prediction strategy, which uses a differential autoregressive moving average model to predict the motion differences.

[0086] In a preferred embodiment, the motion pattern analysis can predict the motion within the area to be repaired based on the motion patterns corresponding to the undamaged areas within the area to be analyzed. The core of this prediction process lies in performing time-series analysis on the motion data. First, the motion reference vectors corresponding to numerous comparison nodes within the area to be analyzed are arranged in order of distance from the predicted position within the area to be repaired, thereby constructing a reference motion vector sequence. This sequence reflects the trend of motion vector changes with spatial position (or implicit time / structural order) on the same object.

[0087] Given that the motion of objects in a video may not be strictly stationary, and their motion vector sequences may exhibit trends or periodicity, directly predicting future motion may be inaccurate. Therefore, this embodiment employs a differencing operation. Specifically, the motion vector sequence is differencing to obtain a motion difference sequence. This motion difference sequence characterizes the amount of change between two adjacent motion vectors, i.e., the "instantaneous" difference or rate of change in motion. Through differencing, the non-stationarity present in the original motion vector sequence can be eliminated, making the data more stable and thus more easily revealing its inherent patterns and regularities.

[0088] Based on this, the motion pattern analysis employs a differential prediction strategy, which uses a Differential Autoregressive Moving Average (DARMA) model to predict the motion differences. The DARMA model, as a classic time series analysis model, can effectively handle non-stationary time series data with trends or seasonal fluctuations. In this embodiment, the DARMA model predicts the motion differences of each node in the area to be repaired relative to its previous neighbor based on the motion difference sequence values ​​of several past time points (or spatial points) and past prediction errors (moving average part). The model effectively predicts future changes in the sequence by establishing a linear relationship between the current value and past values, as well as with past prediction errors. The DARMA model integrates the autocorrelation information (AR part) of the historical difference sequence and the moving average information (MA part) of the residuals, enabling effective modeling of dynamic changes in local motion patterns. In specific implementation, the autoregressive order p and the moving average order q of the DARMA model can be set according to the motion stationarity between the reference frame and the frame to be repaired. Preferably, a sliding window can be used to perform parameter estimation and model training on the motion vector difference sequences of multiple neighboring comparison nodes to improve the accuracy and robustness of the prediction.

[0089] It should be noted that the result predicted by the DARMA model is the motion difference between a node in the region to be repaired and its adjacent known nodes. This motion difference is used to generate the corrected motion vector by superimposing it on the known reference motion vector. The corrected motion vector is then used to construct the sub-pixel mapping relationship and reconstruct the reference image.

[0090] Optionally, to ensure that the model is applicable to scenes with complex textures or dynamic backgrounds, the differential prediction process can be combined with an error control mechanism to re-mark regions where the prediction residual exceeds a preset threshold as low-confidence regions and hand them over to other alternative methods (such as structural interpolation or image completion-based repair) for further processing.

[0091] Optionally, step 105 includes: dividing the region to be repaired into multiple neighborhoods centered on feature points; mapping the pixel values ​​of the neighborhoods in the reference frame to the corresponding regions in the frame to be repaired according to the corresponding corrected motion vectors; and constructing a corrected reference image corresponding to the region to be repaired.

[0092] It should be noted that, firstly, the region to be repaired is divided into multiple neighborhood regions centered on feature points. This neighborhood division is based on clustering the feature point coordinates in the obtained corrected motion vector. Preferably, K-means clustering, Voronoi diagram partitioning, or methods based on the image distance field can be used to divide the entire region to be repaired into several non-overlapping sub-regions, with the feature point at the center of each sub-region serving as its representative point. This partitioning method ensures that the motion trend within each neighborhood remains relatively consistent in space, thus facilitating subsequent unified mapping and reconstruction processing.

[0093] Furthermore, for each of the neighborhood regions, the pixel mapping position in the reference frame is calculated based on the corrected motion vector corresponding to its central feature point. The corrected motion vector provides the correspondence between the reference frame and the frame to be repaired; using this vector, the original pixel block in the reference frame can be mapped to the current neighborhood position. To improve mapping accuracy, preferably, sub-pixel interpolation technology is used to reconstruct grayscale or color values ​​for the mapping positions of non-integer coordinates. The interpolation method may include, but is not limited to, bilinear interpolation, cubic convolution interpolation, Lanczos interpolation, etc., and the specific selection can be configured according to the actual image clarity requirements.

[0094] Furthermore, by traversing all neighboring regions and performing pixel mapping and filling from the reference frame to the frame to be repaired using the method described above, a complete corrected reference image can be obtained. The corrected reference image structurally preserves the image information of the lossless areas in the reference frame, while also highly matching the original content distribution of the area to be repaired in terms of texture detail and edge coherence, laying the foundation for subsequent image fusion and the generation of the final repaired frame.

[0095] Optionally, after the construction is completed, the corrected reference image can also be subjected to uniform brightness normalization processing to adapt to the brightness deviation caused by changes in illumination between different shooting frames, thereby improving the fusion accuracy.

[0096] Optionally, step 106 includes: applying mean filtering to the corrected reference image to obtain a filtered image; and performing pixel-level fusion of the filtered image with the corresponding region to be repaired in the frame to be repaired.

[0097] In a preferred embodiment, the fusion process includes two sub-processes. First, a mean filter is applied to the corrected reference image to obtain a filtered image. It should be noted that the corrected reference image is directly mapped or reconstructed from the pixels in the reference frame through motion compensation, typically retaining its original pixel value characteristics (e.g., grayscale value or brightness). However, due to differences in video shooting environment, lens parameters, or time points, there may be overall or local differences in brightness between the reference frame and the frame to be repaired. This difference may lead to obvious visual discontinuities at the region boundaries when the corrected reference image is directly replaced with the region to be repaired. To address the aforementioned brightness difference problem and ensure the smoothness and visual consistency of the fusion, a mean filter is applied to the corrected reference image in this step to adjust its pixel values. The mean filter is a standard image smoothing technique that softens the image and adjusts its brightness distribution by averaging the pixel values ​​of a pixel and its neighborhood, resulting in a filtered image.

[0098] Next, the filtered image is fused pixel-level with the corresponding region to be repaired in the frame to be repaired. In this embodiment, the fusion process involves replacing or combining the pixel information of the corrected reference image, adjusted by mean filtering, with the region to be repaired located in step 103 in the frame to be repaired, thereby filling the area originally covered by the hard captions with effective image information from the reference frame. The fusion is performed at the pixel level, that is, replacing or combining the pixel values ​​in the filtered corrected reference image with the corresponding pixel positions in the region to be repaired in the frame to be repaired according to certain rules.

[0099] Optionally, step 101 includes: simultaneously performing spatial domain mean filtering and temporal domain mean filtering on the original video frame.

[0100] In a preferred embodiment, the preprocessing of the original video is a necessary step aimed at improving the image quality of the video and suppressing noise that may affect subsequent processing.

[0101] Specifically, the preprocessing focuses on simultaneously performing spatial domain mean filtering and temporal mean filtering on the original video frames. Spatial domain mean filtering achieves image smoothing and noise suppression by averaging the pixel values ​​of each pixel and its spatial neighborhood in the video frame. Temporal mean filtering, on the other hand, effectively reduces high-frequency noise prevalent in the video frames and enhances inter-frame texture consistency by averaging the pixel values ​​of each pixel and its temporal neighborhood (i.e., corresponding positions in adjacent video frames). This method of simultaneously performing spatial and temporal mean filtering can more effectively remove high-frequency noise from numerous video frames, obtaining the preprocessed original video. The noise reduction algorithm, through the above processing, effectively reduces the prevalent high-frequency noise in the video frames, obtaining the preprocessed original video.

[0102] Optionally, step 104 further includes: Edge detection is performed on the reference frame, and broken edges are connected through morphological processing to divide the reference frame into multiple item regions; the matching feature points are matched only within the item regions.

[0103] It should be noted that this section elaborates on the specific implementation of the above optional steps. This optional step aims to optimize the subsequent feature point matching and motion vector calculation process by utilizing the object structure information in the reference frame.

[0104] In one embodiment, the optional step includes two sub-processes. First, edge detection is performed on the reference frame, and broken edges are connected through morphological processing to divide the reference frame into multiple object regions. This process is based on the image content of the reference frame, using image processing techniques to identify and segment different objects or regions. Preferably, the edge detection can employ common algorithms such as Sobel, Canny, or LoG (Laplacian of Gaussian) operators to identify areas of strong gradient change in the image, thereby outlining the edge contours of potential objects. For example, in a specific implementation, edges in the reference frame can be detected using edge detection operators (such as the Sobel operator); then, potentially broken edge lines are connected through morphological processing (such as closing operations) to form closed contours; next, the internal regions of the closed edges are filled using methods including but not limited to line scan algorithms or filling algorithms, thereby dividing the reference frame into multiple independent object regions. Thus, the object region segmentation result of the reference frame is obtained.

[0105] Secondly, the matching of image feature points to be extracted is limited to the same object region. This process utilizes the object region segmentation results obtained in the previous step to constrain the feature point matching process in subsequent steps (typically step 104, the feature point matching stage). Specifically, after extracting image feature points from the reference frame and the frame to be repaired, feature point matching is only attempted between feature points located within the same object region. For example, a feature point located in object region A in the reference frame is only attempted to be matched with feature points in the same object region A (or other corresponding regions determined based on preliminary motion estimation) in the frame to be repaired. By restricting the matching to the same object region, erroneous matching between different objects can be eliminated, as different objects typically move independently. This improves the regional consistency of the matched feature points and enhances the accuracy of motion vector calculation based on more reliable matching points.

[0106] This application provides an adaptive video restoration method based on digital video technology. It employs a unique comparison method to select reference frames and frames to be restored, and combines pixel-level difference analysis to accurately locate the area to be restored. Crucially, it constructs a corrected motion vector for the area to be restored using a method based on image feature point motion vectors, object structural relationships, and a prediction model. Then, it performs sub-pixel-level pixel mapping and interpolation reconstruction based on the corrected motion vector, and performs fusion processing on the reconstructed image. This effectively overcomes the problems of detail loss and poor restoration effect in the original area after hard subtitle removal in existing technologies, demonstrating significant technological advancement and practical value. The method of this invention can achieve high-precision, high-fidelity restoration of areas after hard subtitle removal, restoring the structural and texture details of the original video image, improving the spatial consistency between the restored area and the surrounding image, and ensuring the temporal continuity of the video sequence, significantly improving the overall visual quality and viewing experience of the restored video.

[0107] Figure 2 An adaptive video restoration device 20 based on digital video technology, provided in an embodiment of this application, is schematically illustrated. This device is typically integrated into an electronic device with a processor and memory. The device includes: The receiving and preprocessing module 201 is used to preprocess the original video containing hard subtitles and automatically remove the hard subtitles to obtain the video to be repaired; Functional module 202 is used to: compare and filter reference frames and frames to be repaired based on the original video and the video to be repaired; perform pixel grayscale difference analysis on the corresponding frames of the original video and the frames to be repaired to determine the area to be repaired; construct a corrected motion vector for the area to be repaired, the construction of which includes extracting matching feature points of the reference frames and the frames to be repaired, and calculating the reference motion vector of the matching feature points; perform object segmentation to obtain multiple object regions and identify the regions to be analyzed that intersect with the area to be repaired; perform structure prediction on the region to be analyzed to predict the possible coordinates of points within the region to be repaired; perform motion pattern analysis to predict the movement difference of the region to be repaired; and combine the reference motion vector and the movement difference to generate the corrected motion vector; and determine the source position of the subpixel corresponding to the pixels in the area to be repaired based on the corrected motion vector, and perform subpixel interpolation to obtain pixel values ​​to construct a corrected reference image. The fusion generation module 203 is used to fuse the corrected reference image with the area to be repaired to generate a repaired video frame.

[0108] Optionally, the receiving and preprocessing module 201 is further configured to: locate the area where the hard subtitle is located using an image recognition algorithm, and replace or cover the pixels in the area to remove the hard subtitle.

[0109] Optionally, the functional module 202 is further configured to: Hash values ​​are calculated for video frames at the same time point in the original video and the video to be repaired, and it is determined whether they are reference frames by comparing them bit by bit. The hash values ​​are perceptual hash values ​​or other hash types that can reflect the content characteristics of the frame image.

[0110] Optionally, the functional module 202 is further configured to: perform grayscale processing on the frame to be repaired and its corresponding original frame; calculate the difference between the grayscale values ​​of pixels at the same location, and mark the pixel as a pixel to be repaired when the difference exceeds a preset threshold; and identify and determine the region to be repaired based on the pixel to be repaired using a region merging algorithm.

[0111] Optionally, the functional module 202 is further configured to: predict the possible coordinates of points within the area to be repaired using a structural prediction model based on the spatial node relationship of the objects to which each feature point belongs in the reference frame and the position of the corresponding feature point in the frame to be repaired, wherein the possible coordinates are obtained by averaging the prediction results of multiple analysis nodes.

[0112] Optionally, the functional module 202 is further configured to: The motion pattern analysis employs a differential prediction strategy, which uses a differential autoregressive moving average model to predict the motion differences.

[0113] Optionally, the functional module 202 is further configured to: divide the region to be repaired into multiple neighborhoods centered on feature points; map the pixel values ​​of the neighborhoods in the reference frame to the corresponding regions in the frame to be repaired according to the corresponding corrected motion vectors; and construct a corrected reference image corresponding to the region to be repaired.

[0114] Optionally, the fusion generation module 203 is further configured to: apply mean filtering to the corrected reference image to obtain a filtered image; and perform pixel-level fusion of the filtered image with the corresponding region to be repaired in the frame to be repaired.

[0115] Optionally, the receiving and preprocessing module 201 is further configured to: simultaneously perform spatial domain mean filtering and temporal domain mean filtering on the original video frame.

[0116] Optionally, the functional module 202 is further configured to: Edge detection is performed on the reference frame, and broken edges are connected through morphological processing to divide the reference frame into multiple item regions; the matching feature points are matched only within the item regions.

[0117] This application's embodiments, by combining global frame-level hash filtering, feature point motion modeling, and structure-aware spatiotemporal mapping fusion, can accurately locate and reconstruct details of damaged areas after removing hard subtitles, thereby significantly improving the image restoration quality of hard subtitle areas. It achieves high-quality video restoration that is structure-aware and spatiotemporally consistent, possessing significant practical value and promising prospects for wider application.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0119] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the computing processing device according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a non-transient computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0120] For example, Figure 3 A block diagram schematically illustrates a computing processing apparatus for performing a method according to an embodiment of this application. The computing processing apparatus conventionally includes a processor 310 and a computer program product or non-transitory computer-readable medium in the form of a memory 320. The memory 320 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 320 has storage space 330 for program code 331 for performing any of the method steps described above. For example, the storage space 330 for program code may include various program codes 331 respectively for implementing the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact discs (CDs), memory cards, or floppy disks. Such computer program products are typically as shown in the reference. Figure 4 The portable or fixed storage unit is described above. This storage unit may have the same characteristics as... Figure 3 The memory 320 in the computing processing device is arranged in similar storage segments, storage spaces, etc. Program code can be compressed, for example, in an appropriate form. Typically, the storage unit includes computer-readable code 431, that is, code that can be read by a processor such as 310, which, when run by the computing processing device, causes the computing processing device to perform the various steps in the methods described above.

[0121] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0122] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this application. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0123] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An adaptive video restoration method based on digital video technology, characterized in that, The method includes: The original video containing hard-coded subtitles is preprocessed and the hard-coded subtitles are automatically removed to obtain the video to be repaired; Based on the original video and the video to be repaired, reference frames and frames to be repaired are compared and selected. Pixel grayscale difference analysis is performed between the frame to be repaired and the corresponding frame of the original video to determine the area to be repaired; The process involves constructing a corrected motion vector for the region to be repaired. This construction includes extracting matching feature points between the reference frame and the frame to be repaired, calculating a reference motion vector for the matching feature points, performing object segmentation on the reference frame to obtain multiple object regions, and identifying regions to be analyzed that intersect with the region to be repaired, performing structure prediction on the region to be analyzed to predict the possible coordinates of points within the region to be repaired, performing motion pattern analysis on the region to be repaired to predict the movement differences in the region to be repaired, and combining the reference motion vector and the movement differences to generate the corrected motion vector. Based on the corrected motion vector, the source position corresponding to the sub-pixel of the pixel in the region to be repaired is determined, and sub-pixel interpolation is performed to obtain the pixel value in order to construct a corrected reference image; The corrected reference image is fused with the region to be repaired to generate a repaired video frame.

2. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The automatic removal of hard captions includes: locating the area where the hard captions are located using an image recognition algorithm, and replacing or covering the pixels in the area to remove the hard captions.

3. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The comparison includes: Hash values ​​are calculated for video frames at the same time point in the original video and the video to be repaired, and it is determined whether they are reference frames by comparing them bit by bit. The hash values ​​are perceptual hash values ​​or other hash types that can reflect the content characteristics of the frame image.

4. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The step of determining the region to be repaired by performing pixel grayscale difference analysis between the frame to be repaired and the corresponding frame of the original video includes: converting the frame to be repaired and its corresponding original frame to grayscale; calculating the difference between the grayscale values ​​of pixels at the same location, and marking the pixel as a pixel to be repaired when the difference exceeds a preset threshold; and identifying and determining the region to be repaired based on the pixel to be repaired using a region merging algorithm.

5. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The step of performing structural prediction on the region to be analyzed to predict the possible coordinates of points within the region to be repaired includes: Based on the spatial node relationship of the object to which each feature point belongs in the reference frame and the position of the corresponding feature point in the frame to be repaired, a structural prediction model is used to predict the possible coordinates of points in the area to be repaired. The possible coordinates are obtained by averaging the prediction results of multiple analysis nodes.

6. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The step of performing motion pattern analysis to predict the movement differences in the area to be repaired includes: The motion pattern analysis employs a differential prediction strategy, which uses a differential autoregressive moving average model to predict the motion differences.

7. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The construction of the correction reference image includes: dividing the region to be repaired into multiple neighborhoods centered on feature points; mapping the pixel values ​​of the neighborhoods in the reference frame to the corresponding regions in the frame to be repaired according to the corresponding correction motion vectors; and constructing a correction reference image corresponding to the region to be repaired.

8. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The step of fusing the corrected reference image with the region to be repaired includes: applying mean filtering to the corrected reference image to obtain a filtered image; and performing pixel-level fusion of the filtered image with the corresponding region to be repaired in the frame to be repaired.

9. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The preprocessing of the original video containing hard subtitles includes: simultaneously performing spatial domain mean filtering and temporal domain mean filtering on the original video frames.

10. The adaptive video restoration method based on digital video technology according to claim 1, characterized in that, The process of segmenting objects to obtain multiple item regions includes: Edge detection is performed on the reference frame, and broken edges are connected through morphological processing to divide the reference frame into multiple item regions; the matching feature points are matched only within the item regions.

Images