Adaptive video inpainting method based on digital video technology

Abstract

The application discloses a kind of based on adaptive video repair method of digital video technology, it is related to image communication technical field.The method includes: the original video containing hard subtitle is preprocessed and automatically removes hard subtitle, obtains to be repaired video;By comparing original video with to be repaired video screening reference frame and to be repaired frame;According to pixel gray difference determination to be repaired area;Extract matching feature point and calculate reference motion vector, combine object structure segmentation and area prediction, estimate the possible position and motion difference of point in to be repaired area, generate correction motion vector;Sub-pixel interpolation is carried out using the vector, and the corrected reference image is constructed;Finally, fusion correction image and to be repaired area, generate the video frame after repair.The method can effectively improve the restoration quality of the area blocked by subtitle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image communication, in particular to an adaptive video repair method based on digital video technology. BACKGROUND

[0002] With the wide application of digital video technology, in order to ensure the compatibility and stability of hard subtitles on different platforms and terminals, hard subtitles are often used in scenes such as film and television works, educational videos, and conference recordings. However, in the process of video re-creation and secondary editing, removing hard subtitles often produces holes in the original hard subtitle area, seriously damaging the details of the area.

[0003] The existing hard subtitle removal video repair method usually relies on spatial interpolation of surrounding pixels or optical flow hole filling technology, but since the hard subtitle area pixels are covered in a large range, the available pixel points for reference are limited, resulting in a decrease in the repaired picture quality, which is manifested as edge detail blur, color distortion, and increased noise. SUMMARY

[0004] Therefore, the embodiments of the present application provide an adaptive video repair method based on digital video technology, which at least partially solves the problems in the prior art.

[0005] The embodiments of the present application provide an adaptive video repair method based on digital video technology, which comprises:

[0006] Pretreating an original video containing hard subtitles and automatically removing the hard subtitles to obtain a video to be repaired;

[0007] Based on the original video and the video to be repaired, reference frames and frames to be repaired are compared and screened;

[0008] According to the frames to be repaired and the corresponding frames of the original video, pixel gray value difference analysis is performed to determine the repair area;

[0009] The repair motion vector of the repair area is constructed, which includes extracting matching feature points of the reference frames and the frames to be repaired, calculating the reference motion vector of the matching feature points, performing object segmentation to obtain a plurality of object regions and identifying an analysis area intersecting with the repair area, performing structure prediction on the analysis area to predict the possible coordinates of the points in the repair area, performing motion mode analysis to predict the movement difference of the repair area, and combining the reference motion vector and the movement difference to generate the repair motion vector;

[0010] According to the repair motion vector, the source position corresponding to the pixel sub-pixel of the repair area is determined, and sub-pixel interpolation is performed to obtain the pixel value to construct a repair reference image;

[0011] fusing the modified reference image and the to-be-repaired region to generate a repaired video frame.

[0012] Optionally, the automatic removal of hard subtitles comprises: locating the region where the hard subtitles are located by an image recognition algorithm, and replacing or covering the pixels in the region to remove the hard subtitles.

[0013] Optionally, the comparison comprises:

[0014] calculating hash values of video frames at the same time point in the original video and the to-be-repaired video respectively, and judging whether the video frames are reference frames through bit-by-bit accurate comparison, wherein the hash values are perceptual hash values or other hash types capable of reflecting the content features of the frames.

[0015] Optionally, the pixel gray value difference analysis of the to-be-repaired frame and the corresponding frame of the original video to determine the to-be-repaired region comprises: performing gray processing on the to-be-repaired frame and the corresponding frame of the original video; calculating the difference value of the gray values of pixels at the same position, and marking the pixel as a to-be-repaired pixel point when the difference value exceeds a preset threshold; identifying and determining the to-be-repaired region based on the to-be-repaired pixel point through a region merging algorithm.

[0016] Optionally, the structure prediction of the to-be-analyzed region to predict the possible coordinates of the points in the to-be-repaired region comprises: predicting the possible coordinates of the points in the to-be-repaired region based on the spatial node relationship of the objects to which the feature points in the reference frame belong and the positions of the corresponding feature points in the to-be-repaired frame, using a structure prediction model, wherein the possible coordinates are obtained by averaging the prediction results of multiple analysis nodes.

[0017] Optionally, the motion pattern analysis to predict the movement difference situation of the to-be-repaired region comprises:

[0018] The motion pattern analysis adopts a difference prediction strategy, and the difference prediction strategy adopts a difference autoregressive moving average model to predict the movement difference situation.

[0019] Optionally, the construction of the modified reference image comprises: dividing the to-be-repaired region into a plurality of neighborhoods centered on feature points; mapping the pixel values of the neighborhoods in the reference frame to the corresponding regions in the to-be-repaired frame according to the corresponding modified motion vectors; and constructing a modified reference image corresponding to the to-be-repaired region.

[0020] Optionally, the fusing of the modified reference image and the to-be-repaired region comprises: applying mean filtering processing to the modified reference image to obtain a filtered image; and performing pixel-level fusion of the filtered image and the corresponding to-be-repaired region in the to-be-repaired frame.

[0021] Optionally, the pre-processing of the original video containing hard subtitles comprises: simultaneously performing spatial domain mean filtering and time domain mean filtering on the original video frames.

[0022] Optionally, the object segmentation to obtain a plurality of object regions comprises:

[0023] performing edge detection on the reference frame and connecting broken edges through morphological processing to divide the reference frame into a plurality of object regions; and the matching of the feature points is only performed within the object regions.

[0024] The embodiments of the present application only relate to a self-adaptive video repair method based on digital video technology, which can be executed by a general electronic device with a processor and a memory, or can be deployed in a server, a workstation or a terminal device as a computer program product. When the processor of the device executes the program code stored in the memory, the steps of the present application can be completed.

[0025] The self-adaptive video repair scheme based on digital video technology provided by the embodiments of the present application effectively solves the problems of detail loss in the original region after hard subtitle removal and poor repair effect in the prior art, and has significant beneficial effects.

[0026] Specifically, the method can efficiently obtain clean data and reliable reference sources required for repair by pre-processing and automatically removing hard subtitles from the original video, and based on comparison and screening of reference frames and frames to be repaired. The frame to be repaired is accurately positioned by pixel-level difference analysis. In particular, the method can construct a corrected motion vector of the frame to be repaired, so that even in the case where the region is damaged and the feature points cannot be directly obtained, the motion of the region can be accurately estimated based on the extracted image feature points, the calculated reference motion vector, and the object structure analysis and prediction model. This accurate motion estimation enables subsequent determination of sub-pixel level pixel corresponding source positions and sub-pixel interpolation according to the corrected motion vector, thereby reconstructing the image content of the frame to be repaired with high precision. Finally, the reconstructed image is fused with the original region, and filtering processing is applied to reduce the discontinuity of the boundary, thereby realizing natural connection between the repaired region and the surrounding picture. Through the synergistic effect of the above key stages, the method of the present application overcomes the limitations of traditional methods, realizes high-quality and high-fidelity repair of the region after hard subtitle removal, and significantly improves the video picture effect.

[0027] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the specific embodiments of the present application can be implemented in accordance with the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0029] Figure 1 A flowchart of a self-adaptive video repairing method based on digital video technology provided by an embodiment of the present application is schematically shown;

[0030] Figure 2 A structural diagram of a self-adaptive video repairing device based on digital video technology provided by an embodiment of the present application is schematically shown;

[0031] Figure 3 A block diagram of a computing processing device for executing the method according to the embodiments of the present application is schematically shown;

[0032] Figure 4 A storage unit for holding or carrying program codes for implementing the method according to the embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0033] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0034] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0035] It should be noted that the drawings provided in the following embodiments are only schematic and are not drawn to scale. They are provided merely to illustrate the basic understanding of the present disclosure. In actual implementation, components shown in the drawings can have a different shape, number, and size, and the layout of the components can be more complex. The drawings provided in the following embodiments only schematically illustrate the basic concept of the present disclosure, and only show the components related to the present disclosure, not the number, shape, and size of the components in actual implementation. In actual implementation, the shape, number, and size of the components can be arbitrarily changed, and the layout of the components can be more complex.

[0036] Also in the following description, specific details are provided to thoroughly understand examples. However, one of ordinary skill in the art will appreciate that the described aspects can be practiced without these specific details.

[0037] The embodiments of the present disclosure provide a video restoration method. The video restoration method provided by the embodiments can be executed by a computing device, which can be implemented as software or a combination of software and hardware, and can be integrated into a server, a terminal device, or the like.

[0038] Referring to Figure 1 , a flowchart of a self-adaptive video restoration method based on digital video technology is shown schematically, which includes the following steps:

[0039] In step 101, the original video containing hard subtitles is preprocessed and the hard subtitles are automatically removed to obtain a video to be restored.

[0040] It should be noted that the original video refers to source video data to be processed for restoration, which usually contains hard subtitles. The original video can be obtained in various ways, usually by collecting, digitizing, and storing video sources in analog or digital form into digital video files through various video collection devices. The video collection devices include, but are not limited to, digital video cameras, video capture cards, and other devices capable of digitizing video signals. The collection process converts video signals into a series of digital video frames and stores them as original video files.

[0041] In the embodiments of the present disclosure, after obtaining the original video file, preliminary image preprocessing of the original video file is necessary. The preprocessing aims to improve the image quality of the video and suppress noise that may affect subsequent processing. Specifically, the preprocessing mainly focuses on applying noise reduction algorithms including but not limited to filtering algorithms in the spatial and / or temporal domain of the video data to effectively reduce the high-frequency noise commonly present in video frames. The noise reduction algorithm can achieve image smoothing and noise suppression by processing the pixel values in the video frame and its spatial and temporal neighborhood, obtaining the original video after preprocessing.

[0042] It should be noted that the hard-coded subtitles refer to fixed text or graphic information directly encoded and burned into the pixel data of each frame of the video, becoming part of the image itself. Unlike soft subtitles that can be selectively turned off by a player, the hard subtitles are fixed and cannot be removed by player functions, and their existence directly covers and destroys the original image content below. Such hard subtitles are commonly found in various video sources including but not limited to historical images, educational videos, conference recordings, and the like.

[0043] In the embodiments of the present application, an artificial intelligence (AI) based algorithm is adopted to automatically analyze and process the pre-processed original video to detect, locate and remove the hard subtitles. The removal operation can include but is not limited to reconstructing the background area covered by the subtitles by using image inpainting techniques, or removing the subtitle information from the picture by using other image processing techniques. After the automatic hard subtitle removal processing, a video file without the hard subtitles is obtained, denoted as a to-be-repaired video. The to-be-repaired video is the basis for the subsequent video repair process.

[0044] In step 102, based on the original video and the to-be-repaired video, reference frames and to-be-repaired frames are compared and screened.

[0045] It should be noted that the to-be-repaired video is obtained based on the pre-processed original video only by automatic hard subtitle removal processing. Therefore, for the video frames that do not originally contain hard subtitles, the image content should remain highly consistent between the corresponding frames in the pre-processed original video and the to-be-repaired video; and for the video frames that originally contain hard subtitles and have been removed, since the hard subtitle area is modified or removed, pixel differences will be generated between the corresponding frames of the two videos. This step is to use such pixel differences to identify the frames affected by hard subtitles.

[0046] In the embodiments of the present application, based on the above principle, preferably, the video frames at the same time (for example, having the same frame number or time stamp) in the pre-processed original video and the to-be-repaired video are compared one by one at the image level. Specifically, for each pair of video frames at the same time, the hash values of the image contents thereof are calculated respectively. The hash value is a compact digital fingerprint that can represent the unique content features of an image. The hash algorithm can include but is not limited to various hash algorithms suitable for image content comparison. Then, the two hash values calculated are compared, for example, by using bit-by-bit comparison to determine whether the two hash values are exactly the same.

[0047] According to the comparison result of the hash values, the video frames in the video to be repaired are classified and screened. If the hash value of a frame in the original video is exactly the same as the hash value of the corresponding frame at the same time in the video to be repaired, it is determined that the frame in the video to be repaired has not been changed in the process of removing the hard subtitles. This indicates that the frame originally does not contain hard subtitles, and its image content is complete and undamaged. At this time, the corresponding frame in the video to be repaired is marked as a reference frame that does not need to be repaired. The reference frame can be used as a source of image information for repairing the to-be-repaired frame in subsequent steps. On the contrary, if there is any difference between the two hash values, it is determined that the frame in the video to be repaired has been changed in the process of removing the hard subtitles because the hard subtitles have been modified or removed. This indicates that the frame originally contains hard subtitles and has been processed for removal. At this time, the corresponding frame in the video to be repaired is marked as a to-be-repaired frame that needs subsequent repair. The corresponding frame in the original video can be regarded as a damaged frame containing original hard subtitles, and its content can be used to assist in analyzing the original image information of the damaged area.

[0048] Through the above comparison process, this step automatically completes the preliminary classification of all video frames in the video to be repaired, and divides them into the reference frame set and the to-be-repaired frame set. The reference frame set provides a large amount of image information that is not damaged or affected by hard subtitles, and the to-be-repaired frame set clearly indicates specific video frames that need subsequent repair processing. This classification result provides a clear data basis and processing target for accurate damage area positioning and efficient image repair process in subsequent steps.

[0049] Step 103, according to the pixel gray difference analysis of the to-be-repaired frame and the corresponding frame in the original video, to determine the to-be-repaired area.

[0050] It should be noted that the to-be-repaired frame is a video frame that has been automatically removed from the hard subtitles in step 101, and has been determined to have changed content compared to the original frame in step 102. The method of automatically removing hard subtitles usually determines the screen area where the hard subtitles are located, and then performs pixel-level processing on the area to remove the subtitle information. In this process, the pixels in the non-subtitle area usually remain unchanged. Therefore, by comparing the to-be-repaired frame with the damaged frame (i.e. the frame containing original hard subtitles in the original video) at the same time after the pre-processing of the original video, the area where the pixel value has changed only due to the hard subtitle removal operation can be accurately isolated, thereby locating the to-be-repaired area.

[0051] In this embodiment, specifically, for each frame to be repaired, a pixel-level difference analysis is performed on the corresponding damaged frame at the same time in the preprocessed original video to identify changed pixels. First, the image data of the frame to be repaired and the image data of the damaged frame are converted to grayscale to obtain corresponding grayscale images. Then, each pixel position in the grayscale image is traversed, and the grayscale value of the pixel at that position in the frame to be repaired is compared with the grayscale value of the pixel at the same position in the damaged frame. If the absolute value of the difference between the grayscale values ​​of the two corresponding pixels is less than or equal to a preset grayscale difference threshold, it is determined that the pixel at that position has not changed significantly; conversely, if the absolute value of the difference between the grayscale values ​​is greater than the grayscale difference threshold, it is determined that the pixel at that position belongs to a pixel affected or changed by hard caption removal, and it is marked as a pixel to be repaired. The grayscale difference threshold is a preset parameter used to set the identification of pixel differences.

[0052] After identifying all the pixels to be repaired in the frame image to be repaired, these discretely distributed pixels need to be clustered or combined into one or more connected regions, i.e., preliminary damaged regions. (The preliminary damaged region refers to an unmerged cluster of pixels identified from the pixels to be repaired using preliminary local spatial criteria). The identification criteria for the preliminary damaged region can be based on criteria including, but not limited to, the spatial connectivity or local density of the pixels to be repaired. The preliminary damaged region can be generated using various spatial relationship-based methods, including but not limited to clustering or region growing. After generating all the preliminary damaged regions, a region merging process is required to obtain the final set of regions to be repaired. The merging process aims to merge spatially related, adjacent, or overlapping preliminary damaged regions into one or more larger, logically continuous regions to be repaired. The merging rules include, but are not limited to: if two preliminary damaged regions share common pixels or are adjacent to each other in space, they are merged into a unified region to be repaired. By iteratively applying the merging rules until no more regions meeting the conditions can be merged, the precisely defined set of regions to be repaired is finally obtained. The set of regions to be repaired is the target range for pixel reconstruction and texture synthesis using reference image information in subsequent steps.

[0053] Through the aforementioned processes of pixel difference detection, pixel marking to be repaired, region clustering, and region merging, this step accurately locates the specific spatial range within the frame to be repaired that requires subsequent image restoration, i.e., the region to be repaired. The precise definition of this region provides a foundation and guidance for subsequent efficient and accurate pixel reconstruction and texture synthesis of the damaged area using reference image information.

[0054] Step 104: Construct the corrected motion vector for the region to be repaired. The construction includes extracting matching feature points between the reference frame and the frame to be repaired, calculating the reference motion vector of the matching feature points; performing object segmentation to obtain multiple object regions and identifying regions to be analyzed that intersect with the region to be repaired; performing structure prediction on the region to be analyzed to predict the possible coordinates of points within the region to be repaired; performing motion pattern analysis to predict the movement differences of the region to be repaired; and combining the reference motion vector and the movement differences to generate the corrected motion vector.

[0055] It should be noted that within a single frame of a video, there is always a foreground and a background, and some frames may even contain multiple layers of scenery. The shapes of these objects are fixed and do not change, meaning that the same object has the same shape and structure at different times. Therefore, the positional information of the corresponding object in the image is determined based on the intersection points and feature points of numerous structures in the image. In a specific implementation, an algorithm processes the reference frame and the frame to be repaired to obtain the feature corner points in the two frame images.

[0056] In one specific implementation, the feature point extraction method may include, but is not limited to, the following process: first, convert the frame image into an integral image; then, generate multiple images at different scales using box filters of different sizes (e.g., preferably 9×9 and 7×7); subsequently, for each scale image, calculate the matrix determinant value or similar response value of all pixels to generate a response map; then, perform non-maximum suppression on the numerous response values, and may introduce methods including, but not limited to, two-dimensional entropy filtering mechanisms, to finally filter and obtain stable feature points. The threshold for filtering stable feature points can be adaptively obtained through methods including, but not limited to, calculating the global contrast of the image (e.g., the average gradient magnitude). The obtained feature points are recorded as contrast nodes.

[0057] It is to be noted that in a video, various items in a picture are constantly changing and moving with time, i.e. various objects in the video are in different positions in different video frames, but the movement of the various objects does not change the shape of the objects themselves, which makes the feature points of the same object in two different video frames correspond to each other, so the movement of each part of the object is described according to the position difference of the two corresponding feature points in the video frame image. Accordingly, the reference motion vector of each node is constructed. In a specific embodiment, the two frame images are first matched by a method including but not limited to brute force matching, which can be combined with a method including but not limited to mesh partitioning to accelerate the matching speed by spatial constraint. For example, the distance (e.g. Euclidean distance) between the corresponding descriptors of the contrast nodes in the mesh partitioned in the two frame images (e.g. the reference frame and the frame to be repaired) is calculated, and the descriptors are obtained by converting the contrast nodes into high-dimensional floating-point vectors based on a method including but not limited to gradient or wavelet response. The corresponding relationship between the contrast nodes, i.e. the matching feature points, can be obtained by finding the minimum sum of the distances between the descriptors.

[0058] In a specific embodiment, after obtaining the set of contrast nodes of the reference frame and the frame to be repaired, matching between the contrast nodes is performed to establish the corresponding point relationship between the two frames. The feature point matching method aims to find the best corresponding point of each contrast node in the reference frame in the frame to be repaired, and the method can include but is not limited to a matching method based on feature descriptors. By comparing the similarity of the descriptors of the contrast nodes, and in combination with a method including but not limited to spatial constraint, the corresponding relationship between the contrast nodes in the reference frame and their corresponding matching nodes in the frame to be repaired is determined. Thus, a plurality of matching pairs are obtained.

[0059] Based on the above-obtained corresponding relationship (i.e. matching pairs) between the contrast nodes, the reference motion vector of the matching feature points is calculated. Each matching pair connects the positions of the same feature point in the reference frame and the frame to be repaired, and the spatial displacement between these matching points estimates the movement of the region represented by the feature point between frames.

[0060] For example, in a preferred embodiment, the reference motion vector corresponding to each matching pair is obtained by calculating the coordinate difference of the matching points in a two-dimensional coordinate system . The coordinate system is preset, for example, with the geometric center of the frame image as the coordinate origin, with the positive right direction as the positive direction of the X axis and the positive up direction as the positive direction of the Y axis. The reference motion vector can be calculated by the following formula:

[0061] (1)

[0062] wherein, Indicates the comparison node in the reference frame exist Coordinate values ​​on the axis; Indicates the comparison node in the reference frame exist Coordinate values ​​on the axis; Indicates the node in the frame to be repaired compared with the node in the comparison. Matching nodes exist Coordinate values ​​on the axis; Indicates the node in the frame to be repaired compared with the node in the comparison. Matching nodes exist The coordinate values ​​on the axis.

[0063] The above-described coordinate system yields the reference motion vector, which represents the motion of the same location on the object from the time corresponding to the reference frame to the time corresponding to the frame to be repaired. These calculated reference motion vectors reflect the motion of their respective feature points between frames.

[0064] Furthermore, the purpose of object segmentation to obtain multiple object regions and identifying regions to be analyzed that intersect with the region to be repaired is to provide structural context and data sources for subsequent construction of corrected motion vectors. In a frame image, different objects form contacting or non-contact edges. These edges allow for the segmentation of numerous objects in the frame image, and each object has its own key feature nodes. Based on this, numerous contrasting nodes are segmented: for example, first through... The operator performs edge detection on the frame image, detecting the edges of each object in the frame image; then, morphological processing (such as closing operations) is used to connect partially broken edges; subsequently, a line scan algorithm is used to fill the interior of closed edges, dividing the frame image into different object regions. Finally, numerous comparison nodes within the same object region are recorded as comparison nodes of the same object. Furthermore, not all regions in the video need repair, so all object regions need to be compared. If an object region intersects with the region to be repaired, it is recorded as the region to be analyzed, and its internal comparison nodes are recorded as nodes to be analyzed; otherwise, the corresponding object region does not need to be analyzed.

[0065] To construct the corrected motion vector more accurately, particularly for the region to be repaired, the structural information of the image content itself can be used to constrain and refine the motion vector. In one specific approach, the frame to be repaired or the reference frame can undergo object or region segmentation to extract object structural information, for example, through edge detection and region filling.

[0066] Furthermore, regarding structural prediction of the region to be analyzed to predict the possible coordinates of points within the region to be repaired, this process utilizes the structural features of an object for prediction. Due to image blurring or missing data, the corresponding feature points in the region to be repaired are often difficult to detect accurately. However, undamaged and damaged areas on the same object form a whole, and the positional relationships between their feature points are approximately consistent. Therefore, more relative positions can be referenced to determine the possible coordinates of points in the damaged area. It should be noted that the "possible coordinates" mentioned here refer to the predicted position coordinates of missing or unstable points in the region to be repaired, estimated through spatial structure mapping. These coordinates are not actually detected image points, but rather inferred values ​​calculated based on the structural relationships of multiple known feature points, used for subsequent construction of motion difference sequences and correction vector prediction. These possible coordinates provide a method for approximately reconstructing the spatial evolution path of feature points across time frames within regions where they cannot be extracted, and are a core intermediate variable for correction vector generation throughout the repair process.

[0067] In a preferred embodiment, a method based on object structural features preliminarily predicts the possible coordinates of points within the region to be repaired. The prediction method utilizes the positional relationship between the comparison nodes in the reference frame and their corresponding matching nodes in the frame to be repaired, combined with object structural features, to make the inference. The calculation process is as follows:

[0068] (2)

[0069] In the formula, Indicates the comparison node in the reference frame The coordinates of the location; Indicates the comparison node in the reference frame The coordinates of the location; Indicates the node in the frame to be repaired compared with the node in the comparison. Matching nodes Location coordinates; Indicates the node in the frame to be repaired compared with the node in the comparison. The total number of nodes to be analyzed within the same region to be analyzed.

[0070] Simply put, its function is to utilize known structural relationships; it assumes that on the same object, points... and points Relative positional relationship in the reference frame , and point and points Relative positional relationships in the frame to be repaired - They are similar. Because the points are... The location of the to-be-repaired region (image blur or missing), we do not know its exact location in the to-be-repaired frame. But we know the points of other undamaged areas on the same object In the to-be-repaired frame ( ), and the point and the point In the reference frame ( and ). For each known point and its corresponding point in the reference frame, the formula attempts to estimate the location of point in the to-be-repaired frame based on structural similarity. The first of the formula represents the sum of all correlated known points and take the average. This means that it does not rely on only one known point , but rather takes into account the information provided by multiple known points , taking the average of their predicted results to obtain a more stable and robust point Possible coordinates of point in the to-be-repaired frame.

[0071] The above method preliminarily determines the possible coordinates of the potential feature points in the to-be-repaired region based on the structural characteristics of the object itself and the known motion information.

[0072] It should be noted that the motion pattern analysis is performed to predict the movement difference of the to-be-repaired region, and the reference motion vector and the movement difference are combined to generate the corrected motion vector. This process aims to predict the movement difference of the to-be-repaired region and ultimately generate the corrected motion vector. Different positions exhibit different situations in the video, such as some positions close to the video lens have smaller movement amplitude, but positions far from the lens appear more obvious in movement, and the movement difference (i.e., the reference motion vector) between two video frames also presents different characteristics. Considering that both to-be-analyzed nodes are located on the same object, and the movement of the object is determined, the movement of numerous nodes between the two nodes is also the same, and from a close-up to a far shot, the movement scale becomes larger as it moves farther away. Accordingly, the possible motion reference vector of each to-be-repaired region point is calculated.

[0073] In a preferred embodiment, a prediction method based on motion pattern analysis is adopted. The motion in the to-be-repaired region can be predicted according to the motion law corresponding to the undamaged region in the to-be-analyzed region. This can include constructing a motion vector sequence and a motion difference sequence (for example, first arranging the motion reference vectors corresponding to the contrast nodes in the to-be-analyzed region according to the order of the contrast nodes to obtain a motion vector sequence corresponding to the to-be-analyzed region, wherein the node farthest from the to-be-repaired region node is arranged at the front of the sequence; then differentiating the motion vector sequence to obtain a motion difference sequence, that is, the difference of the movement between the contrast nodes in the same object in the two video frames), and using a prediction model for prediction. The prediction model can use a time series prediction model including but not limited to a difference autoregressive moving average (DARMA) model. According to the prediction model, the movement difference of each position point in the to-be-repaired region relative to its last adjacent node is predicted.

[0074] Finally, the corrected motion vector is generated according to the movement difference obtained by the above motion pattern analysis and the original motion vector. The corrected motion vector is usually obtained according to the movement difference and the reference motion vector corresponding to the nearest to-be-analyzed node of the to-be-repaired region node, and the calculation process is as follows:

[0075] (3)

[0076] In the formula, represents the reference motion vector corresponding to the nearest to-be-analyzed node of the to-be-repaired region node ; represents the movement difference of the to-be-repaired region node relative to the to-be-analyzed node .

[0077] At this point, by analyzing the to-be-repaired frame and the object region where it is located, combining the reference motion vector, object structure information, and motion pattern prediction, the corrected motion vector of each position in the to-be-repaired region is obtained. The corrected motion vector set constitutes motion guidance information describing how the pixels in the to-be-repaired region should be reconstructed according to the motion information from the reference frame.

[0078] Step 105, according to the corrected motion vector, determine the source position of the to-be-repaired region pixel sub-pixel, and perform sub-pixel interpolation to obtain the pixel value to construct the corrected reference image.

[0079] ​It should be noted that the corrected motion vector indicates the displacement of pixels in the region to be repaired between the frame to be repaired and the reference frame. Based on the corrected motion vector, the corresponding source position of each pixel in the region to be repaired in the reference frame can be determined.

[0080] In a preferred embodiment, constructing the corrected reference image includes the following process. First, the process of dividing the region to be repaired into multiple neighborhoods centered on feature points can be specifically implemented as follows: based on the positions of the nodes to be analyzed identified in step 104 within the region to be repaired and their corresponding corrected motion vectors. In a specific implementation, the division can be achieved, for example, by a clustering method, where the cluster center can be set as a comparison node associated with the region to be repaired, thereby dividing the region to be repaired into numerous small regions centered on the comparison nodes, denoted as the neighborhoods of each comparison node. The number of neighborhoods can be, for example, preset to the total number of comparison nodes associated with the region to be repaired.

[0081] Secondly, for each pixel within the region to be repaired, the correspondence between the pixel and the reference frame is determined based on its neighborhood and the corresponding corrected motion vector, thereby determining its corresponding source position in the reference frame. This correspondence defines the spatial mapping between the pixel position within the region to be repaired and the source position in the reference frame, and this mapping is determined by the corrected motion vector. In a specific implementation, the correspondence between pixels in the frame to be repaired and pixels in the reference frame can be expressed as:

[0082] (4)

[0083] In the formula, Indicates the comparison node in the reference frame Within the neighborhood of The coordinates of the pixel; Indicates the comparison node The corresponding corrected motion vector; Indicates the node in the frame to be repaired compared with the node in the comparison. Matching nodes Within the neighborhood of The coordinates of the pixel.

[0084] The formula describes the spatial mapping relationship between pixel positions in the reference frame and corresponding pixel positions in the frame to be repaired. After determining the corresponding source position in the reference frame, the pixel value at that position is obtained from the reference frame.

[0085] It should be noted that the source position is usually a non-integer coordinate. Therefore, in a preferred embodiment, the process of obtaining the pixel value needs to be sub-pixel interpolation. The sub-pixel interpolation method can include but is not limited to bilinear interpolation, bicubic interpolation, or weighted average based on surrounding integer pixels, etc. According to the pixel value of the integer pixel around the source position coordinate, a more accurate pixel value is calculated. For example, sub-pixel interpolation: an image is composed of discrete pixel points, each pixel point is located at an integer coordinate position. However, the corresponding source position of the pixel in the reference frame according to the modified motion vector (i.e. sub-pixel corresponding source position) is usually a non-integer coordinate. Since the pixel value of the non-integer position cannot be directly obtained, interpolation technology is needed. Sub-pixel interpolation refers to estimating and obtaining the pixel value of the non-integer position by a specific algorithm according to the pixel value of the integer pixel around the non-integer coordinate (for example, (100, 200), (101, 200), (100, 201), (101, 201)). In step 105, sub-pixel interpolation is performed to obtain the pixel value from the reference frame at the exact position pointed by the motion vector, in order to ensure the accuracy and quality of the reconstructed image.

[0086] In another embodiment, the obtained pixel value is directly obtained and mapped from the reference frame, and the gray value (brightness, etc.) of the pixel is kept unchanged.

[0087] Finally, by repeating the above pixel mapping and pixel obtaining process, all pixel positions in the to-be-repaired region (or processing in the neighborhood of the contrast node) are traversed, the corresponding pixel value is obtained from the reference frame according to the modified motion vector, and filled into the corresponding pixel position in the modified reference image. Thus, a modified reference image with the same spatial range and pixel arrangement as the to-be-repaired region is constructed.

[0088] Through the above process, a modified reference image with the same spatial range and pixel arrangement as the to-be-repaired region is constructed. The modified reference image contains image information accurately mapped from the reference frame according to the principle of motion compensation, aiming to provide accurate and lossless reference image content for subsequent filling of the to-be-repaired region.

[0089] Step 106, fuse the modified reference image with the to-be-repaired region to generate a repaired video frame.

[0090] It is noted that the modified reference image is directly mapped or reconstructed from the pixels in the reference frame by motion compensation, and generally retains its original pixel value characteristics (e.g. gray value or brightness). However, due to the difference in video shooting environment, lens parameters or time point, there can be overall or local light-dark (brightness) difference between the reference frame and the frame to be repaired. Such difference can cause obvious visual discontinuity at the region boundary when the modified reference image is directly replaced into the region to be repaired.

[0091] To solve the above light-dark difference problem and ensure the smoothness and visual consistency of fusion, in one embodiment, when the modified reference image is fused with the region to be repaired, image processing methods including but not limited to filtering can be applied to adjust the pixel values of the region to be repaired or the pixel values of the modified reference image. The adjustment aims to make the pixels at the boundary of the modified reference image and the region to be repaired more matched in light-dark characteristics. For example, in a specific implementation, the filtering method can apply mean filtering processing.

[0092] In the embodiments of the present application, the fusion processing is to replace or combine the pixel information of the adjusted modified reference image into the corresponding region to be repaired in the frame to be repaired, so as to fill the region originally covered by the hard subtitle with effective image information from the reference frame.

[0093] Through the fusion processing, a repaired video frame is generated, in which the region to be repaired has been filled with pixel information from the reference frame, which has been motion compensated and possibly adjusted in light-dark. At this point, the repaired video frame is obtained.

[0094] Optionally, the step 101 comprises: locating the area where the hard subtitle is located by an image recognition algorithm, and replacing or covering the pixels in the area to remove the hard subtitle. It should be noted that when automatic hard subtitle removal is performed in a manner including but not limited to analyzing the original video frame after preprocessing by an image recognition algorithm to locate the hard subtitle area and processing the pixels in the located area, the specific implementation of the processing can be various. For example, an artificial intelligence (AI) based algorithm model can be used to realize automatic detection and accurate positioning of the hard subtitle area. The removal operation includes processing the pixels in the located hard subtitle area. The pixel processing manner can include but is not limited to: reconstructing the background image covered by the subtitle by using an image inpainting technology, filling the subtitle area with reconstructed or predicted background pixels by pixel replacement; or separating the hard subtitle from the video frame by using image segmentation or foreground extraction technology, and then extracting the background image as the result after removing the subtitle; or using other pixel replacement, covering or processing technology to achieve the purpose of removing the hard subtitle. After automatic hard subtitle removal is realized by the above manner, a video file in which the hard subtitle is removed is obtained, which is denoted as a to-be-repaired video. The to-be-repaired video is the basis for the subsequent video repair process.

[0095] Optionally, the step 102 comprises:

[0096] Hash values of video frames at the same time point in the original video and the to-be-repaired video are calculated respectively, and whether it is a reference frame is judged by bit-by-bit accurate comparison, wherein the hash value is a perceptual hash value or other hash type capable of reflecting the content features of the frame image.

[0097] It should be noted that when the reference frame and the frame to be repaired are screened by comparison, the principle is based on the fact that the video to be repaired only changes in pixel content in some frames due to the removal of hard subtitles. Specifically, the original video and the video frames at the same time (e.g., with the same frame number or timestamp) in the video to be repaired are compared one by one. For each pair of corresponding frames, the hash value of the image content is calculated respectively. The hash value is a kind of compact digital fingerprint that can represent the unique content features of the image. The hash algorithm can include but is not limited to perceptual hash, difference hash and other hash algorithms suitable for image content comparison. Then, the two hash values calculated are compared. The comparison is carried out by bit-by-bit accurate comparison, to determine whether the two hash values are exactly the same. According to the result of the bit-by-bit accurate comparison, the video frames in the video to be repaired are classified. If the two hash values are exactly the same, it is determined that the frame in the video to be repaired has not changed due to the removal of hard subtitles, and the original does not contain hard subtitles. The frame is marked as a reference frame that does not need to be repaired. On the contrary, if there is any difference between the two hash values, it is determined that the frame in the video to be repaired has changed due to the removal of hard subtitles, and the original contains hard subtitles. The frame is marked as a frame to be repaired for subsequent repair. The corresponding frame in the original video can be regarded as a damaged frame containing original hard subtitles.

[0098] Optionally, the step 103 comprises: performing grayscale processing on the frame to be repaired and the corresponding original frame; calculating the difference value of the pixel grayscale value at the same position, and marking the pixel as a pixel point to be repaired when the difference value exceeds a preset threshold; based on the pixel point to be repaired, identifying and determining the repair area by a region merging algorithm.

[0099] It should be noted that this part elaborates the specific implementation of locating the repair area according to the frame to be repaired. The principle is based on the fact that the video to be repaired only changes in pixel content in some frames due to the removal of hard subtitles. By comparing the frame to be repaired with the damaged frame (i.e., the frame containing original hard subtitles in the original video) corresponding to the same time in the original video after preprocessing, the area whose pixel value changes only due to the hard subtitle removal operation can be accurately isolated, thereby locating the repair area.

[0100] In the embodiments of the present application, specifically, for each frame of the frame to be repaired, pixel-level difference analysis is performed on the damaged frame corresponding to the same time in the original video after the preprocessing to identify the changed pixels. First, the image data of the frame to be repaired and the image data of the damaged frame are subjected to grayscale processing to obtain corresponding grayscale images. Then, each pixel position of the grayscale images is traversed, and the grayscale value of the pixel at the position in the frame to be repaired is compared with the grayscale value of the pixel at the same position in the damaged frame. If the absolute value of the difference between the two corresponding pixel grayscale values is less than or equal to a preset grayscale difference threshold , it is determined that the pixel at the position has not changed significantly; otherwise, if the absolute value of the difference between the two grayscale values is greater than the grayscale difference threshold , it is determined that the pixel at the position is a pixel affected or changed due to the removal of the hard subtitle, and is marked as a pixel to be repaired. The grayscale difference threshold is a preset parameter for setting the identification of pixel difference (for example, preferably, = 3).

[0101] After all the to-be-repaired pixels in the to-be-repaired frame image are identified, these discrete to-be-repaired pixels need to be clustered or combined into one or more connected areas, i.e., preliminary damage areas. The preliminary damage area refers to a pixel aggregation block identified from the to-be-repaired pixels by a preliminary local spatial standard (such as the spatial connectivity or local density of the pixels). The identification standard of the preliminary damage area can be based on, but not limited to, the spatial connectivity or local density of the to-be-repaired pixels. For example, when a plurality of (e.g., a preset number of consecutive pixels, preferably 6) to-be-repaired pixels are detected in the horizontal or vertical direction, a line segment composed of these consecutive pixels can be determined as a preliminary damage area. Alternatively, when the number of to-be-repaired pixels in a local neighborhood of a preset size (e.g., preferably a 3x3 pixel region) reaches or exceeds a preset threshold (e.g., preferably 6 / 9 of the total number of pixels in the region), the local neighborhood can be determined as a preliminary damage area. The preliminary damage areas can be generated by various spatial relationship-based methods including but not limited to clustering or region growing. After all the preliminary damage areas are generated, a region merging process is needed to obtain a final to-be-repaired region set. The to-be-repaired region set refers to the final connected region obtained after merging, which is the target range for actual pixel reconstruction. The merging process aims to merge spatially related, adjacent or overlapping preliminary damage areas into one or more larger, logically connected to-be-repaired regions. The merging rules can include but are not limited to: if two preliminary damage areas have common pixels or are adjacent to each other, they are merged into a unified to-be-repaired region. By iteratively applying the merging rules until no regions meet the conditions for further merging, the to-be-repaired region set is finally obtained with precise boundaries. The to-be-repaired region set is the target range for pixel reconstruction and texture synthesis using reference image information in subsequent steps.

[0102] Through the above pixel difference detection, to-be-repaired pixel marking, region clustering and region merging processes, the accurate definition of the to-be-repaired region provides a basis and guidance for subsequent efficient and accurate pixel reconstruction and texture synthesis of the damaged region using reference image information.

[0103] Optionally, the step 104 comprises: based on the spatial node relationship of the objects to which the feature points in the reference frame belong and the positions of the corresponding feature points in the to-be-repaired frame, a structure prediction model is used to predict the possible coordinates of the points in the to-be-repaired region, and the possible coordinates are obtained by averaging the prediction results of multiple analysis nodes.

[0104] In a preferred embodiment, the structure prediction model preliminarily predicts possible coordinates of potential points in the to-be-repaired region based on object structure features. The prediction method can utilize the positional relationship of the contrast nodes in the reference frame and the positional relationship of the corresponding matching nodes in the to-be-repaired frame, combined with object structure features, to make an inference.

[0105] The to-be-analyzed region contains both undamaged regions that do not need to be repaired and damaged regions that need to be repaired. Due to image blur in the damaged region, the corresponding feature points cannot be accurately detected, or the positions of the feature points deviate. However, the feature points in the damaged region are on the same object as the feature points in the undamaged region, so the positional relationship of the feature points in the damaged region relative to the feature points in the undamaged region is nearly consistent with the positional relationship of the two feature points in the corresponding reference frame of the object. However, considering that the object moves slightly when the video is being shot, the feature structure is deformed to some extent, so more relative positions need to be referred to to determine the possible coordinates of the possible feature points in the damaged region.

[0106] The calculation process of the possible coordinates and the related formula (Formula 2) and the definition of the variables are described in detail in the section on structure prediction in the main part of step 104 described above. The possible coordinates are obtained by averaging the prediction results of multiple analysis nodes, which helps to improve the robustness and accuracy of the prediction.

[0107] Optionally, the step 104 includes:

[0108] The motion pattern analysis adopts a difference prediction strategy, and the difference prediction strategy adopts a difference autoregressive moving average model to predict the movement difference.

[0109] In a preferred embodiment, the motion pattern analysis can predict the motion in the to-be-repaired region according to the motion law of the undamaged region in the to-be-analyzed region. The core of the prediction process is time series analysis of the motion data. First, the motion reference vectors corresponding to the numerous contrast nodes in the to-be-analyzed region are arranged in order of distance from the to-be-predicted position in the to-be-repaired region, thereby constructing a reference motion vector sequence. This sequence reflects the trend of the motion vector changing with the spatial position (or the implicit time / structure order) on the same object.

[0110] Since the motion of objects in a video can not be strictly stationary, the motion vector sequence can have a trend or periodicity, and direct prediction of future motion can not be accurate. Therefore, the embodiment adopts a differencing operation. Specifically, the motion vector sequence is differentiated to obtain a motion difference sequence. This motion difference sequence represents the amount of change between two adjacent motion vectors, i.e., the "instantaneous" difference or rate of change of motion. Through the differencing process, the non-stationarity present in the original motion vector sequence can be eliminated, making the data more stable and thus more easily revealing its inherent patterns and rules.

[0111] On this basis, the motion pattern analysis adopts a difference prediction strategy that uses a difference autoregressive moving average (DARMA) model to predict the movement difference. As a classic time series analysis model, the DARMA model can effectively process non-stationary time series data with trend or seasonal fluctuations. In the embodiment, the DARMA model predicts the movement difference of each node in the repair region relative to its previous adjacent node based on the motion difference sequence values at a number of past time points (or spatial points) and the past prediction errors (moving average part). The model effectively predicts the future changes in the sequence by establishing a linear relationship between the current value and the past value, and the past prediction error. The DARMA model combines the autocorrelation information of the historical difference sequence (AR part) and the sliding average information of the residual error (MA part), and can effectively model the dynamic changes in the local motion pattern. In specific implementation, the autoregressive order p and the moving average order q of the DARMA model can be set according to the motion stationarity between the reference frame and the repair frame. Preferably, a sliding window can be used to estimate the parameters and train the model for a plurality of adjacent contrast nodes to improve the accuracy and robustness of the prediction.

[0112] It should be noted that the result predicted by the DARMA model is the motion difference amount of the node in the repair region relative to its adjacent known node. This motion difference amount is used to superimpose the corrected motion vector based on the known reference motion vector. The corrected motion vector is further used to construct the sub-pixel mapping relationship and reconstruct the reference image.

[0113] Alternatively, to ensure that the model is applicable to complex texture or dynamic background scenes, the difference prediction process can combine an error control mechanism to re-label areas with prediction residuals exceeding a preset threshold as low-confidence areas and further process them by other alternative methods (such as structure interpolation or repair based on image completion).

[0114] Optionally, the step 105 comprises: dividing the to-be-repaired region into a plurality of neighborhoods centered on feature points; mapping pixel values of the neighborhoods in the reference frame to corresponding regions in the to-be-repaired frame according to the corresponding corrected motion vectors; and constructing a corrected reference image corresponding to the to-be-repaired region.

[0115] It should be noted that, first, the to-be-repaired region is divided into a plurality of neighborhood regions centered on feature points. The neighborhood division is performed according to the feature point coordinates in the obtained corrected motion vectors. Preferably, the entire to-be-repaired region is divided into a plurality of non-overlapping sub-regions by using K-means clustering, Voronoi diagram division, or an image distance field-based method, and each sub-region takes a feature point at the center of the sub-region as a representative point. This division mode ensures that the motion trend within each neighborhood is relatively consistent in space, thereby facilitating subsequent unified mapping and reconstruction processing.

[0116] Further, for each of the neighborhood regions, a pixel mapping position in the reference frame is calculated according to the corrected motion vector corresponding to the center feature point. The corrected motion vector provides a correspondence between the reference frame and the to-be-repaired frame, and by means of the vector, an original pixel block in the reference frame can be mapped to the current neighborhood position. To improve the mapping accuracy, preferably, a sub-pixel interpolation technique is used to reconstruct the gray value or color value of the non-integer coordinate mapping position. The interpolation method can include but is not limited to bilinear interpolation, cubic convolution interpolation, Lanczos interpolation, etc., and the specific selection can be configured according to the actual image definition requirement.

[0117] Further, all neighborhood regions are traversed, and pixel mapping and filling of the reference frame to the to-be-repaired frame are completed according to the above method, that is, a complete corrected reference image is obtained. The corrected reference image maintains the image information of the lossless region in the reference frame in structure, and highly matches the original content distribution of the to-be-repaired region in texture details and edge coherence, thereby laying a foundation for subsequent image fusion and final repair frame generation.

[0118] Optionally, after the construction is completed, the corrected reference image can also be subjected to unified brightness normalization processing to adapt to the brightness deviation caused by illumination changes between different shooting frames, thereby improving the fusion accuracy.

[0119] Optionally, the step 106 comprises: applying mean filtering to the corrected reference image to obtain a filtered image; and performing pixel-level fusion between the filtered image and the corresponding to-be-repaired region in the to-be-repaired frame.

[0120] In a preferred embodiment, the fusion process includes two sub-processes. First, mean filter is applied to the modified reference image to obtain a filtered image. It is noted that the modified reference image is directly mapped or reconstructed from the pixels in the reference frame, and usually retains its original pixel value characteristics (e.g. gray value or brightness). However, due to the difference in video shooting environment, lens parameters or time points, there may be overall or local light-dark (brightness) difference between the reference frame and the frame to be repaired. Such difference may cause obvious visual discontinuity at the region boundary when the modified reference image is directly replaced by the region to be repaired. In order to solve the above-mentioned light-dark difference problem and ensure the smoothness and visual consistency of fusion, in this link, mean filter is applied to the modified reference image to adjust its pixel value. The mean filter is a standard image smoothing technique, which can soften the image and adjust its brightness distribution by averaging the pixel value and the pixel value in its neighborhood, to obtain a filtered image.

[0121] Secondly, the filtered image is fused with the corresponding region to be repaired in the frame to be repaired at the pixel level. In the embodiments of the present application, the fusion process is to replace or combine the pixel information of the modified reference image adjusted by mean filter into the region to be repaired in the frame to be repaired positioned by step 103, so as to fill the region originally covered by hard subtitle with effective image information from the reference frame. The fusion is carried out at the pixel level, that is, the pixel value in the filtered modified reference image is replaced or combined into the pixel position in the corresponding region to be repaired in the frame to be repaired according to certain rules.

[0122] Optionally, the step 101 includes: simultaneously performing spatial domain mean filtering and time domain mean filtering on the original video frame.

[0123] In a preferred embodiment, the preprocessing of the original video is a necessary link, which aims to improve the image quality of the video and suppress the noise that may affect the subsequent processing.

[0124] Specifically, the preprocessing focuses on simultaneously performing spatial domain mean filtering and temporal domain mean filtering on the original video frames. The spatial domain mean filtering achieves picture smoothing and noise suppression by averaging the pixel values in a pixel point and its spatial neighborhood in a video frame. The temporal domain mean filtering effectively reduces the high-frequency noise commonly existing in video frames and enhances the inter-frame texture consistency by averaging the pixel values in a pixel point and its temporal neighborhood (i.e., the corresponding positions in adjacent video frames). The method of simultaneously performing spatial domain and temporal domain mean filtering can more effectively remove the high-frequency noise on numerous video frames to obtain the original video after preprocessing. The noise reduction algorithm can effectively reduce the high-frequency noise commonly existing in video frames to obtain the original video after preprocessing through the above processing.

[0125] Optionally, the step 104 further comprises:

[0126] Edge detection is performed on the reference frame, and a morphological process is used to connect broken edges to divide the reference frame into a plurality of object regions; and the matching feature points are matched only in the object regions.

[0127] It should be noted that this part elaborates the specific implementation of the above optional step. This optional step aims to utilize the object structure information in the reference frame to optimize the subsequent feature point matching and motion vector calculation processes.

[0128] In one embodiment, the optional step includes two sub-processes. First, edge detection is performed on the reference frame, and a morphological process is used to connect broken edges to divide the reference frame into a plurality of object regions. This process identifies and divides different objects or regions based on the image content of the reference frame through image processing techniques. Preferably, the edge detection can use common algorithms such as Sobel, Canny, or LoG (Laplacian of Gaussian) operators to identify strong gradient changes in the image and then outline the edge contours of potential objects. For example, in a specific implementation, the edges in the reference frame can be detected using edge detection operators including but not limited to the Sobel operator; then, a morphological process including but not limited to a closing operation is used to connect possibly broken edge lines to form closed contours; subsequently, methods including but not limited to line scanning algorithms or filling algorithms are used to fill the interior regions of the closed edges, thereby dividing the reference frame into a plurality of independent object regions. Thus, the object region division result of the reference frame is obtained.

[0129] Secondly, the image feature points to be extracted are matched only within the same article region. The process is to use the article region division result obtained in the previous step to constrain the feature point matching process in the subsequent step (usually step 104, the feature point matching link). Specifically, after extracting the image feature points of the reference frame and the frame to be repaired, when performing feature point matching, only matching between feature points located in the same article region is attempted. For example, a feature point located in the article A region in the reference frame is only attempted to match with the feature point located in the same article A region (or other corresponding region determined according to the preliminary motion estimation) in the frame to be repaired. By limiting the matching to the same article region, it is possible to exclude false matching between different objects, because different objects usually move independently. Thus, the regional consistency of the matched feature points is improved, and the accuracy of the motion vector calculation is improved based on more reliable matching points.

[0130] The adaptive video repair method based on digital video technology provided by the embodiment of the application filters the reference frame and the frame to be repaired by using a unique comparison method, and accurately locates the region to be repaired in combination with pixel-level difference analysis. In particular, the method based on the image feature point motion vector, the object structure relationship and the prediction model is used to construct the correction motion vector of the region to be repaired, and then sub-pixel level pixel mapping and interpolation reconstruction are performed according to the correction motion vector, and the reconstructed image is fused, which effectively overcomes the problems of loss of original region details and poor repair effect after removal of the hard subtitle in the prior art, and has significant technical progress significance and practical value. The method of the application can realize high-precision and high-fidelity repair of the region after removal of the hard subtitle, restore the structure and texture details of the original video picture, improve the spatial consistency of the repaired region and the surrounding picture, and ensure the time continuity of the video sequence, and significantly improve the overall visual quality and viewing experience of the repaired video.

[0131] Figure 2 An adaptive video repair device 20 based on digital video technology provided by the embodiment of the application is schematically shown, which is usually integrated in an electronic device with a processor and a memory. The device comprises:

[0132] The receiving and preprocessing module 201 is configured to pre-process the original video containing the hard subtitle and automatically remove the hard subtitle to obtain a video to be repaired;

[0133] The function module 202 is configured to: perform comparison and screening on reference frames and frames to be repaired based on the original video and the video to be repaired; perform pixel gray value difference analysis on the frames to be repaired and corresponding frames of the original video to determine a region to be repaired; construct a modified motion vector of the region to be repaired, the construction including extracting matching feature points of the reference frames and the frames to be repaired, and calculating reference motion vectors of the matching feature points; performing object segmentation to obtain a plurality of object regions and identifying an analysis region intersecting with the region to be repaired; performing structure prediction on the analysis region to predict possible coordinates of points in the region to be repaired; performing motion mode analysis to predict a movement difference condition of the region to be repaired; and generating the modified motion vector in combination with the reference motion vector and the movement difference condition; and determining pixel and sub-pixel corresponding source positions of the region to be repaired according to the modified motion vector, and performing sub-pixel interpolation to obtain pixel values to construct a modified reference image.

[0134] The fusion generation module 203 is configured to fuse the modified reference image and the region to be repaired to generate a repaired video frame.

[0135] Optionally, the receiving and preprocessing module 201 is further configured to: locate a region where the hard subtitle is located by using an image recognition algorithm, and replace or cover pixels in the region to remove the hard subtitle.

[0136] Optionally, the function module 202 is further configured to:

[0137] The function module 202 is further configured to: calculate hash values of video frames at the same time point in the original video and the video to be repaired respectively, and determine whether the video frames are reference frames by bit-by-bit accurate comparison.

[0138] Optionally, the function module 202 is further configured to: perform gray processing on the frames to be repaired and corresponding original frames; perform difference calculation on pixel gray values at the same positions, and mark a pixel as a pixel point to be repaired when a difference value exceeds a preset threshold; and identify and determine the region to be repaired based on the pixel point to be repaired by using a region merging algorithm.

[0139] Optionally, the function module 202 is further configured to: predict possible coordinates of points in the region to be repaired by using a structure prediction model based on spatial node relationships of objects to which feature points in the reference frames belong and positions of corresponding feature points in the frames to be repaired, the possible coordinates being obtained by averaging prediction results of a plurality of analysis nodes.

[0140] Optionally, the function module 202 is further configured to:

[0141] The motion pattern analysis adopts a difference prediction strategy, which adopts a difference autoregressive moving average model to predict the movement difference.

[0142] Optionally, the function module 202 is further configured to: divide the to-be-repaired region into a plurality of neighborhoods centered on feature points; map pixel values of the neighborhoods in the reference frame to corresponding regions in the to-be-repaired frame according to corresponding correction motion vectors; and construct a correction reference image corresponding to the to-be-repaired region.

[0143] Optionally, the fusion generation module 203 is further configured to: apply mean filtering processing to the correction reference image to obtain a filtered image; and perform pixel-level fusion between the filtered image and the corresponding to-be-repaired region in the to-be-repaired frame.

[0144] Optionally, the receiving and preprocessing module 201 is further configured to: simultaneously perform spatial domain mean filtering and time domain mean filtering on the original video frame.

[0145] Optionally, the function module 202 is further configured to:

[0146] perform edge detection on the reference frame and connect broken edges through morphological processing to divide the reference frame into a plurality of object regions; and match the feature points only in the object regions.

[0147] The embodiments of the present application can accurately locate and reconstruct the details of the damaged region after removing the hard subtitles by combining global frame-level hash screening, feature point motion modeling, and spatio-temporal mapping fusion, thereby significantly improving the image restoration quality of the hard subtitle region. The embodiments of the present application realize structure-aware, spatio-temporally consistent high-quality video repair, and have significant practical value and promotion prospects.

[0148] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments of the present application according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0149] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or combinations thereof. Those skilled in the art will appreciate that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some or all of the components of the computing device according to embodiments of the present application. The present application can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for performing part or all of the methods described herein. Such a program implementing the present application can be stored on a non-transitory computer readable medium, or can be in the form of signals on the Internet or other transmission media. The signals can be downloaded into the Internet website, or provided on a carrier medium, or in any other form.

[0150] For example, Figure 3 A block diagram of a computing device for performing the methods according to embodiments of the present application is schematically shown. The computing device conventionally comprises a processor 310 and a computer program product or a non-transitory computer readable medium in the form of a memory 320. The memory 320 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. The memory 320 has a storage space 330 for program code 331 for performing any of the method steps in the methods described above. For example, the storage space 330 for program code can comprise individual program codes 331 for implementing the various steps in the above methods, respectively. The program codes can be read from or written to one or more computer program products. The computer program products comprise program code carriers such as a hard disk, a compact disk (CD), a memory card, or a floppy disk. Such computer program products are typically portable or stationary memory units as referred to in Figure 4 Figure 3 The memory 320 in the computing device of the present application can have a similar arrangement of storage segments, storage spaces, etc. as the memory 320 in the computing device of the present application. The program code can be compressed, for example, in a suitable form. Typically, the memory unit comprises computer readable code 431, i.e. code that can be read by a processor such as 310, which, when run by the computing device, causes the computing device to perform the various steps in the methods described above.

[0151] ​It should be understood that, although the steps in the flowcharts of the drawings are shown in a sequence as indicated by arrows, the steps are not necessarily executed in the order as indicated by the arrows. Unless explicitly stated otherwise herein, the execution of the steps is not necessarily limited to the order as indicated by the arrows, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution of which is not necessarily sequential, but can be round-robin or alternately executed with at least some of the steps or sub-steps or stages of other steps.

[0152] As used in this description, the terms "one embodiment", "an embodiment” or "one or more embodiments” mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment” in various places in this description are not necessarily all referring to the same embodiment.

[0153] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0154] It should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, but not limit the present application; although the present application is described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that they can still make modifications to the technical solutions recorded in the above-mentioned embodiments, or replace equivalent replacements to part of the technical features; and the modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for adaptive video inpainting based on digital video technology, characterized in that, The method comprises: preprocessing an original video containing hard subtitles and automatically removing the hard subtitles to obtain a video to be repaired; based on the original video and the video to be repaired, comparing and screening reference frames and frames to be repaired; performing pixel gray value difference analysis on the frames to be repaired and corresponding frames of the original video to determine the region to be repaired; constructing a corrected motion vector of the region to be repaired, which comprises extracting matching feature points of the reference frames and the frames to be repaired, calculating a reference motion vector of the matching feature points, performing object segmentation on the reference frames to obtain a plurality of object regions and identifying an analysis region intersecting with the region to be repaired, performing structure prediction on the analysis region to predict possible coordinates of points in the region to be repaired, performing motion mode analysis on the region to be repaired to predict a movement difference condition of the region to be repaired, and combining the reference motion vector and the movement difference condition to generate the corrected motion vector; determining a corresponding source position of a pixel sub-pixel of the region to be repaired according to the corrected motion vector, and performing sub-pixel interpolation to obtain a pixel value to construct a corrected reference image; fusing the corrected reference image and the region to be repaired to generate a repaired video frame.

2. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The automatic removal of the hard subtitles comprises: locating the region where the hard subtitles are located by an image recognition algorithm, and replacing or covering the pixels in the region where the hard subtitles are located to remove the hard subtitles.

3. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The comparison comprises: calculating hash values of video frames at the same time point in the original video and the video to be repaired respectively, and judging whether they are reference frames through bit-by-bit accurate comparison.

4. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The pixel gray value difference analysis on the frames to be repaired and corresponding frames of the original video to determine the region to be repaired comprises: performing gray processing on the frames to be repaired and their corresponding original frames; performing difference calculation on the gray values of pixels at the same position, and marking the pixels as to-be-repaired pixels when the difference exceeds a preset threshold; based on the to-be-repaired pixels, identifying and determining the region to be repaired through a region merging algorithm.

5. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The structure prediction on the analysis region to predict possible coordinates of points in the region to be repaired comprises: based on the spatial node relationship of objects to which each feature point in the reference frame belongs and the positions of corresponding feature points in the frames to be repaired, a structure prediction model is used to predict possible coordinates of points in the region to be repaired, and the possible coordinates are obtained by averaging the prediction results of a plurality of analysis nodes.

6. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The motion mode analysis to predict the movement difference condition of the region to be repaired comprises: The motion mode analysis uses a difference prediction strategy, and the difference prediction strategy uses a difference autoregressive moving average model to predict the movement difference condition.

7. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The construction of the corrected reference image comprises: dividing the region to be repaired into a plurality of neighborhoods centered on feature points; mapping pixel values of the neighborhoods in the reference frames to corresponding regions in the frames to be repaired according to corresponding corrected motion vectors; constructing a corrected reference image corresponding to the region to be repaired.

8. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The fusion of the corrected reference image and the region to be repaired comprises: applying mean filtering to the modified reference image to obtain a filtered image; performing pixel-level fusion between the filtered image and a corresponding to-be-repaired region in the to-be-repaired frame.

9. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The preprocessing of the original video containing hard subtitles includes: performing spatial domain mean filtering and time domain mean filtering on the original video frame simultaneously.

10. The adaptive video inpainting method based on digital video technology according to claim 1, wherein, The object segmentation to obtain a plurality of object regions includes: performing edge detection on the reference frame and connecting broken edges through morphological processing to divide the reference frame into a plurality of object regions; the matching of the feature points is performed only within the object regions.

Images