Video restoration method and device, electronic equipment and storage medium
By using reference feature points from adjacent video frames to correct feature points during the video restoration process, the problem of foreground object jitter after video restoration is solved, achieving a more stable video restoration effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING QIYI CENTURY SCI & TECH CO LTD
- Filing Date
- 2023-07-26
- Publication Date
- 2026-04-17
AI Technical Summary
In existing technologies, the foreground object in video restoration often exhibits jitter, mainly due to the jitter of feature points caused by lighting or noise.
By acquiring the video frame to be repaired, the object region of the foreground object is determined and its features are labeled. The initial position of the feature points is corrected by using the reference feature points of adjacent video frames. The feature weights and distances are calculated to determine the correction parameters. Finally, the video frame is repaired based on these parameters.
It reduces the impact of light and noise on foreground objects, improves the jitter problem of foreground objects after restoration, and enhances the stability of video restoration.
Smart Images
Figure CN117151997B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video processing technology, and in particular to a video restoration method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the continuous development of internet technology, online video products are becoming increasingly abundant. Users are no longer limited to watching television; they can also search for content of interest online. However, some video products suffer from various quality issues due to their age or aging transmission media, such as noise, blurriness, and distortion. These problems severely affect the viewing experience.
[0003] To address these issues, video products are typically repaired. During this process, the restoration of foreground objects (such as faces) is a crucial step. Currently, the general process for restoring foreground objects in video products involves: for each video frame, extracting the feature points of the foreground objects and their corresponding positions; and then performing restoration work on the foreground objects based on all feature points and their positions.
[0004] However, when video restoration is performed using the above methods, the jitter of feature points caused by the influence of light or noise will result in jitter in the restored foreground object. Summary of the Invention
[0005] The purpose of this application is to provide a video restoration method, apparatus, electronic device, and storage medium to solve the problem of jitter in the restored foreground object. The specific technical solution is as follows:
[0006] Firstly, a video restoration method is provided, the method comprising:
[0007] Obtain the video to be repaired, wherein the video to be repaired includes multiple consecutive video frames containing the same foreground object;
[0008] For each video frame, the object region corresponding to the foreground object is determined in the video frame, and the object region is subjected to feature annotation processing to obtain at least one corresponding feature point and the initial position of each feature point in the video frame.
[0009] In the reference video frame corresponding to the video frame, a reference feature point is determined for each feature point, and the reference video frame is the preceding video frame adjacent to the video frame.
[0010] Based on the reference feature point corresponding to each feature point, the initial position corresponding to each feature point is corrected to obtain the target position corresponding to each feature point;
[0011] Based on each feature point and the target location corresponding to each feature point, the object region in the video frame is repaired.
[0012] In one possible implementation, the step of correcting the initial position of each feature point based on a reference feature point to obtain the target position of each feature point includes:
[0013] Determine the feature weight corresponding to each feature point, and determine the distance between each feature point and the corresponding reference feature point;
[0014] Based on the feature weights corresponding to each feature point and the distances corresponding to each feature point, the correction parameters are determined;
[0015] For each feature point, the initial position corresponding to the feature point and the correction parameter are summed to obtain the target position.
[0016] In one possible implementation, determining the correction parameters based on the feature weights corresponding to each feature point and the distances corresponding to each feature point includes:
[0017] For each feature point, the product of the feature weight corresponding to the feature point and the distance is used as the corresponding correction base.
[0018] The correction parameters are obtained by summing the correction bases corresponding to all feature points.
[0019] In one possible implementation, determining the distance between each feature point and its corresponding reference feature point includes:
[0020] For each feature point, determine the motion trajectory between the feature point and its corresponding reference feature point;
[0021] The motion distance is determined based on the motion trajectory, and the motion distance is defined as the distance between the feature point and the reference feature point.
[0022] In one possible implementation, determining the feature weight corresponding to each feature point includes:
[0023] For each feature point, the feature weight corresponding to the feature point is determined based on the distance between the feature point and the corresponding reference feature point, wherein the greater the distance, the greater the corresponding feature weight.
[0024] In one possible implementation, before determining the reference feature point corresponding to each feature point in the reference video frame corresponding to the video frame, the method further includes:
[0025] During the process of annotating each feature point in the video frame, the first annotation order corresponding to each feature point is recorded; and during the process of annotating each reference feature point in the reference video frame, the second annotation order corresponding to each reference feature point is recorded.
[0026] The step of determining the reference feature point corresponding to each feature point in the reference video frame corresponding to the video frame includes:
[0027] For each feature point, the first annotation order corresponding to the feature point is determined as the target annotation order, and the reference feature points whose second annotation order is the same as the target annotation order are determined as the reference feature points corresponding to the feature points.
[0028] In one possible implementation, after repairing the object region in the video frame based on each feature point and the target location corresponding to each feature point, the method further includes:
[0029] Construct an object mask corresponding to the foreground object based on all the aforementioned feature points;
[0030] The video frame and the repaired image obtained by repairing the object region based on the object mask are then synthesized.
[0031] Secondly, a video restoration device is provided, the device comprising:
[0032] An acquisition module is used to acquire the video to be repaired, wherein the video to be repaired includes multiple consecutive video frames containing the same foreground object;
[0033] The annotation module is used to determine the object region corresponding to the foreground object in each video frame, and to perform feature annotation processing on the object region to obtain at least one corresponding feature point and the initial position of each feature point in the video frame.
[0034] The determining module is used to determine a reference feature point corresponding to each feature point in a reference video frame corresponding to the video frame, wherein the reference video frame is the preceding video frame adjacent to the video frame.
[0035] The correction module is used to correct the initial position of each feature point based on the reference feature point corresponding to each feature point, so as to obtain the target position of each feature point.
[0036] The repair module is used to repair the object region in the video frame based on each feature point and the target position corresponding to each feature point.
[0037] In one possible implementation, the correction module is specifically used for:
[0038] Determine the feature weight corresponding to each feature point, and determine the distance between each feature point and the corresponding reference feature point;
[0039] Based on the feature weights corresponding to each feature point and the distances corresponding to each feature point, the correction parameters are determined;
[0040] For each feature point, the initial position corresponding to the feature point and the correction parameter are summed to obtain the target position.
[0041] In one possible implementation, the correction module is further configured to:
[0042] For each feature point, the product of the feature weight corresponding to the feature point and the distance is used as the corresponding correction base.
[0043] The correction parameters are obtained by summing the correction bases corresponding to all feature points.
[0044] In one possible implementation, the correction module is further configured to:
[0045] For each feature point, determine the motion trajectory between the feature point and its corresponding reference feature point;
[0046] The motion distance is determined based on the motion trajectory, and the motion distance is defined as the distance between the feature point and the reference feature point.
[0047] In one possible implementation, the correction module is further configured to:
[0048] For each feature point, the feature weight corresponding to the feature point is determined based on the distance between the feature point and the corresponding reference feature point, wherein the greater the distance, the greater the corresponding feature weight.
[0049] In one possible implementation, the device further includes a labeling module for:
[0050] During the process of annotating each feature point in the video frame, the first annotation order corresponding to each feature point is recorded; and during the process of annotating each reference feature point in the reference video frame, the second annotation order corresponding to each reference feature point is recorded.
[0051] The determining module is specifically used for:
[0052] For each feature point, the first annotation order corresponding to the feature point is determined as the target annotation order, and the reference feature points whose second annotation order is the same as the target annotation order are determined as the reference feature points corresponding to the feature points.
[0053] In one possible implementation, the apparatus further includes a synthesis module for:
[0054] Construct an object mask corresponding to the foreground object based on all the aforementioned feature points;
[0055] The video frame and the repaired image obtained by repairing the object region based on the object mask are then synthesized.
[0056] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0057] Memory, used to store computer programs;
[0058] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.
[0059] Fourthly, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described in the first aspect.
[0060] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform any of the video restoration methods described above.
[0061] Beneficial effects of the embodiments in this application:
[0062] This application provides a video restoration method, apparatus, electronic device, and storage medium. In this embodiment, firstly, a video to be restored is acquired. Then, for each video frame in the video to be restored, an object region corresponding to the foreground object is determined within that video frame, and feature annotation processing is performed on the object region to obtain at least one corresponding feature point and the initial position of each feature point in the video frame. Next, in a reference video frame corresponding to that video frame, a reference feature point corresponding to each feature point is determined, and based on the reference feature point, the initial position of each feature point is corrected to obtain the target position of each feature point. Finally, based on each feature point and its corresponding target position, the object region in the video frame is restored. Thus, during the restoration of the object region in each video frame, the position of each feature point in the object region of that video frame can be corrected based on the preceding video frame (i.e., the reference video frame), thereby reducing the influence of light or noise and improving the jitter problem of the restored foreground object.
[0063] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description
[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0066] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0067] Figure 1 A flowchart illustrating a video restoration method provided in this application embodiment;
[0068] Figure 2 This is a schematic diagram of feature point annotation provided in an embodiment of this application;
[0069] Figure 3 A flowchart illustrating another video restoration method provided in this application embodiment;
[0070] Figure 4This is a schematic diagram of the structure of a video restoration device provided in an embodiment of this application;
[0071] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0073] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.
[0074] The following will describe in detail a video restoration method provided in this application embodiment, with reference to specific implementation methods. Figure 1 As shown, the specific steps are as follows:
[0075] S101, Obtain the video to be repaired, wherein the video to be repaired includes multiple consecutive video frames containing the same foreground object.
[0076] The video to be repaired includes multiple consecutive video frames containing the same foreground object. The foreground object can be a person, animal, plant, or other inanimate object. In practical applications, when watching videos containing people, users are usually more concerned with the clarity of the people's faces; therefore, the foreground object can be only the faces. When watching videos containing animals, users are usually more concerned with the clarity of the animal as a whole; therefore, the foreground object is the entire animal's body. Users can specifically set the foreground object according to their actual needs.
[0077] In one embodiment, users can upload videos that meet certain criteria to be repaired. This allows for flexible selection of videos requiring repair based on user needs.
[0078] In another embodiment, any original video can be acquired, and each video frame in the original video can be detected. Video segments consisting of multiple consecutive video frames containing the same foreground object can be segmented from the original video to obtain the video to be repaired. This eliminates the need for manual processing and uploading of the video to be repaired by the user, thereby improving processing efficiency.
[0079] S102, for each video frame, determine the object region corresponding to the foreground object in the video frame, and perform feature annotation processing on the object region to obtain at least one corresponding feature point and the initial position of each feature point in the video frame.
[0080] The object region refers to the object region that contains only foreground objects.
[0081] In this embodiment, for each video frame, a foreground object can be identified in the video frame using object detection algorithms or image recognition techniques, and the area where the foreground object is located is determined as the object region. Then, feature annotation processing is performed on the object region to obtain at least one corresponding feature point, and the initial position (i.e., annotation position) of each feature point in the video frame.
[0082] Taking a human face as an example, the object region where the face is located can be processed by facial key point annotation (e.g., 68-point annotation, 106-point annotation, 224-point annotation, 228-point annotation, etc.) to obtain at least one corresponding facial key point (i.e., feature point), as well as the initial position of each feature point in the video frame.
[0083] S103, in the reference video frame corresponding to the video frame, determine the reference feature point corresponding to each feature point, wherein the reference video frame is the preceding video frame adjacent to the video frame.
[0084] For each video frame, the reference video frame is the preceding video frame that is adjacent to it.
[0085] In one embodiment, before determining the reference feature point corresponding to each feature point in the reference video frame corresponding to the video frame, the following steps are further included:
[0086] During the process of annotating each feature point in the video frame, the first annotation order corresponding to each feature point is recorded; and during the process of annotating each reference feature point in the reference video frame, the second annotation order corresponding to each reference feature point is recorded.
[0087] In this embodiment, during the process of annotating feature points for each video frame, the annotation is performed according to a pre-set fixed annotation order, and the annotation order of each feature point is recorded during the annotation process.
[0088] Taking the basic 68-point facial landmark annotation as an example, the annotation order is as follows:
[0089] The order of labeling body parts is: facial contour, eyebrows, nose, eyes, and mouth.
[0090] The labeling order for each part is: from left to right, from top to bottom.
[0091] like Figure 2 The image shown is a diagram illustrating the annotation of a face using 68 facial landmarks. Asterisks represent the annotated feature points, and numbers indicate the annotation order of the feature points.
[0092] Based on this, in the embodiments of this application, the specific implementation of determining the reference feature point corresponding to each feature point in the reference video frame corresponding to the video frame may include:
[0093] For each feature point, the first annotation order corresponding to the feature point is determined as the target annotation order, and the reference feature points whose second annotation order is the same as the target annotation order are determined as the reference feature points corresponding to the feature points.
[0094] For example, for a feature point in a video frame with a labeling order of 6, refer to a reference feature point in the same video frame with a labeling order of 6 as its corresponding reference feature point; for a feature point in a video frame with a labeling order of 8, refer to a reference feature point in the same video frame with a labeling order of 8 as its corresponding reference feature point; and so on, to determine the reference feature point corresponding to each feature point.
[0095] This scheme allows for the association of feature points in each video frame with reference feature points in the corresponding reference video frame by using the labeling order, thereby quickly determining the reference feature point corresponding to each feature point.
[0096] S104, based on the reference feature point corresponding to each feature point, the initial position corresponding to each feature point is corrected to obtain the target position corresponding to each feature point.
[0097] S105, based on each feature point and the target position corresponding to each feature point, the object region in the video frame is repaired.
[0098] The following provides a unified explanation of S104 and S105:
[0099] In this embodiment, for each feature point in a video frame, firstly, based on the corresponding reference feature point in the reference video frame, the initial position of the directly detected feature point is corrected to obtain the corresponding target position. Then, based on each feature point and its corresponding target position, the object region in the video frame is repaired. This reduces the influence of light or noise and improves the jitter problem of the repaired foreground object.
[0100] The specific method for correcting the initial position of each feature point based on the reference feature point corresponding to each feature point to obtain the target position of each feature point will be explained in detail through the following embodiments, and will not be elaborated here.
[0101] In addition, following S105, the following steps may also be included:
[0102] Step A1: Construct an object mask corresponding to the foreground object based on all the feature points;
[0103] Step A2: Combine the video frame and the repaired image obtained by repairing the object region based on the object mask.
[0104] The following provides a unified explanation of steps A1 and A2:
[0105] In this embodiment, the object mask marks the pixel position of the foreground object. Specifically, the synthesized image is obtained using the following formula:
[0106] Imgout=face*mask+imgori*(1-mask)
[0107] Wherein, Imgori: the original image of the video frame, Face: the repaired image after repairing the object region, Mask: the object mask, and Imgout: the composite complete image.
[0108] In another embodiment, the edges of the foreground object can be blurred based on the object mask to smoothly transition into the background area. Specifically, the object mask is first feathered (in practice, this can be achieved by applying a low-pass filter (such as a Gaussian filter) to the object mask), and then the video frame and the repaired image obtained by repairing the object region are synthesized based on the feathered object mask. This makes the transition between the foreground object and the background area more natural, further improving the repair effect.
[0109] In this embodiment, firstly, the video to be repaired is acquired. Then, for each video frame in the video to be repaired, the object region corresponding to the foreground object is determined in the video frame, and feature annotation processing is performed on the object region to obtain at least one corresponding feature point and the initial position of each feature point in the video frame. Next, in the reference video frame corresponding to the video frame, a reference feature point corresponding to each feature point is determined, and based on the reference feature point, the initial position of each feature point is corrected to obtain the target position of each feature point. Finally, based on each feature point and its corresponding target position, the object region in the video frame is repaired. Thus, during the repair process of the object region in each video frame, the position of each feature point in the object region of the video frame can be corrected based on the adjacent preceding video frame (i.e., the reference video frame), thereby reducing the influence of light or noise and improving the jitter problem of the repaired foreground object.
[0110] See Figure 3 This is a flowchart illustrating an embodiment of another video restoration method provided in this application. Figure 3 The process shown above Figure 1 Based on the illustrated process, this section describes how to correct the initial position of each feature point based on a reference feature point corresponding to each feature point, thereby obtaining the target position of each feature point. For example... Figure 3 As shown, the process may include the following steps:
[0111] S301, determine the feature weight corresponding to each feature point, and determine the distance between each feature point and the corresponding reference feature point.
[0112] In one embodiment, the specific implementation of determining the distance between each feature point and its corresponding reference feature point may include: for each feature point, determining the motion trajectory between the feature point and its corresponding reference feature point, determining the motion distance based on the motion trajectory, and determining the motion distance as the distance between the feature point and the reference feature point.
[0113] In applications, the motion of an object region is analyzed by calculating the pixel displacements between different video frames, thus obtaining the motion trajectory of each feature point in the object region. Specifically, optical flow algorithms can be used to calculate the optical flow field (e.g., using the pyramid method or gradient-based methods) to represent the pixel movement between adjacent frames. Optical flow algorithms can estimate the displacement of each pixel in an image between adjacent frames, i.e., the motion trajectory, based on changes in the grayscale values and positions of image pixels.
[0114] In another embodiment, determining the feature weight corresponding to each feature point may include: for each feature point, determining the feature weight corresponding to the feature point based on the distance between the feature point and the corresponding reference feature point, wherein the greater the distance, the greater the corresponding feature weight. This can improve the accuracy of subsequent calculations.
[0115] In another embodiment, the specific implementation of determining the feature weight corresponding to each feature point may further include: determining the weight set by the user for each feature point as the corresponding feature weight. For example, the weight of each feature point is set to 1 / n, where n is the number of feature points (e.g., for a 68-point facial landmark annotation, the number is 68). In this way, the feature weight of each feature point can be set according to user needs.
[0116] S302, determine the correction parameters based on the feature weights corresponding to each feature point and the distances corresponding to each feature point.
[0117] S303, for each feature point, sum the initial position corresponding to the feature point and the correction parameter to obtain the target position.
[0118] The following provides a unified explanation of S302 and S303:
[0119] Correction parameters are used to correct the initial position of each feature point.
[0120] Specifically, the correction parameter can be determined based on the feature weight and distance corresponding to each feature point through the following steps: for each feature point, the product of the feature weight and distance corresponding to the feature point is used as the corresponding correction base, and the correction bases corresponding to all feature points are summed to obtain the correction parameter.
[0121] Next, for each feature point, the initial position and correction parameters corresponding to that feature point are summed to obtain the corresponding target position.
[0122] In other words, assuming the initial position of the kth feature point is P o,k The corrected target position is P. u,k (For example, for a 68-point facial landmark annotation, the range of k is [1-68]).
[0123] but:
[0124] Where, d j w represents the distance of motion of feature point j between two adjacent video frames; j This represents the weight of feature point j.
[0125] pass Figure 3 The process shown can determine the distance between two adjacent video frames (i.e., the distance the feature point moves from the reference video frame to the current video frame) based on two adjacent video frames (i.e., the video frame being repaired and the reference video frame corresponding to that video frame). Then, based on the weight and distance corresponding to each feature point, the position of each feature point is corrected, thereby reducing the influence of light or noise and improving the jitter problem of the foreground object after repair.
[0126] Based on the same technical concept, embodiments of this application also provide a video restoration device, such as... Figure 4 As shown, the device includes:
[0127] The acquisition module 401 is used to acquire the video to be repaired, wherein the video to be repaired includes multiple consecutive video frames containing the same foreground object;
[0128] The annotation module 402 is used to determine the object region corresponding to the foreground object in each video frame, and to perform feature annotation processing on the object region to obtain at least one corresponding feature point and the initial position of each feature point in the video frame.
[0129] The determining module 403 is used to determine a reference feature point corresponding to each feature point in a reference video frame corresponding to the video frame, wherein the reference video frame is the preceding video frame adjacent to the video frame.
[0130] The correction module 404 is used to correct the initial position of each feature point based on the reference feature point corresponding to each feature point, so as to obtain the target position of each feature point.
[0131] Repair module 405 is used to repair the object region in the video frame based on each feature point and the target position corresponding to each feature point.
[0132] In one possible implementation, the correction module is specifically used for:
[0133] Determine the feature weight corresponding to each feature point, and determine the distance between each feature point and the corresponding reference feature point;
[0134] Based on the feature weights corresponding to each feature point and the distances corresponding to each feature point, the correction parameters are determined;
[0135] For each feature point, the initial position corresponding to the feature point and the correction parameter are summed to obtain the target position.
[0136] In one possible implementation, the correction module is further configured to:
[0137] For each feature point, the product of the feature weight corresponding to the feature point and the distance is used as the corresponding correction base.
[0138] The correction parameters are obtained by summing the correction bases corresponding to all feature points.
[0139] In one possible implementation, the correction module is further configured to:
[0140] For each feature point, determine the motion trajectory between the feature point and its corresponding reference feature point;
[0141] The motion distance is determined based on the motion trajectory, and the motion distance is defined as the distance between the feature point and the reference feature point.
[0142] In one possible implementation, the correction module is further configured to:
[0143] For each feature point, the feature weight corresponding to the feature point is determined based on the distance between the feature point and the corresponding reference feature point, wherein the greater the distance, the greater the corresponding feature weight.
[0144] In one possible implementation, the device further includes a labeling module for:
[0145] During the process of annotating each feature point in the video frame, the first annotation order corresponding to each feature point is recorded; and during the process of annotating each reference feature point in the reference video frame, the second annotation order corresponding to each reference feature point is recorded.
[0146] The determining module is specifically used for:
[0147] For each feature point, the first annotation order corresponding to the feature point is determined as the target annotation order, and the reference feature points whose second annotation order is the same as the target annotation order are determined as the reference feature points corresponding to the feature points.
[0148] In one possible implementation, the apparatus further includes a synthesis module for:
[0149] Construct an object mask corresponding to the foreground object based on all the aforementioned feature points;
[0150] The video frame and the repaired image obtained by repairing the object region based on the object mask are then synthesized.
[0151] In this embodiment, firstly, the video to be repaired is acquired. Then, for each video frame in the video to be repaired, the object region corresponding to the foreground object is determined in the video frame, and feature annotation processing is performed on the object region to obtain at least one corresponding feature point and the initial position of each feature point in the video frame. Next, in the reference video frame corresponding to the video frame, a reference feature point corresponding to each feature point is determined, and based on the reference feature point, the initial position of each feature point is corrected to obtain the target position of each feature point. Finally, based on each feature point and its corresponding target position, the object region in the video frame is repaired. Thus, during the repair process of the object region in each video frame, the position of each feature point in the object region of the video frame can be corrected based on the adjacent preceding video frame (i.e., the reference video frame), thereby reducing the influence of light or noise and improving the jitter problem of the repaired foreground object.
[0152] Based on the same technical concept, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, it includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.
[0153] Memory 113 is used to store computer programs;
[0154] When processor 111 executes a program stored in memory 113, it performs the following steps:
[0155] Obtain the video to be repaired, wherein the video to be repaired includes multiple consecutive video frames containing the same foreground object;
[0156] For each video frame, the object region corresponding to the foreground object is determined in the video frame, and the object region is subjected to feature annotation processing to obtain at least one corresponding feature point and the initial position of each feature point in the video frame.
[0157] In the reference video frame corresponding to the video frame, a reference feature point is determined for each feature point, and the reference video frame is the preceding video frame adjacent to the video frame.
[0158] Based on the reference feature point corresponding to each feature point, the initial position corresponding to each feature point is corrected to obtain the target position corresponding to each feature point;
[0159] Based on each feature point and the target location corresponding to each feature point, the object region in the video frame is repaired.
[0160] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0161] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0162] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0163] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0164] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described video restoration methods.
[0165] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the video repair methods described above.
[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0168] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.
[0169] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A video restoration method, characterized in that, The method includes: Obtain the video to be repaired, wherein the video to be repaired includes multiple consecutive video frames containing the same foreground object; For each video frame, the object region corresponding to the foreground object is determined in the video frame, and the object region is subjected to feature annotation processing to obtain at least one corresponding feature point and the initial position of each feature point in the video frame. In the reference video frame corresponding to the video frame, a reference feature point is determined for each feature point, and the reference video frame is the preceding video frame adjacent to the video frame. Based on the reference feature point corresponding to each feature point, the initial position corresponding to each feature point is corrected to obtain the target position corresponding to each feature point; Based on each feature point and the target location corresponding to each feature point, the object region in the video frame is repaired; The step of correcting the initial position of each feature point based on a reference feature point to obtain the target position of each feature point includes: Determine the feature weight corresponding to each feature point, and determine the distance between each feature point and the corresponding reference feature point; Based on the feature weights corresponding to each feature point and the distances corresponding to each feature point, the correction parameters are determined; For each feature point, the initial position corresponding to the feature point and the correction parameter are summed to obtain the target position.
2. The method according to claim 1, characterized in that, The step of determining the correction parameters based on the feature weights corresponding to each feature point and the distances corresponding to each feature point includes: For each feature point, the product of the feature weight corresponding to the feature point and the distance is used as the corresponding correction base. The correction parameters are obtained by summing the correction bases corresponding to all feature points.
3. The method according to claim 1, characterized in that, Determining the distance between each feature point and its corresponding reference feature point includes: For each feature point, determine the motion trajectory between the feature point and its corresponding reference feature point; The motion distance is determined based on the motion trajectory, and the motion distance is defined as the distance between the feature point and the reference feature point.
4. The method according to claim 1, characterized in that, Determining the feature weight corresponding to each feature point includes: For each feature point, the feature weight corresponding to the feature point is determined based on the distance between the feature point and the corresponding reference feature point, wherein the greater the distance, the greater the corresponding feature weight.
5. The method according to claim 1, characterized in that, Before determining the reference feature point corresponding to each feature point in the reference video frame corresponding to the video frame, the method further includes: During the process of annotating each feature point in the video frame, the first annotation order corresponding to each feature point is recorded; and during the process of annotating each reference feature point in the reference video frame, the second annotation order corresponding to each reference feature point is recorded. The step of determining the reference feature point corresponding to each feature point in the reference video frame corresponding to the video frame includes: For each feature point, the first annotation order corresponding to the feature point is determined as the target annotation order, and the reference feature points whose second annotation order is the same as the target annotation order are determined as the reference feature points corresponding to the feature points.
6. The method according to claim 1, characterized in that, After repairing the object region in the video frame based on each feature point and the target position corresponding to each feature point, the method further includes: Construct an object mask corresponding to the foreground object based on all the aforementioned feature points; The video frame and the repaired image obtained by repairing the object region based on the object mask are then synthesized.
7. A video restoration device, characterized in that, The device includes: An acquisition module is used to acquire the video to be repaired, wherein the video to be repaired includes multiple consecutive video frames containing the same foreground object; The annotation module is used to determine the object region corresponding to the foreground object in each video frame, and to perform feature annotation processing on the object region to obtain at least one corresponding feature point and the initial position of each feature point in the video frame. The determining module is used to determine a reference feature point corresponding to each feature point in a reference video frame corresponding to the video frame, wherein the reference video frame is the preceding video frame adjacent to the video frame. The correction module is used to correct the initial position of each feature point based on the reference feature point corresponding to each feature point, so as to obtain the target position of each feature point. The repair module is used to repair the object region in the video frame based on each feature point and the target position corresponding to each feature point; Specifically, the correction module is used for: Determine the feature weight corresponding to each feature point, and determine the distance between each feature point and the corresponding reference feature point; Based on the feature weights corresponding to each feature point and the distances corresponding to each feature point, the correction parameters are determined; For each feature point, the initial position corresponding to the feature point and the correction parameter are summed to obtain the target position.
8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.
Citation Information
Patent Citations
Video jitter processing method and device, equipment and storage medium
CN112929562A