Video processing method and device and electronic equipment

Abstract

The invention discloses a video processing method and device and electronic equipment, and belongs to the technical field of video processing. The method comprises the steps that a to-be-processed video is acquired, a shaded first element exists in the to-be-processed video, and a to-be-processed video frame comprises a first video frame and a second video frame; repairing a first area in the first video frame according to the second video frame, the first area being an area which is covered by the first element in the first video frame and is not covered by the first element in at least one second video frame; repairing a second area in the first video frame according to the background information in the first video frame, the second area being an area covered by the first element in the first video frame and the second video frame; and repairing the second video frame according to the repaired first video frame to obtain a target video.

Description

Technical Field

[0001] The present application belongs to the field of video processing technology, and specifically relates to a video processing method, device and electronic equipment. Background Art

[0002] Video element removal refers to removing specific elements from a video and repairing its background so that the viewer cannot detect the existence of the elements. Existing technologies are not effective in removing elements with complex backgrounds and large-area elements, which may cause the picture to be blurred, unnatural or incoherent, resulting in low video restoration quality. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a video processing method, device and electronic device, which can solve the technical problem of low restoration quality after element removal in existing videos.

[0004] In a first aspect, an embodiment of the present application provides a method for processing a video, the method comprising:

[0005] Acquire a video to be processed, wherein the video to be processed contains a masked first element, and the video frames to be processed include a first video frame and a second video frame;

[0006] Repairing a first area in the first video frame according to the second video frame, wherein the first area is covered by the first element in the first video frame, and there is at least one area in the second video frame that is not covered by the first element;

[0007] Repairing a second area in the first video frame according to the background information in the first video frame, wherein the second area is an area covered by the first element in both the first video frame and the second video frame;

[0008] The second video frame is repaired according to the repaired first video frame to obtain a target video.

[0009] In a second aspect, an embodiment of the present application provides a video processing device, the device comprising:

[0010] A first acquisition module is used to acquire a video to be processed, wherein the video to be processed contains a masked first element, and the video to be processed includes a first video frame and a second video frame;

[0011] A first repairing module, configured to repair a first area in the first video frame according to the second video frame, wherein the first area is covered by the first element in the first video frame, and there is at least one area in the second video frame that is not covered by the first element;

[0012] A second repairing module, configured to repair a second area in the first video frame according to background information in the first video frame, wherein the second area is an area covered by the first element in both the first video frame and the second video frame;

[0013] The third repairing module is used to repair the second video frame according to the repaired first video frame to obtain a target video.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method provided in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method provided in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method provided in the first aspect.

[0018] In the video processing method, device and electronic device of the present application, after the first element in each video frame in the mask video, the first video frame can be determined from multiple video frames, and then the area covered by the first element in the first video frame can be divided into a first area and a second area. Although the first area is covered by the first element in the first video frame, the background information of the uncovered first area can be obtained in other video frames, and the second area is the area covered by the first element in all video frames. Therefore, the first area can be repaired by combining the information of multiple video frames, and the second area can be repaired by the background information in the first video frame. In this way, the information of multiple frames in the video is fully utilized, the consistency and accuracy of the background repair are enhanced, and the quality of the video repair is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0020] Figure 2 is a structural schematic diagram of a video processing device provided by another embodiment of the present application;

[0021] Figure 3 is a structural schematic diagram of an electronic device provided by another embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0025] In order to solve the above technical problems, the present application provides a video processing method. The video processing method provided by the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown, Figure 1 : is a flow chart of a video processing method provided by an embodiment of the present application. The embodiment of the present application provides a video processing method, which may include:

[0027] S101, obtaining a video to be processed, wherein the video to be processed contains a masked first element, and the video frames to be processed include a first video frame and a second video frame;

[0028] In this embodiment, the video to be processed includes multiple video frames, and the multiple image frames can be divided into a first video frame and at least one second video frame. There is a masked first element in both the first image frame and the second video frame, wherein the masked first element is a foreground element in the video frame that is masked and marked as needing to be repaired.

[0029] In these video frames, the first element blocks the background area. After eliminating the first element, the background information needs to be used to repair the area where the first element is eliminated. In the repair process, the first video frame is the reference video frame for repair. The first video frame can be repaired first, and then the second video frame can be repaired based on the background information provided by the repaired first video frame to ensure the consistency of the background repair of the entire video.

[0030] In some embodiments, obtaining the video to be processed includes:

[0031] Get the original video;

[0032] Determine the first element in the original video;

[0033] Mask the first element in the original video to obtain the video to be processed.

[0034] In this embodiment, the original video refers to a complete video without any editing, processing or repair, including all original content and elements. After obtaining the original video, the element that the user wants to remove can be determined as the first element in the original video, and then the first element can be marked out from the original video by masking to form a video to be processed.

[0035] In this embodiment, this method can be used to accurately identify and process the unnecessary first element in the video, ensuring that the first element can be effectively removed in the subsequent processing process, providing a clear basis for further background restoration.

[0036] In some embodiments, determining the first element in the original video includes:

[0037] For any video frame in the original video, receiving a first input of the video frame by a user;

[0038] In response to the first input, the element corresponding to the first input is determined as the first element. In this embodiment, the original video can be played to the user through an interactive interface, and during the original video playback, the user can perform a first input on any video frame in the original video through the interactive interface.

[0039] Among them, the first input can be a touch input of the user to the electronic device, for example, the touch input can be a touch click operation, or a sliding operation, or other operations. The first input can also be an input operation performed by the user with the help of other input devices such as a mouse, a remote control device, a keyboard, etc. connected to the electronic device. Unless otherwise specified, the input to the electronic device in the following introduction can be a touch click operation, a sliding operation or other operations, and can also be an input operation performed by the user with the help of other input devices such as a mouse, a remote control device, a keyboard, etc. connected to the electronic device. In this embodiment, the first input can be an input for selecting the first element in the original video played in the electronic device.

[0040] In response to the first input, the system determines the element corresponding to the input position of the first input as the first element, and generates a mask at the position of the first element in the video frame of the first input. After determining the first element, the object segmentation model (SegmentAnything Model) can be used to accurately segment the area where the first element is located from the video frame, and the object tracking model (TrackingAnything) can be used to track the position of the first element in the video frame in the original video based on the segmented area, and also generate a corresponding mask. Among them, the input position of the first input can be multiple, so correspondingly, the first element can also be multiple.

[0041] In the above manner, the first element to be removed can be accurately selected in the entire video through the user's interaction with any video frame during video playback.

[0042] In some embodiments, the first input includes a first sub-input and the second sub-input, and in response to the first input, determining an element corresponding to the first input as the first element includes:

[0043] In response to the first sub-input, when the input position of the first sub-input corresponds to a plurality of elements, receiving a second sub-input for a second element among the plurality of elements;

[0044] In response to the second sub-input, the second element is determined to be the first element, and elements other than the first element among the plurality of elements are determined to be retained elements.

[0045] In this embodiment, the first input may include a first sub-input and a second sub-input. The user first performs the first sub-input on the third video frame, and the system determines the input position of the first sub-input and identifies the element corresponding to the input position of the first sub-input.

[0046] If the input position of the first sub-input corresponds to only one element, the element can be directly determined as the first element. If the input position of the first sub-input corresponds to multiple connected elements or multiple elements with a containment relationship, the user can further select a second element from the multiple elements and perform a second sub-input on the second element. In response to the second sub-input, the system can determine the second element corresponding to the second sub-input as the first element. After determining the first element, elements other than the first element in the multiple elements can be determined as retained elements that need to be retained.

[0047] In addition, when the input position of the first sub-input corresponds to multiple connected elements or multiple elements with a containment relationship, the user can also directly make a second input for the element that needs to be retained among the multiple elements. The system will then respond to the second input and determine the element corresponding to the second input as the retained element that needs to be retained.

[0048] In the above manner, the system can determine which elements in the video need to be removed and which need to be retained based on the user's interactive selection, thereby providing a more refined foundation for subsequent background restoration and video processing.

[0049] In some embodiments, masking the first element in the original video to obtain the video to be processed includes:

[0050] For each video frame in the original video, a mask of multiple elements in the video frame is generated to obtain the video to be processed; wherein the pixel value of the mask of the first element is a first pixel value, the pixel value of the mask of the retained element is a second pixel value, and the retained element is an element other than the first element among the multiple elements.

[0051] In this embodiment, after the user determines the first element to be removed and the retained element to be retained in the original video through the interactive interface, the system can generate a mask for each video frame in the original video, and the mask is a binary image used to identify the first element to be removed and the retained element to be retained in the video frame. In this way, each element in the video frame will be accurately marked, ensuring that different operations can be performed on different areas in the subsequent processing process.

[0052] For example, the mask of different regions can be set to different pixel values ​​to distinguish the retained elements and the first elements to be eliminated in each video frame. The pixel value of the mask of the first element can be set to 0, and the pixel value of the mask of the retained element can be set to 1.

[0053] In this embodiment, by generating a binary mask, the system can quickly identify the areas to be eliminated and retained, and maintain consistency in each frame, thereby making the process of repairing the background more flexible and accurate, thereby improving the quality and naturalness of video repair.

[0054] S102, repairing a first area in the first video frame according to the second video frame, wherein the first area is covered by the first element in the first video frame, and there is at least one area in the second video frame that is not covered by the first element;

[0055] In this embodiment, the first area is blocked by the first element in the first video frame, and the background information of the first area is lost due to being blocked by the foreground element. However, due to the deviation between the position of the first element and the blocked area in different video frames, there is at least one second video frame where the first area is not blocked by the first element. Then, this part of the second video frame can provide the background information of the first area. Then, the unblocked area in the second video frame can be used to repair the first area in the first video frame, and the background of this part can be restored by filling the background information of the second video frame into the blocked first area in the first video frame.

[0056] In some embodiments, before repairing the first area in the first video frame according to the second video frame, the method further includes:

[0057] Obtaining a transformation matrix sequence corresponding to the video to be processed, wherein the transformation matrix sequence includes a plurality of transformation matrices, each of which is used to represent a transformation relationship between two adjacent video frames in the video to be processed;

[0058] Aligning each video frame in the to-be-processed video according to the transformation matrix sequence, and aligning the mask of the first element in each video frame according to the transformation matrix sequence;

[0059] A video frame in which the mask of the first element occupies the smallest proportion of the video frame area in the aligned video frames is determined as the first video frame.

[0060] In this embodiment, before performing video restoration on the processed video, it is necessary to first align the video frames and align the masks of the first elements in each video frame to ensure the consistency of the background and avoid unnatural restoration effects due to differences between video frames.

[0061] Specifically, during the alignment process, a transformation matrix sequence may be first obtained, where the transformation matrix sequence represents a geometric transformation relationship between adjacent video frames.

[0062] For example, for any two adjacent video frames in the video to be processed, the feature points in the video frames can be extracted by using the scale-invariant feature transform (SIFT), and the wrong matching points can be eliminated by the RANSAC algorithm to calculate the transformation matrix (HomographyMatrix) between the two video frames to represent the geometric transformation relationship between the frames. Repeating this process can obtain a transformation matrix sequence containing multiple transformation matrices, which is used to describe the transformation relationship between adjacent frames in the video.

[0063] After obtaining the transformation matrix sequence, the transformation matrix sequence can be applied to align the video frames and align the mask of the first element in each frame. Next, the proportion of the mask of the first element in each video frame to the area of ​​the video frame is calculated, and the video frame with the smallest mask area ratio is selected as the first video frame. In this way, the first video frame is the video frame with the most background information retained in the video to be processed and the least area to be repaired.

[0064] In this way, by applying the transformation matrix sequence, the video frame and the corresponding mask can be accurately aligned to avoid the incoherent restoration problem caused by misalignment between frames. In addition, selecting the video frame with the smallest mask ratio of the first element as the first video frame of the benchmark can maximize the retention of reliable background information in the existing video and improve the naturalness and accuracy of video restoration.

[0065] In some embodiments, the repairing the first area in the first video frame according to the second video frame includes:

[0066] For any first pixel point in the first area of ​​the first video frame, determining a second pixel point in each of the second video frames whose position corresponds to the first pixel point;

[0067] Determine the second video frame in which the second pixel is not covered by the first element as the third video frame;

[0068] Determine a first mean and a first variance of pixel values ​​of a second pixel point in the third video frame;

[0069] A first pixel in the first video frame is repaired according to the first mean and the first variance.

[0070] In this embodiment, the first area can be repaired at the pixel level. Then, for any first pixel point in the first area of ​​the first video frame, the position of the first pixel point in the first video frame can be determined first, and then the corresponding second pixel point can be found in each second video frame.

[0071] For each second video frame, if the second pixel in the second video frame is not covered by the first element, the second video frame can be determined as a third video frame, and the third video frame is used to provide background information to repair the missing part in the first video frame.

[0072] After determining the third video frame, the first mean and the first variance of the pixel values ​​of the second pixel points in all the third video frames can be calculated, and the first pixel points in the first video frame can be repaired according to the first mean and the first variance. In this way, the repair of all the pixel points in the first region can be completed.

[0073] For example, the coordinates (x, y) of the first pixel point in the image coordinate system corresponding to the first video frame may be first determined. Then, correspondingly, the coordinates of all second pixel points in the image coordinate system corresponding to the second video frame are also (x, y). The first mean and the first variance may be calculated according to the following formula:

[0074]

[0075] Among them, μ is the first mean, σ 2 is the first variance, i is used to represent the sequence number of the second video frame in the video to be processed, A pixel value used to represent the mask of the second pixel point in the i-th second video frame, The pixel value of the second pixel in the i-th second video frame is used to represent the pixel value of the second pixel. In addition, in each second video frame, the pixel value of the mask of the first element is set to 0, and the pixel value of the mask of the retained element is set to 1. In this way, only the first mean and the first variance of the second pixel that is not masked by the first element can be calculated.

[0076] In this way, through the calculation formula of the first mean and the first variance, in the process of repairing each first pixel, the second pixel corresponding to the first pixel in each video frame and not covered by the first element can be screened out, and the first variance and the first mean of their pixel values ​​can be calculated. By repairing the first pixel using the first variance and the first mean, the occluded area of ​​the first element can be repaired using the background information in the context of the first video frame, so that the missing part in the video can be filled, ensuring the consistency of the background of each video frame in the video.

[0077] In some embodiments, the step of repairing the first pixel in the first video frame according to the first mean and the first variance includes:

[0078] Determine, according to the pixel value of each second pixel point in the third video frame, the first mean value, and the first variance, a degree of deviation between each second pixel point in the third video frame and the first mean value;

[0079] Determine the second pixel point whose deviation degree is less than the deviation threshold in the third video frame as the third pixel point;

[0080] Determine a second mean and a second variance of the pixel values ​​of a third pixel;

[0081] The first pixel in the first video frame is repaired according to the second mean and the second variance.

[0082] In this embodiment, in the process of determining the first mean and the first variance of the pixel value of the second pixel point in the third video frame, and repairing the first pixel point in the first video frame according to the first mean and the first variance, there may be some second pixel points in the third video frame that do not belong to the background but are misjudged as the background, and these second pixel points can be considered as outliers. These outliers will affect the repair effect, so it is necessary to remove these outliers through a certain algorithm.

[0083] Specifically, for each second pixel in the third video frame, the degree of deviation between the pixel value of each second pixel and the first mean value can be calculated. If the degree of deviation is less than a set deviation threshold, the second pixel is considered to belong to the background part, and is determined as the third pixel. If the degree of deviation is greater than or equal to the set deviation threshold, the second pixel is considered to be a noise point, does not belong to the background part, and therefore needs to be removed.

[0084] Therefore, only the second mean and the third mean of the pixel values ​​of all the third pixels may be calculated, and the first pixel in the first video frame may be repaired according to the second mean and the third mean.

[0085] For example, in the process of removing outliers, the z-score method can be used to remove outliers, that is, if the deviation of a second pixel exceeds a preset deviation threshold, it is considered an outlier and removed, and the new mean and variance are calculated iteratively until there are no obvious outliers. When there are no obvious outliers, the second mean and second variance of the pixel values ​​of the second pixel points that are not finally removed are calculated, and the obtained second mean and second variance are used to repair the first pixel point in the first video frame, thereby ensuring that the repaired background and context are consistent.

[0086] For example, in the process of removing outliers using the Z-score method, the Z-score calculation formula can be used to calculate the Z-score of the second pixel in each third video frame:

[0087] Wherein, Z is used to represent the standard deviation unit of the corresponding second pixel point from the mean. The calculation formula of Z score can be:

[0088]

[0089] in, is the first mean, σ is the standard deviation, The pixel value used to represent the second pixel point in the i-th second video frame.

[0090] After the Z score is calculated, the Z score can be compared with a set deviation threshold. The set deviation threshold can be 3. Then, the second pixel points with Z greater than 3 can be determined as outliers that need to be removed.

[0091] In this way, the second pixel point in the third video frame that is misjudged as the background can be eliminated to ensure that the restoration process only relies on real background information, avoiding unnatural or inaccurate background restoration.

[0092] S103: repairing a second area in the first video frame according to background information in the first video frame, wherein the second area is an area covered by the first element in both the first video frame and the second video frame.

[0093] In an embodiment of the present application, the second area is obscured by the first element in the first video frame. Since it is obscured by the foreground element, the background information of the second area is lost, and in the second video frame, the background of the second area is also obscured. Therefore, the first video frame cannot be repaired based on the second video frame. The background information of the part of the first video frame that is not obscured by the element can be analyzed to fill in the background content in the second area. Specifically, the background information of the first video frame can be used to make reasonable inferences to repair the second area.

[0094] Since the second model cannot be directly repaired based on the content in the second video frame, a generative diffusion model can be used to generate background content in the second region to fill in the missing part after eliminating the first element. For example, the generative diffusion model can be a Stable Diffusion model or a Flux model.

[0095] The generative diffusion model generates detailed textures through a process of adding noise and denoising. First, the generative diffusion model adds noise to the area to be repaired, and then gradually denoises it until clear content that matches the background is generated. This process can generate image content that matches the surrounding environment.

[0096] Specifically, the first video frame to be repaired and the corresponding mask can be input into the generative diffusion model together. The mask is a binary image in which the second area to be repaired is identified. Then the variational autoencoder (VAE) in the generative diffusion model can convert the input first video frame into a latent space representation. The latent space representation is a compressed form of the image, which retains the key features of the image. Subsequently, the generative diffusion model can gradually restore the noisy image to a complete image and generate new background content in the masked area. In addition, the generative diffusion model will guide the generation process according to the mask information to ensure that the generated content only appears in the area marked by the mask without affecting the original pixels in the non-masked area. Since the model has learned a large number of image patterns during training, it can generate reasonable content to fill the masked area so that it is consistent with the surrounding environment. After the generation is completed, the model will decode the repair content in the latent space back to the original image space through VAE to obtain the final repaired first video frame.

[0097] S104: Repair the second video frame according to the repaired first video frame to obtain a target video.

[0098] In the present application, after the first video frame is repaired, the repaired background information and image details are filled into the area covered by the first element in the first video frame. Therefore, the first video frame has relatively complete background information. Then, the second video frame can be repaired using the repaired background information of the first video frame. In this way, the area blocked by the first element in the second video frame can also be supplemented, thereby completing the repair of the second video frame and further obtaining the repaired target video.

[0099] For example, before the first video frame is repaired, the pixel value of the mask of the first element is the first pixel value, and the pixel value of the mask of the retained element is the second pixel value, wherein the first pixel value means that the image of the area needs to be repaired, and the second pixel value means that the image of the area does not need to be repaired. After the first video frame is repaired, the pixel value of the mask of the repaired first element can also be adjusted to the second pixel value, indicating that these areas do not need to be repaired either.

[0100] Next, the restored first video frame and the corresponding mask are input into the video restoration algorithm. Since most of the background information in the video can be provided by the restored first video frame, and the mask clearly identifies the area that needs to be restored, the video restoration algorithm can use similar backgrounds from other video frames to further restore the area in the second video frame where the first element was removed.

[0101] For example, the ProPainter algorithm can be used to repair the second video frame, that is, after inserting the repaired first video frame into the video to be processed, the optical flow between the video frames is first calculated by the RAFT convolutional neural network. The optical flow is used to capture the motion information between frames, and then the calculated optical flow field is repaired through the recurrent flow network to ensure that accurate optical flow information is used to guide the repair steps during the repair process. Finally, the neural network is used to perform information propagation between the feature domain and the image domain to ensure that the background information extracted from the surrounding frames can correctly fill the masked area of ​​the first element in the second video frame.

[0102] In this way, the video restoration algorithm successfully removes the first element in the video to be processed by completing the mask area of ​​the first element and restores the background, so that the removed part is naturally integrated into the video, generating a restored target video.

[0103] In an embodiment of the present application, after the first element in each video frame in the masked video, the first video frame can be determined from multiple video frames, and then the area covered by the first element in the first video frame can be divided into a first area and a second area. Although the first area is covered by the first element in the first video frame, the background information of the uncovered first area can be obtained in other video frames, and the second area is the area covered by the first element in all video frames. Therefore, the first area can be repaired by combining the information of multiple video frames, and the second area can be repaired by the background information in the first video frame. In this way, the information of multiple frames in the video is fully utilized, the consistency and accuracy of the background repair are enhanced, and the quality of the video repair is significantly improved.

[0104] In some embodiments, the repairing the second video frame according to the repaired first video frame to obtain a target video includes:

[0105] The area covered by the first element in the second video frame is repaired according to the repaired first video frame to obtain the target video.

[0106] In this embodiment, the restoration has been completed in the first video frame, that is, the first element in the mask area has been eliminated and the background of the area has been restored, and the first video frame has relatively complete background information. Then, the background information in the restored first video frame can be further used to restore the area covered by the first element in each second video frame.

[0107] In this way, the restored background information of the first video frame can be directly used to restore the second video frame, thereby completing efficient video restoration.

[0108] In some embodiments, the repairing the second video frame according to the repaired first video frame to obtain a target video includes:

[0109] For any second video frame, repair a third area in the second video frame according to other video frames other than the second video frame, wherein the third area is an area in the second video frame that is covered by the first element, and there is at least one area in the video frame that is not covered by the first element;

[0110] A fourth area in the second video frame is repaired according to the background information in the first video frame, wherein the fourth area is an area covered by the first element in both the first video frame and the second video frame.

[0111] In this embodiment, the area blocked by the first element in the second video frame can be divided into a third area and a fourth area. Among them, the third area is blocked by the first element in the second video frame, and the background information of the third area is lost due to being blocked by the foreground element. Since there are deviations in the position of the first element and the blocked area in different video frames, there is at least one video frame in the video to be processed in which the third area is not blocked by the first element. Then, this part of the video frames can provide background information of the third area. Then, the unblocked areas in these video frames can be used to repair the third area in the second video frame, and the background of this part is restored by filling the background information of the second video frame into the blocked third area in the second video frame.

[0112] The fourth region refers to a region that is blocked by the first element in all video frames of the video to be processed. Since this region is blocked in all video frames, background information cannot be directly obtained through other video frames. At this time, because the background of the first video frame has been repaired, the fourth region in the second video frame can be repaired based on the background information in the repaired first video frame.

[0113] In this way, the background information of multiple video frames can be comprehensively utilized to improve the accuracy and naturalness of the restoration of the second video frame.

[0114] Figure 2 is a structural diagram of a video processing device provided by another embodiment of the present application, such as Figure 2 As shown, the video processing device may include:

[0115] A first acquisition module 201 is used to acquire a video to be processed, wherein the video to be processed contains a masked first element, and the video to be processed includes a first video frame and a second video frame;

[0116] A first repairing module 202, configured to repair a first area in the first video frame according to the second video frame, wherein the first area is covered by the first element in the first video frame, and there is at least one area in the second video frame that is not covered by the first element;

[0117] A second repairing module 203, configured to repair a second area in the first video frame according to background information in the first video frame, wherein the second area is an area covered by the first element in both the first video frame and the second video frame;

[0118] The third repairing module 204 is used to repair the second video frame according to the repaired first video frame to obtain a target video.

[0119] In the present application, after the first element in each video frame in the masked video, the first video frame can be determined from multiple video frames, and then the area covered by the first element in the first video frame can be divided into a first area and a second area. Although the first area is covered by the first element in the first video frame, the background information of the uncovered first area can be obtained in other video frames, and the second area is the area covered by the first element in all video frames. Therefore, the first area can be repaired by combining the information of multiple video frames, and the second area can be repaired by the background information in the first video frame. In this way, the information of multiple frames in the video is fully utilized, the consistency and accuracy of the background repair are enhanced, and the quality of the video repair is significantly improved.

[0120] In another optional example, the first acquisition module 201 includes:

[0121] A first acquisition unit, used to acquire an original video;

[0122] A first determining unit, configured to determine a first element in the original video;

[0123] The first masking unit is used to mask the first element in the original video to obtain the video to be processed.

[0124] In another optional example, the first determining unit includes:

[0125] A first receiving subunit, configured to receive a first input of a user on any video frame in the original video;

[0126] The first determining subunit is configured to, in response to the first input, determine an element corresponding to the first input as the first element.

[0127] In another optional example, the first masking unit further includes:

[0128] The mask subunit is used to generate a mask of the multiple elements in each video frame in the original video to obtain the video to be processed; wherein the pixel value of the mask of the first element is a first pixel value, and the pixel value of the mask of the retained element is a second pixel value.

[0129] In another optional example, the video processing device may further include:

[0130] A second acquisition module is used to acquire a transformation matrix sequence corresponding to the video to be processed, wherein the transformation matrix sequence includes a plurality of transformation matrices, and each of the transformation matrices is used to represent a transformation relationship between two adjacent video frames in the video to be processed;

[0131] an alignment module, configured to align the video frames in the to-be-processed video according to the transformation matrix sequence, and to align the masks of the first elements in the video frames according to the transformation matrix sequence;

[0132] The first determination module is configured to determine, among the aligned video frames, a video frame in which the mask of the first element occupies the smallest proportion of the video frame area as the first video frame.

[0133] In another optional example, the first repair module 202 further includes:

[0134] A second determining unit, configured to determine, for any first pixel point in the first area of ​​the first video frame, a second pixel point whose position corresponds to the first pixel point in each of the second video frames;

[0135] A third determining unit, configured to determine the second video frame in which the second pixel is not covered by the first element as the third video frame;

[0136] a fourth determining unit, configured to determine a first mean and a first variance of a pixel value of a second pixel point in the third video frame;

[0137] A first restoration unit is configured to restore a first pixel in the first video frame according to the first mean and the first variance.

[0138] In another optional example, the third repair module 204 includes:

[0139] The second repairing unit is used to repair the area covered by the first element in the second video frame according to the repaired first video frame to obtain the target video.

[0140] In another optional example, the third repair module 204 further includes:

[0141] a third repairing unit, configured to repair, for any second video frame, a third area in the second video frame according to other video frames other than the second video frame, wherein the third area is an area in the second video frame that is covered by the first element and there is at least one area in the video frame that is not covered by the first element;

[0142] A fourth repairing unit is used to repair a fourth area in the second video frame according to the background information in the first video frame, wherein the fourth area is an area covered by the first element in both the first video frame and the second video frame.

[0143] The video processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0144] The video processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0145] The video processing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0146] Alternatively, if Figure 3 As shown, an embodiment of the present application also provides an electronic device 100, including a processor 110, a memory 119, and a program or instruction stored in the memory 119 and executable on the processor 110. When the program or instruction is executed by the processor 110, each process of the above-mentioned video processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0147] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0148] Please refer to Figure 4 , Figure 4 The hardware structure diagram of an electronic device for implementing an embodiment of the present application is shown in FIG. The electronic device 100 includes, but is not limited to, a radio frequency unit 121, a network module 122, an audio output unit 123, an input unit 124, a sensor 125, a display unit 126, a user input unit 127, an interface unit 128, a memory 129, and a processor 120.

[0149] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 120 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0150] The input unit 124 is used to obtain a video to be processed, wherein the video to be processed contains a masked first element, and the video frame to be processed includes a first video frame and a second video frame;

[0151] The processor 120 is configured to repair a first area in the first video frame according to the second video frame, wherein the first area is covered by the first element in the first video frame, and there is at least one area in the second video frame that is not covered by the first element;

[0152] The processor 120 is configured to repair a second area in the first video frame according to the background information in the first video frame, wherein the second area is an area covered by the first element in both the first video frame and the second video frame;

[0153] The processor 120 is configured to repair the second video frame according to the repaired first video frame to obtain a target video.

[0154] In the present application, after the first element in each video frame in the masked video, the first video frame can be determined from multiple video frames, and then the area covered by the first element in the first video frame can be divided into a first area and a second area. Although the first area is covered by the first element in the first video frame, the background information of the uncovered first area can be obtained in other video frames, and the second area is the area covered by the first element in all video frames. Therefore, the first area can be repaired by combining the information of multiple video frames, and the second area can be repaired by the background information in the first video frame. In this way, the information of multiple frames in the video is fully utilized, the consistency and accuracy of the background repair are enhanced, and the quality of the video repair is significantly improved.

[0155] In another optional example, the input unit 124 is used to obtain the original video;

[0156] The processor 120 is configured to determine a first element in the original video;

[0157] The processor 120 is used to mask the first element in the original video to obtain the video to be processed.

[0158] In another optional example, the user input unit 127 is used to receive a first input of a user on any video frame in the original video;

[0159] The processor 120 is configured to, in response to the first input, determine an element corresponding to the first input as the first element.

[0160] In another optional example, the processor 120 is also used to generate, for each video frame in the original video, masks of multiple elements in the video frame to obtain the video to be processed; wherein the pixel value of the mask of the first element is a first pixel value, the pixel value of the mask of the retained element is a second pixel value, and the retained element is an element other than the first element among the multiple elements.

[0161] In another optional example, the processor 120 is further used to obtain a transformation matrix sequence corresponding to the video to be processed, wherein the transformation matrix sequence includes a plurality of transformation matrices, and each of the transformation matrices is used to represent a transformation relationship between two adjacent video frames in the video to be processed;

[0162] Aligning each video frame in the to-be-processed video according to the transformation matrix sequence, and aligning the mask of the first element in each video frame according to the transformation matrix sequence;

[0163] A video frame in which the mask of the first element occupies the smallest proportion of the video frame area in the aligned video frames is determined as the first video frame.

[0164] In another optional example, the processor 120 is further configured to, for any first pixel in the first area of ​​the first video frame, determine, in each of the second video frames, a second pixel whose position corresponds to the first pixel;

[0165] Determine the second video frame in which the second pixel is not covered by the first element as the third video frame;

[0166] Determine a first mean and a first variance of pixel values ​​of a second pixel point in the third video frame;

[0167] A first pixel in the first video frame is repaired according to the first mean and the first variance.

[0168] In another optional example, the processor 120 is further configured to:

[0169] Determine, according to the pixel value of each second pixel point in the third video frame, the first mean value, and the first variance, a degree of deviation between each second pixel point in the third video frame and the first mean value;

[0170] Determine the second pixel point whose deviation degree is less than the deviation threshold in the third video frame as the third pixel point;

[0171] Determine a second mean and a second variance of the pixel values ​​of a third pixel;

[0172] The first pixel in the first video frame is repaired according to the second mean and the second variance.

[0173] In another optional example, the processor 120 is further configured to: repair the area covered by the first element in the second video frame according to the repaired first video frame to obtain the target video.

[0174] In another optional example, the processor 120 is further configured to: for any second video frame, repair a third area in the second video frame according to other video frames other than the second video frame, wherein the third area is an area in the second video frame that is covered by the first element, and there is at least one area in the video frame that is not covered by the first element;

[0175] A fourth area in the second video frame is repaired according to the background information in the first video frame, wherein the fourth area is an area covered by the first element in both the first video frame and the second video frame.

[0176] It should be understood that in the embodiment of the present application, the input unit 124 may include a graphics processor (Graphics Processing Unit, GPU) 1241 and a microphone 1242, and the graphics processor 1241 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 126 may include a display panel 1261, and the display panel 1261 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 127 includes a touch panel 1271 and at least one of other input devices 1272. The touch panel 1271 is also called a touch screen. The touch panel 1271 may include two parts: a touch detection device and a touch controller. Other input devices 1272 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0177] The memory 129 can be used to store software programs and various data. The memory 129 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 129 may include a volatile memory or a non-volatile memory, or the memory 129 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 129 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0178] The processor 120 may include one or more processing units; optionally, the processor 120 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 120.

[0179] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0180] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0181] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0182] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0183] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0184] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0186] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video processing method, characterized in that: include: Acquire a video to be processed, wherein the video to be processed contains a masked first element, and the video frames to be processed include a first video frame and a second video frame; Repairing a first area in the first video frame according to the second video frame, wherein the first area is covered by the first element in the first video frame, and there is at least one area in the second video frame that is not covered by the first element; Repairing a second area in the first video frame according to the background information in the first video frame, wherein the second area is an area covered by the first element in both the first video frame and the second video frame; The second video frame is repaired according to the repaired first video frame to obtain a target video.

2. The method according to claim 1, characterized in that The step of obtaining the video to be processed includes: Get the original video; Determining a first element in the original video; Mask the first element in the original video to obtain the video to be processed.

3. The method according to claim 2, characterized in that The determining the first element in the original video includes: For any video frame in the original video, receiving a first input of the video frame by a user; In response to the first input, an element corresponding to the first input is determined as the first element.

4. The method according to claim 2, characterized in that: The masking the first element in the original video to obtain the video to be processed includes: For each video frame in the original video, a mask of multiple elements in the video frame is generated to obtain the video to be processed; wherein the pixel value of the mask of the first element is a first pixel value, the pixel value of the mask of the retained element is a second pixel value, and the retained element is an element other than the first element among the multiple elements.

5. The method according to claim 1, characterized in that Before repairing the first area in the first video frame according to the second video frame, the method further includes: Obtaining a transformation matrix sequence corresponding to the video to be processed, wherein the transformation matrix sequence includes a plurality of transformation matrices, each of which is used to represent a transformation relationship between two adjacent video frames in the video to be processed; Aligning each video frame in the to-be-processed video according to the transformation matrix sequence, and aligning the mask of the first element in each video frame according to the transformation matrix sequence; A video frame in which the mask of the first element occupies the smallest proportion of the video frame area in the aligned video frames is determined as the first video frame.

6. The method according to claim 1, characterized in that The repairing the first area in the first video frame according to the second video frame includes: For any first pixel point in the first area of ​​the first video frame, determining a second pixel point in each of the second video frames whose position corresponds to the first pixel point; Determine the second video frame in which the second pixel is not covered by the first element as the third video frame; Determine a first mean and a first variance of pixel values ​​of a second pixel point in the third video frame; A first pixel in the first video frame is repaired according to the first mean and the first variance.

7. The method according to claim 1, characterized in that The step of repairing the second video frame according to the repaired first video frame to obtain a target video includes: The area covered by the first element in the second video frame is repaired according to the repaired first video frame to obtain the target video.

8. The method according to claim 1, characterized in that The step of repairing the second video frame according to the repaired first video frame to obtain a target video includes: For any second video frame, repair a third area in the second video frame according to other video frames other than the second video frame, wherein the third area is an area in the second video frame that is covered by the first element, and there is at least one area in the video frame that is not covered by the first element; A fourth area in the second video frame is repaired according to the background information in the first video frame, wherein the fourth area is an area covered by the first element in both the first video frame and the second video frame.

9. A video processing device, characterized in that: include: A first acquisition module is used to acquire a video to be processed, wherein the video to be processed contains a masked first element, and the video to be processed includes a first video frame and a second video frame; A first repairing module, configured to repair a first area in the first video frame according to the second video frame, wherein the first area is covered by the first element in the first video frame, and there is at least one area in the second video frame that is not covered by the first element; A second repairing module, configured to repair a second area in the first video frame according to background information in the first video frame, wherein the second area is an area covered by the first element in both the first video frame and the second video frame; The third repairing module is used to repair the second video frame according to the repaired first video frame to obtain a target video.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the video processing method according to any one of claims 1 to 8 are implemented.

Images