Video object erasing method, device and equipment and readable storage medium

Abstract

The invention discloses a video object erasing method, apparatus and device, and a computer readable storage medium. The method comprises the steps of obtaining a first original video containing a to-be-erased video object; selecting a to-be-erased video object from the first original video according to a received object selection instruction; performing mask processing on the to-be-erased video object to obtain an object mask; performing object erasing on the original video according to a video point tracking result corresponding to the video object to be erased and an object mask by utilizing each Transform layer obtained by performing mask supervised learning including the object to be erased and side effects through a mask decoder in the target video object erasing model; each Transform layer includes a point tracking guided attention module that divides points in the first original video into foreground points and background points. According to the method, the video object to be erased and the side effect can be erased together, so that the erasing efficiency and the erasing accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device, and computer-readable storage medium for erasing video objects. Background Technology

[0002] In film and television post-production, removing continuity errors is a crucial step before the final product is made. It ensures the quality of the work while improving production efficiency.

[0003] In traditional film and television post-production workflows, handling continuity errors mainly relies on manual operation or basic image processing techniques. When manually handling continuity errors, post-production staff need to meticulously process each frame containing the error, which is not only extremely inefficient but also a huge test of the professional skills and patience of the post-production staff, resulting in high labor costs.

[0004] While early image processing techniques, such as simple image content cloning and content-aware fill, could handle some simple static continuity errors to a certain extent, such as small objects appearing in a fixed position in the frame, they fell short when it came to complex dynamic continuity errors. In dynamic scenes, objects are constantly in motion, and the surrounding environment of the error is also constantly changing. Traditional techniques struggle to accurately track the movement trajectory of the error, making it impossible to achieve a smooth and natural restoration effect.

[0005] With the development of generative artificial intelligence (AIGC) technology, current erasing techniques can support coherent and natural erasing effects on objects in moving scenes. However, when erasing objects, these methods often only focus on the object itself and emphasize erasing the object, while ignoring the side effects caused by processing the object, such as shadows and reflections. These side effects are part of the content that will be exposed. Although the side effects caused by the erased object can be eliminated by two or more erasing during manual verification, it affects the overall efficiency. If the verification is not careful, it is easy to overlook these side effects.

[0006] In summary, how to effectively address the side effects of current video object erasure methods that require two or more erasures to eliminate the erased object, thus affecting overall efficiency, and how these side effects can easily be overlooked if verification is not thorough, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] The purpose of this application is to provide a video object erasing method that not only erases the selected object, but also erases the side effects that the object causes to the video, thereby improving erasing efficiency and accuracy. Another purpose of this application is to provide a video object erasing apparatus, device, and computer-readable storage medium.

[0008] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0009] A method for erasing video objects includes:

[0010] Get the first raw video containing the video object to be erased;

[0011] The video object to be erased is selected from the first original video according to the received object selection instruction;

[0012] The video object to be erased is masked to obtain an object mask;

[0013] Using the Transformer layers obtained by mask-supervised learning (containing the object to be erased and side effects) through a mask decoder in the target video object erasure model, the original video is erased based on the video point tracking results corresponding to the video object to be erased and the object mask.

[0014] Each Transformer layer includes an attention module that guides point tracking by dividing points in the first original video into foreground and background points based on the video point tracking results and the object mask; the foreground points are the points that constitute the video object to be erased, and the background points are the other points in the first original video besides the points that constitute the video object to be erased.

[0015] In one specific embodiment of this application, a training process for the target video object erasure model is also included, the training process of the target video object erasure model comprising:

[0016] Obtain the mask of the object to be erased, the second original video, the real video after erasure, and the truth mask containing the object to be erased and side effects;

[0017] Each variational autoencoder is used to perform feature compression on the mask of the object to be erased, the second original video, and the real video after erasure, respectively, to obtain each latent space feature;

[0018] The features of each latent space are concatenated to obtain the first concatenated feature;

[0019] The video point tracking results between adjacent video frame sequences are determined based on the second original video.

[0020] The attention module, guided by point tracking within each Transformer layer of the initial video object erasure model, divides the points in the second original video into foreground points and background points based on the video point tracking results and the mask of the object to be erased.

[0021] Adjust the spatiotemporal attention mechanism of each Transformer layer by using each foreground point and each background point;

[0022] The first concatenated feature is processed layer by layer using each Transformer layer to obtain the features of each layer;

[0023] The diffusion loss is calculated based on the erased real video and the output features of the last Transformer layer.

[0024] Shallow features, middle features, and high-level features are selected from the features of each layer, and the shallow features, middle features, and high-level features are concatenated to obtain the second concatenated feature;

[0025] The second splicing feature is processed using the mask decoder in the initial video object erasure model to output a real mask containing the object to be erased and side effects;

[0026] Calculate the mean squared error loss based on the true mask and the ground truth mask;

[0027] The initial video object erasure model is iteratively trained based on the diffusion loss and the mean square error loss to obtain the target video object erasure model.

[0028] In one specific embodiment of this application, the spatiotemporal attention mechanism of each Transformer layer is adjusted by each foreground point and each background point, including:

[0029] Obtain the temporal and spatial attention corresponding to each background point;

[0030] The mean of attention at each time point is calculated to obtain the mean of background time attention.

[0031] The time attention scores between each foreground point and each background point are adjusted based on the average time attention value of the background points.

[0032] The mean value of spatial attention for each spatial attention point is calculated to obtain the mean value of spatial attention for the background point.

[0033] The spatial attention scores between each foreground point and each background point are adjusted based on the mean spatial attention of the background points.

[0034] In one specific embodiment of this application, the temporal attention score between each foreground point and each background point is corrected based on the average temporal attention value of the background points, including:

[0035] The average temporal attention of the background points is used as the temporal attention between each foreground point and each background point.

[0036] In one specific embodiment of this application, the mean of attention at each time point is calculated, including:

[0037] Obtain the time weight factors corresponding to each background point;

[0038] The mean of attention at each time point is calculated based on the weighting factors of each time point.

[0039] In one specific embodiment of this application, the spatial attention score between each foreground point and each background point is corrected based on the mean spatial attention value of the background points, including:

[0040] The average spatial attention of the background points is used as the spatial attention between each foreground point and each background point.

[0041] In one specific embodiment of this application, the mean value of each spatial attention is calculated, including:

[0042] Obtain the spatial weight factors corresponding to each background point;

[0043] The mean of attention in each space is calculated based on the weighting factors of each space.

[0044] A video object erasing device, comprising:

[0045] The first original video acquisition module is used to acquire the first original video containing the video object to be erased.

[0046] The video object to be erased selection module is used to select the video object to be erased from the first original video according to the received object selection instruction;

[0047] The object mask acquisition module is used to perform mask processing on the video object to be erased to obtain the object mask;

[0048] The object erasure module is used to use each Transformer layer obtained by mask-supervised learning, which includes the object to be erased and side effects, through a mask decoder in the target video object erasure model, to erase the object in the original video based on the video point tracking results corresponding to the video object to be erased and the object mask.

[0049] Each Transformer layer includes an attention module that guides point tracking by dividing points in the first original video into foreground and background points based on the video point tracking results and the object mask; the foreground points are the points that constitute the video object to be erased, and the background points are the other points in the first original video besides the points that constitute the video object to be erased.

[0050] A video object erasing device, comprising:

[0051] Memory, used to store computer programs;

[0052] A processor for executing the computer program to implement the steps of the video object erasure method as described above.

[0053] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the video object erasure method described above.

[0054] The video object erasure method provided in this application includes: acquiring a first original video containing a video object to be erased; selecting a video object to be erased from the first original video according to a received object selection instruction; performing masking processing on the video object to be erased to obtain an object mask; and using each Transformer layer obtained by mask-supervised learning containing the object to be erased and side effects through a mask decoder in the target video object erasure model to erase the original video based on the video point tracking results corresponding to the video object to be erased and the object mask; wherein each Transformer layer includes a point tracking-guided attention module that divides the points in the first original video into foreground points and background points based on the video point tracking results and the object mask; the foreground points are the points constituting the video object to be erased, and the background points are other points in the first original video besides the points constituting the video object to be erased.

[0055] As can be seen from the above technical solution, by setting a point tracking-guided attention module in each Transformer layer of the target video object erasure model, the points in the first original video are divided into foreground and background points using the point tracking-guided attention module. Video point tracking technology guides the calculation of spatiotemporal attention, thus erasing the video object to be erased and its side effects more smoothly and accurately. By introducing a mask reconstruction mechanism, the difference between the reconstructed mask and the ground truth mask is used as a loss signal to supervise the model training, guiding the model to discover features that can represent the video object to be erased and its side effects, thereby achieving accurate erasure. When erasing objects in post-production, not only are the selected objects erased, but the side effects they bring to the video are also erased, improving erasure efficiency and accuracy.

[0056] Accordingly, this application also provides a video object erasing apparatus, device, and computer-readable storage medium corresponding to the above-mentioned video object erasing method, which have the above-mentioned technical effects, and will not be described in detail here. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating one implementation of the video object erasure method in this application.

[0059] Figure 2 This is a flowchart illustrating another implementation of the video object erasure method in this application.

[0060] Figure 3 This is a schematic diagram illustrating the training process of a video object erasure model in an embodiment of this application;

[0061] Figure 4 This is a structural block diagram of a video object erasing device according to an embodiment of this application;

[0062] Figure 5 This is a structural block diagram of a video object erasing device according to an embodiment of this application;

[0063] Figure 6 This is a schematic diagram of the specific structure of a video object erasing device provided in an embodiment of this application. Detailed Implementation

[0064] From an artistic quality perspective, post-production erasure in film and television maintains the realism and consistency of the work's historical context. Even a minor continuity error (such as an inappropriate modern object, crew member, or equipment) can easily detach the audience from the story, damaging the work's rigor and credibility. Precisely erasing these errors ensures the integrity and consistency of the narrative, thereby enhancing the work's artistic merit and professional reputation. From a production efficiency and cost control perspective, post-production erasure serves as a safety net for the final product. Film sets are constantly changing, and many continuity errors are difficult to completely avoid in pre-production. Relying on returning to the set, rebuilding sets, and reshooting with actors would incur enormous time and financial costs. Post-production erasure technology offers a highly cost-effective solution, efficiently resolving these issues in post-production at a significantly lower cost than reshooting. This not only directly saves on the budget but also relieves pressure on the pre-production team, allowing them to focus more on performances and capturing key shots, thus optimizing the efficiency of the entire production process.

[0065] In conclusion, post-production erasing in film and television is not merely a simple "repair of flaws," but rather an efficient and economical way to achieve final control and enhancement of the final film's quality through post-production techniques. It is an indispensable part of the modern film and television industry process.

[0066] Current video object erasure methods often focus only on the object itself when erasing it, emphasizing the removal of the object while ignoring the side effects caused by the processing, such as shadows and reflections. These side effects are part of the content that will be exposed. Although the side effects caused by the erased object can be eliminated by two or more erasures during manual verification, it affects the overall efficiency. If the verification is not careful, it is easy to overlook these side effects.

[0067] Therefore, the video object erasure method provided in this application not only erases the selected object, but also erases the side effects that the object brings to the video, thereby improving erasure efficiency and accuracy.

[0068] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0070] See Figure 1 , Figure 1 This is a flowchart illustrating one implementation of the video object erasure method in this application. The method may include the following steps:

[0071] S101: Obtain the first raw video containing the video object to be erased.

[0072] When it is necessary to erase the video object to be erased in the video, the video object erasure processing center obtains the first original video containing the video object to be erased.

[0073] The video object erasure processing center can be a server, processor, etc., that is equipped with a target video object erasure model for automatically erasing the video objects to be erased.

[0074] S102: Select the video object to be erased from the first original video according to the received object selection instruction.

[0075] After acquiring the first original video containing the video object to be erased, the user client can send an object selection command to the video object erasure processing center. This command includes information used to identify the video object to be erased. The video object erasure processing center then selects the video object to be erased from the first original video based on the received object selection command.

[0076] S103: Perform masking on the video object to be erased to obtain the object mask.

[0077] After selecting the video object to be erased from the first original video according to the received object selection instruction, the video object to be erased is masked to obtain the object mask.

[0078] S104: Using the Transformer layers obtained by mask-supervised learning (containing the object to be erased and side effects) through a mask decoder in the target video object erasure model, the original video is erased based on the video point tracking results corresponding to the video object to be erased and the object mask.

[0079] Each Transformer layer includes an attention module that divides the points in the first original video into foreground and background points based on the video point tracking results and object masks. Foreground points are the points that constitute the video object to be erased, and background points are the other points in the first original video besides those that constitute the video object to be erased.

[0080] The trained target video object erasure model contains multiple Transformer layers, each including a point tracking-guided attention module. This module divides the points in the first original video into foreground and background points based on the video point tracking results and the object mask. Foreground points are the points that constitute the video object to be erased, while background points are all other points in the first original video besides those constituting the video object.

[0081] After masking the video object to be erased to obtain the object mask, the Transformer layers obtained through mask decoder-supervised learning of the mask containing the object to be erased and its side effects are used in the target video object erasure model. Based on the video point tracking results corresponding to the video object to be erased and the object mask, the original video is erased. Video point tracking technology guides the computation of spatiotemporal attention, thus erasing the video object to be erased and its side effects more smoothly and accurately. By introducing a mask reconstruction mechanism, the difference between the reconstructed mask and the ground truth mask is used as a loss signal to supervise the model training, guiding the model to mine features that can represent the video object to be erased and its side effects, thereby achieving accurate erasure.

[0082] As can be seen from the above technical solution, by setting a point tracking-guided attention module in each Transformer layer of the target video object erasure model, the points in the first original video are divided into foreground and background points using the point tracking-guided attention module. Video point tracking technology guides the calculation of spatiotemporal attention, thus erasing the video object to be erased and its side effects more smoothly and accurately. By introducing a mask reconstruction mechanism, the difference between the reconstructed mask and the ground truth mask is used as a loss signal to supervise the model training, guiding the model to discover features that can represent the video object to be erased and its side effects, thereby achieving accurate erasure. When erasing objects in post-production, not only are the selected objects erased, but the side effects they bring to the video are also erased, improving erasure efficiency and accuracy.

[0083] It should be noted that, based on the above embodiments, this application also provides corresponding improvement solutions. In subsequent embodiments, steps that are the same as or corresponding to those in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other. These improvements will not be elaborated upon in the following improved embodiments.

[0084] See Figure 2 , Figure 2 This is a flowchart illustrating another implementation of the video object erasure method in this application. The method may include the following steps:

[0085] S201: Obtain the mask of the object to be erased, the second original video, the real video after erasure, and the truth mask containing the object to be erased and its side effects.

[0086] When training the video object erasure model, we obtain the mask of the object to be erased, the second original video, the real video after erasure, and the ground truth mask containing the object to be erased and side effects.

[0087] S202: Use each variational autoencoder to perform feature compression on the mask of the object to be erased, the second original video, and the real video after erasure, to obtain each latent space feature.

[0088] After obtaining the mask of the object to be erased, the second original video, and the real video after erasure, each variational autoencoder (VAE) is used to compress features of the mask of the object to be erased, the second original video, and the real video after erasure to obtain the latent space features.

[0089] S203: Concatenate the features of each latent space to obtain the first concatenated feature.

[0090] After using each variational autoencoder to compress features from the mask of the object to be erased, the second original video, and the real video after erasure, and obtaining each latent space feature, the latent space features are concatenated to obtain the first concatenated feature.

[0091] S204: Determine the video point tracking results between adjacent video frame sequences based on the second original video.

[0092] After acquiring the second original video, the video point tracking results between adjacent video frame sequences are determined based on the second original video.

[0093] S205: The attention module, guided by point tracking in each Transformer layer of the initial video object erasure model, divides the points in the second original video into foreground points and background points based on the video point tracking results and the mask of the object to be erased.

[0094] After obtaining the mask of the object to be erased and determining the video point tracking results between adjacent video frame sequences based on the second original video, the attention module guided by point tracking in each Transformer layer of the initial video object erasure model divides the points in the second original video into foreground points and background points based on the video point tracking results and the mask of the object to be erased.

[0095] S206: Adjust the spatiotemporal attention mechanism of each Transformer layer by using each foreground point and each background point.

[0096] After the attention module guided by point tracking in each Transformer layer of the initial video object erasure model divides the points in the second original video into foreground and background points based on the video point tracking results and the mask of the object to be erased, the spatiotemporal attention mechanism of each Transformer layer is adjusted through each foreground and background point.

[0097] In one specific embodiment of this application, step S206 may include the following steps:

[0098] Step 1: Obtain the temporal and spatial attention corresponding to each background point;

[0099] Step 2: Calculate the mean of attention at each time point to obtain the mean of attention at the background point.

[0100] Step 3: Adjust the time attention scores between each foreground point and each background point based on the average time attention score of the background points;

[0101] Step 4: Calculate the mean of spatial attention for each point to obtain the mean of spatial attention for the background point;

[0102] Step 5: Adjust the spatial attention scores between each foreground point and each background point based on the mean spatial attention score of the background points.

[0103] For ease of description, the five steps above can be combined for explanation.

[0104] The process involves acquiring the temporal and spatial attention values ​​for each background point, averaging these values ​​to obtain the average temporal attention score for each background point, and then adjusting the temporal attention score between each foreground point and each background point based on this average. Similarly, averaging the spatial attention values ​​for each background point is performed to obtain the average spatial attention score, which is then used to adjust the spatial attention score between each foreground point and each background point. By using the average attention score between background points as a benchmark to adjust the temporal attention score between foreground and background points, a more consistent restoration effect is achieved. Similarly, by using the average attention score between background points as a benchmark to adjust the spatial attention score between foreground and background points, a more consistent restoration effect is achieved.

[0105] In one specific embodiment of this application, correcting the temporal attention score between each foreground point and each background point based on the average temporal attention value of the background points may include the following steps:

[0106] The average temporal attention of the background points is used as the temporal attention between each foreground point and each background point.

[0107] After calculating the average of the attention values ​​at each time point, the average background attention value is used as the time attention between each foreground point and each background point. By using the average attention value between background points as a benchmark, the time attention scores between foreground and background points are adjusted, resulting in a more consistent restoration effect.

[0108] In one specific embodiment of this application, calculating the average attention at each time point may include the following steps:

[0109] Step 1: Obtain the time weighting factors corresponding to each background point;

[0110] Step 2: Calculate the mean of attention at each time based on the weighting factors of each time period.

[0111] For ease of description, the two steps above can be combined for explanation.

[0112] After obtaining the temporal attention corresponding to each background point, the temporal weighting factor corresponding to each background point is obtained, and the average value of each temporal attention is calculated based on the temporal weighting factor. By taking into account the difference in time distance, the temporal weighting factor is introduced for adjustment when correcting the temporal attention, thereby improving the accuracy of temporal attention correction.

[0113] The time attention scores between each foreground point and each background point can be corrected using the following formula:

[0114] ;

[0115] ;

[0116] in, This represents the set of background points between time-series frames. This represents the background point in the j-th video frame before correction. and the background point in the k-th video frame Time attention between Indicates the location before the object to be erased. Indicates the time weighting factor. The mean of time attention for background points. The time attention between each foreground point and each background point.

[0117] In one specific embodiment of this application, correcting the spatial attention score between each foreground point and each background point based on the mean spatial attention of background points may include the following steps:

[0118] The average spatial attention of the background points is used as the spatial attention between each foreground point and each background point.

[0119] After calculating the average spatial attention of each point and obtaining the average spatial attention of the background points, this average spatial attention is used as the spatial attention between each foreground point and each background point. By using the average attention between background points as a benchmark, the spatial attention scores between foreground and background points are adjusted, achieving a more consistent restoration effect.

[0120] In one specific embodiment of this application, calculating the mean of each spatial attention may include the following steps:

[0121] Step 1: Obtain the spatial weight factors corresponding to each background point;

[0122] Step 2: Calculate the mean of the attention for each space based on the weighting factors of each space.

[0123] For ease of description, the two steps above can be combined for explanation.

[0124] After obtaining the spatial attention corresponding to each background point, the spatial weight factor corresponding to each background point is obtained, and the average value of each spatial attention is calculated based on the spatial weight factor. By taking into account the difference in positional distance, a distance weight factor is introduced to adjust the spatial attention when correcting it, thereby improving the accuracy of spatial attention correction.

[0125] The spatial attention score between each foreground point and each background point can be corrected using the following formula:

[0126] ;

[0127] ;

[0128] in, This represents the set of background points in the current frame. This represents the background point in the i-th video frame before correction. and background points Spatial attention between Indicates spatial weighting factor, The mean of spatial attention for background points. Spatial attention is required between each foreground point and each background point.

[0129] S207: Process the first concatenated features layer by layer using each Transformer layer to obtain the features of each layer.

[0130] After concatenating the latent space features to obtain the first concatenated feature, each Transformer layer is used to process the first concatenated feature layer by layer to obtain the features of each layer.

[0131] S208: Calculate the diffusion loss based on the erased real video and the output features of the last Transformer layer.

[0132] After processing the first stitched features layer by layer using each Transformer layer to obtain the features of each layer, the diffusion loss is calculated based on the erased real video and the output features of the last Transformer layer.

[0133] S209: Select shallow features, middle features, and high-level features from the features of each layer, and concatenate the shallow features, middle features, and high-level features to obtain the second concatenated feature.

[0134] After processing the first concatenated feature layer by layer using each Transformer layer to obtain the features of each layer, shallow features, middle features and high-level features are selected from the features of each layer, and the shallow features, middle features and high-level features are concatenated to obtain the second concatenated feature.

[0135] S210: The second splicing feature is processed using the mask decoder in the initial video object erasure model, and the output is a real mask containing the object to be erased and its side effects.

[0136] An initial video object erasure model is pre-built. Shallow, mid-level, and high-level features are selected from the features of each layer. The shallow, mid-level, and high-level features are then concatenated to obtain the second concatenated features. The mask decoder in the initial video object erasure model is then used to process the second concatenated features and output a real mask containing the object to be erased and its side effects.

[0137] S211: Calculate the mean squared error loss based on the true mask and the ground truth mask.

[0138] After processing the second splicing features using the mask decoder in the initial video object erasure model and outputting a true mask containing the object to be erased and its side effects, the mean squared error (MSE) loss is calculated based on the true mask and the ground truth mask.

[0139] S212: Iteratively train the initial video object erasure model based on the diffusion loss and mean square error loss to obtain the target video object erasure model.

[0140] After calculating the diffusion loss based on the output features of the erased real video and the last Transformer layer, and calculating the mean squared error loss based on the ground truth mask and the ground truth mask, the initial video object erasure model is iteratively trained based on the diffusion loss and the mean squared error loss to obtain the target video object erasure model.

[0141] S213: Get the first raw video containing the video object to be erased.

[0142] S214: Select the video object to be erased from the first original video according to the received object selection instruction.

[0143] S215: Perform masking on the video object to be erased to obtain the object mask.

[0144] S216: Using the Transformer layers obtained by mask-supervised learning (containing the object to be erased and side effects) through a mask decoder in the target video object erasure model, the original video is erased based on the video point tracking results corresponding to the video object to be erased and the object mask.

[0145] Each Transformer layer includes an attention module that divides the points in the first original video into foreground and background points based on the video point tracking results and object masks. Foreground points are the points that constitute the video object to be erased, and background points are the other points in the first original video besides those that constitute the video object to be erased.

[0146] By combining arbitrary point tracking technology in video with a DIT diffusion model, point tracking technology is used to assist the DIT model in understanding the relationship between the flawed object and the background environment, improving the understanding of surrounding image features. This constructs a new spatiotemporal attention guidance mechanism to guide the model in uncovering the side effects caused by the flawed object in the video. Simultaneously, to enable the model to accurately uncover the side effects caused by the flawed object, a supervised mask decoder for flawed object side effects is introduced, guiding the model training through supervised learning. This application's embodiments implement object erasure in film and television post-production. When erasing a flawed object, not only is the flawed object removed, but the side effects it causes, such as shadows, reflections, lighting, and mirror images, are also eliminated. This ensures the integrity and consistency of the image, improves the production efficiency of film and television works, and reduces costs.

[0147] See Figure 3 , Figure 3 This diagram illustrates the training process of a video object erasure model according to an embodiment of this application. The training process of the video object erasure model mainly involves four modules: VAE Encoder, DIT module, point tracking-guided attention module, and MASK Decoder. Among them, VAE Encoder and DIT module are standard modules in AIGC, while point tracking-guided attention module and MASK decoding module are the core modules of this embodiment.

[0148] from Figure 3As can be seen, there are four inputs during training: the object mask, the original video (the video to be erased), the erased video (the ground truth), and the object + side effect mask. The object mask, the original video, and the erased video are each input into the VAE Encoder for feature compression. The compressed latent space features are then concatenated and input into the DIT module. The DIT module contains multiple transformer blocks, each with a point tracking-guided attention module. This module divides the points in the video into foreground and background points based on the point tracking results and the object mask. It adjusts the spatiotemporal attention mechanism of the DIT module using point tracking information between adjacent video frame sequences, enabling the model to uncover the side effects of object erasure. After the data has been processed in all transformer blocks, a diffusion loss is calculated based on the output to supervise model training. In addition, in order to more accurately discover the side effects caused by the erasure of objects, the features processed by multiple transformer blocks in the DIT module are concatenated and input into a Mask Decoder. The Mask Decoder will output a mask containing the erasure objects and their side effects. Finally, the MSE loss is calculated by combining the mask and the ground truth mask, and together with the diffusion loss, the training of the model is supervised.

[0149] The pseudocode involved in the training process of the video object erasure model is as follows:

[0150]

[0151] Corresponding to the above method embodiments, this application also provides a video object erasing device, and the video object erasing device described below can be referred to in correspondence with the video object erasing method described above.

[0152] See Figure 4 , Figure 4 This is a structural block diagram of a video object erasing device according to an embodiment of this application. The device may include:

[0153] The first original video acquisition module 41 is used to acquire the first original video containing the video object to be erased.

[0154] The video object selection module 42 is used to select the video object to be erased from the first original video according to the received object selection instruction;

[0155] The object mask acquisition module 43 is used to perform mask processing on the video object to be erased to obtain the object mask;

[0156] The object erasure module 44 is used to use the Transformer layers obtained by mask-supervised learning, which includes the object to be erased and side effects, through the mask decoder in the target video object erasure model, to erase the object in the original video based on the video point tracking results corresponding to the video object to be erased and the object mask.

[0157] Each Transformer layer includes an attention module that divides the points in the first original video into foreground and background points based on the video point tracking results and object masks. Foreground points are the points that constitute the video object to be erased, and background points are the other points in the first original video besides those that constitute the video object to be erased.

[0158] As can be seen from the above technical solution, by setting a point tracking-guided attention module in each Transformer layer of the target video object erasure model, the points in the first original video are divided into foreground and background points using the point tracking-guided attention module. Video point tracking technology guides the calculation of spatiotemporal attention, thus erasing the video object to be erased and its side effects more smoothly and accurately. By introducing a mask reconstruction mechanism, the difference between the reconstructed mask and the ground truth mask is used as a loss signal to supervise the model training, guiding the model to discover features that can represent the video object to be erased and its side effects, thereby achieving accurate erasure. When erasing objects in post-production, not only are the selected objects erased, but the side effects they bring to the video are also erased, improving erasure efficiency and accuracy.

[0159] In one specific embodiment of this application, the device may further include a model training module, which may include:

[0160] The data acquisition submodule is used to acquire the mask of the object to be erased, the second original video, the real video after erasure, and the truth mask containing the object to be erased and side effects.

[0161] The latent space feature acquisition submodule is used to compress features from the mask of the object to be erased, the second original video, and the real video after erasure using each variational autoencoder to obtain each latent space feature.

[0162] The first splicing feature acquisition submodule is used to splice the latent space features to obtain the first splicing feature;

[0163] The video point tracking result determination submodule is used to determine the video point tracking results between adjacent video frame sequences based on the second original video.

[0164] The foreground and background point segmentation submodule is used to divide the points in the second original video into foreground and background points based on the point tracking in each Transformer layer of the initial video object erasure model and the video point tracking results and the mask of the object to be erased.

[0165] The Spatiotemporal Attention Mechanism Adjustment Submodule is used to adjust the spatiotemporal attention mechanism of each Transformer layer based on each foreground point and each background point.

[0166] The feature acquisition submodule is used to process the first concatenated features layer by layer using each Transformer layer to obtain the features of each layer.

[0167] The diffusion loss calculation submodule is used to calculate the diffusion loss based on the erased real video and the output features of the last Transformer layer;

[0168] The second splicing feature acquisition submodule is used to select shallow features, middle features and high-level features from each layer of features, and splice the shallow features, middle features and high-level features to obtain the second splicing feature;

[0169] The real mask output submodule is used to process the second splicing features using the mask decoder in the initial video object erasure model and output a real mask containing the object to be erased and side effects.

[0170] The mean squared error loss calculation submodule is used to calculate the mean squared error loss based on the true mask and the ground truth mask.

[0171] The target video object erasure model acquisition submodule is used to iteratively train the initial video object erasure model based on the diffusion loss and mean square error loss to obtain the target video object erasure model.

[0172] In one specific embodiment of this application, the spatiotemporal attention mechanism adjustment submodule may include:

[0173] The attention acquisition unit is used to acquire the temporal and spatial attention corresponding to each background point;

[0174] The background point temporal attention mean acquisition unit is used to calculate the mean of each temporal attention to obtain the background point temporal attention mean.

[0175] The time attention score correction unit is used to correct the time attention scores between each foreground point and each background point based on the average time attention score of the background points;

[0176] The background point spatial attention mean acquisition unit is used to calculate the mean of each spatial attention and obtain the background point spatial attention mean.

[0177] The spatial attention score correction unit is used to correct the spatial attention scores between each foreground point and each background point based on the average spatial attention of the background points.

[0178] In one specific embodiment of this application, the time attention score correction unit is specifically a unit that uses the average time attention of the background points as the time attention between each foreground point and each background point.

[0179] In one specific embodiment of this application, the background point temporal attention mean acquisition unit may include:

[0180] The time weight factor acquisition sub-unit is used to acquire the time weight factors corresponding to each background point.

[0181] The time attention mean calculation subunit is used to calculate the mean of each time attention based on each time weight factor.

[0182] In one specific embodiment of this application, the spatial attention score correction unit is specifically a unit that uses the average spatial attention of the background points as the spatial attention between each foreground point and each background point.

[0183] In one specific embodiment of this application, the background point spatial attention mean acquisition unit may include:

[0184] The spatial weight factor acquisition sub-unit is used to acquire the spatial weight factors corresponding to each background point.

[0185] The spatial attention mean calculation subunit is used to calculate the mean of each spatial attention based on each spatial weight factor.

[0186] For the method embodiments described above, see [link to relevant documentation]. Figure 5 , Figure 5 This is a schematic diagram of the video object erasing device provided in this application. The device may include:

[0187] Memory 332 is used to store computer programs;

[0188] The processor 322 is used to implement the steps of the video object erasure method of the above method embodiment when executing a computer program.

[0189] For details, please refer to Figure 6 , Figure 6This is a schematic diagram illustrating the specific structure of a video object erasing device provided in this embodiment. The video object erasing device can vary significantly due to different configurations or performance characteristics. It may include a processor (central processing unit, CPU) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or persistent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the video object erasing device 301.

[0190] The video object erasure device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0191] The steps in the video object erasure method described above can be implemented by the structure of the video object erasure device.

[0192] Corresponding to the above method embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps:

[0193] The process involves: acquiring a first original video containing the video object to be erased; selecting the video object to be erased from the first original video according to the received object selection instruction; performing masking on the video object to be erased to obtain an object mask; and using each Transformer layer obtained through mask-supervised learning of the mask containing the object to be erased and its side effects via a mask decoder in the target video object erasure model to erase the original video object based on the video point tracking results corresponding to the video object to be erased and the object mask. Each Transformer layer includes a point tracking-guided attention module that divides the points in the first original video into foreground points and background points based on the video point tracking results and the object mask. Foreground points are the points that constitute the video object to be erased, and background points are the other points in the first original video besides those that constitute the video object to be erased.

[0194] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0196] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0197] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for erasing video objects, characterized in that, include: Get the first raw video containing the video object to be erased; The video object to be erased is selected from the first original video according to the received object selection instruction; The video object to be erased is masked to obtain the object mask; Using the Transformer layers obtained by mask-supervised learning (which includes the object to be erased and side effects) through a mask decoder in the target video object erasure model, the original video is erased based on the video point tracking results corresponding to the video object to be erased and the object mask. Each Transformer layer includes a point tracking-guided attention module that divides the points in the first original video into foreground and background points based on the video point tracking results and the object mask. The foreground point is the point that constitutes the video object to be erased, and the background point is any other point in the first original video besides the point that constitutes the video object to be erased.

2. The video object erasing method according to claim 1, characterized in that, It also includes the training process of the target video object erasure model, which includes: Obtain the mask of the object to be erased, the second original video, the real video after erasure, and the truth mask containing the object to be erased and side effects; Each variational autoencoder is used to perform feature compression on the mask of the object to be erased, the second original video, and the real video after erasure, respectively, to obtain each latent space feature; The features in each latent space are concatenated to obtain the first concatenated feature. The video point tracking results between adjacent video frame sequences are determined based on the second original video. The attention module, guided by point tracking within each Transformer layer of the initial video object erasure model, divides the points in the second original video into foreground points and background points based on the video point tracking results and the mask of the object to be erased. Adjust the spatiotemporal attention mechanism of each Transformer layer by using each foreground point and each background point; The first concatenated feature is processed layer by layer using each Transformer layer to obtain the features of each layer; The diffusion loss is calculated based on the erased real video and the output features of the last Transformer layer. Shallow features, middle features, and high-level features are selected from the features of each layer, and the shallow features, middle features, and high-level features are concatenated to obtain the second concatenated feature; The second splicing feature is processed using the mask decoder in the initial video object erasure model to output a real mask containing the object to be erased and side effects; Calculate the mean squared error loss based on the true mask and the ground truth mask; The initial video object erasure model is iteratively trained based on the diffusion loss and the mean square error loss to obtain the target video object erasure model.

3. The video object erasing method according to claim 2, characterized in that, The spatiotemporal attention mechanism of each Transformer layer is adjusted by considering each foreground and background point, including: Obtain the temporal and spatial attention corresponding to each background point; The mean of attention at each time point is calculated to obtain the mean of background time attention. The time attention scores between each foreground point and each background point are adjusted based on the average time attention value of the background points. The mean value of spatial attention for each spatial attention point is calculated to obtain the mean value of spatial attention for the background point. The spatial attention scores between each foreground point and each background point are adjusted based on the mean spatial attention of the background points.

4. The video object erasing method according to claim 3, characterized in that, The temporal attention scores between each foreground point and each background point are adjusted based on the mean temporal attention value of the background points, including: The average temporal attention of the background points is used as the temporal attention between each foreground point and each background point.

5. The video object erasure method according to claim 3, characterized in that, The mean of attention at different times was calculated, including: Obtain the time weight factors corresponding to each background point; The mean of attention at each time point is calculated based on the weighting factors of each time point.

6. The video object erasure method according to claim 3, characterized in that, The spatial attention scores between each foreground point and each background point are corrected based on the mean spatial attention value of the background points, including: The average spatial attention of the background points is used as the spatial attention between each foreground point and each background point.

7. The video object erasure method according to claim 3, characterized in that, The mean of attention across all spaces is calculated, including: Obtain the spatial weight factors corresponding to each background point; The mean of attention in each space is calculated based on the weighting factors of each space.

8. A video object erasing device, characterized in that, include: The first original video acquisition module is used to acquire the first original video containing the video object to be erased. The video object to be erased selection module is used to select the video object to be erased from the first original video according to the received object selection instruction; The object mask acquisition module is used to perform mask processing on the video object to be erased to obtain the object mask; The object erasure module is used to use each Transformer layer obtained by mask-supervised learning, which includes the object to be erased and side effects, through a mask decoder in the target video object erasure model, to erase the object in the original video based on the video point tracking results corresponding to the video object to be erased and the object mask. Each Transformer layer includes a point tracking-guided attention module that divides the points in the first original video into foreground and background points based on the video point tracking results and the object mask. The foreground point is the point that constitutes the video object to be erased, and the background point is any other point in the first original video besides the point that constitutes the video object to be erased.

9. A video object erasing device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the video object erasure method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video object erasure method as described in any one of claims 1 to 7.

Images