Video data processing method and device, equipment and medium

Abstract

The embodiment of the invention provides a video data processing method and device, equipment and a medium. The method comprises the following steps: acquiring initial noise data, source video data and an object mask sequence of a candidate object in the source video data; performing visual coding processing on the source video data to obtain video potential features, and obtaining a first denoising prediction result of a first time step; constructing denoising condition data of a second time step according to the first denoising prediction result, the potential video features and the object mask sequence; performing self-attention processing on the de-noising condition data to obtain a first self-attention feature of a second time step, obtaining a cross attention feature of the first time step, and generating a second de-noising prediction result of the second time step according to the first self-attention feature and the cross attention feature; and generating repaired video data corresponding to the source video data according to the second denoising prediction result. By implementing the embodiment of the invention, the removal effect of the video object and the visual special effect can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video data processing method, apparatus, device, and medium. Background Technology

[0002] Video object and visual effects removal refers to completely erasing a specified foreground object and its associated visual effects (e.g., shadows, reflections) from a video sequence, and then using the video's background information to fill in the erased area appropriately, so that it remains visually coherent and free of artifacts.

[0003] In current scenarios involving the removal of foreground objects and visual effects, the source video and mask are directly mixed at the network input layer of the model, allowing the model to automatically fill in the area within the mask, thereby removing specific foreground objects and visual effects from the video. However, this method results in the rich spatiotemporal context information in the video being severely diluted or discarded after being processed through multiple layers of the network, leading to poor removal results for video objects and visual effects. Summary of the Invention

[0004] This application provides a video data processing method, apparatus, device, and medium that can improve the removal effect of video objects and visual effects.

[0005] One embodiment of this application provides a video data processing method, including: Obtain the source video data and the object mask sequence of candidate objects in the source video data, and obtain the initial noise data; Visual encoding is performed on the source video data to obtain the video's latent features, and the first denoising prediction result at the first time step is obtained; the first denoising prediction result refers to the prediction result of the initial noise data in the denoising process at the first time step. Based on the first denoising prediction result, video latent features, and object mask sequence, denoising conditional data for the second time step is constructed; the second time step refers to the time step after the first time step. Self-attention processing is performed on the denoised conditional data to obtain the first self-attention feature of the second time step. The cross-attention feature of the first time step is obtained. Based on the first self-attention feature and the cross-attention feature, the second denoising prediction result of the second time step is generated. The cross-attention feature refers to the fusion feature that includes visual information from the source video data. Based on the second denoising prediction result, repaired video data corresponding to the source video data is generated; repaired video data refers to video data from the source video data after removing candidate objects and their visual effects.

[0006] One embodiment of this application provides a video data processing method, including: Acquire training samples and sample noise data. The training samples include sample video data, target video data, and object sample mask sequences of sample objects in the sample video data. Visual encoding is performed on the sample video data to obtain the sample latent features and to obtain the first sample prediction result at the first time step. The first sample prediction result refers to the prediction result of the sample noise data in the denoising process at the first time step. Based on the prediction results of the first sample, the latent features of the sample, and the object sample mask sequence, the sample condition data for the second time step is constructed; the second time step refers to the time step after the first time step. The sample conditional data is processed by self-attention using the initial repair model to obtain the self-attention prediction features of the second time step, and the cross-attention prediction features of the first time step are obtained. Based on the self-attention prediction features and the cross-attention prediction features, the second sample prediction result of the second time step is generated. The cross-attention prediction features refer to the fusion features that include visual information of the sample video data. Based on the prediction results of the second sample, generate the predicted video data corresponding to the sample video data; Based on the target video data and the predicted video data, the network parameters of the initial restoration model are corrected to obtain the video restoration model. The video restoration model is used to remove candidate objects and their visual effects from the source video data based on the object mask sequence of the candidate objects in the source video data.

[0007] One embodiment of this application provides a video data processing apparatus, including: The first acquisition module is used to acquire source video data and the object mask sequence of candidate objects in the source video data, and to acquire initial noise data. The first encoding module is used to perform visual encoding processing on the source video data to obtain the video latent features and acquire the first denoising prediction result at the first time step; the first denoising prediction result refers to the prediction result of the initial noise data in the denoising processing at the first time step. The first construction module is used to construct the denoising conditional data for the second time step based on the first denoising prediction result, video latent features, and object mask sequence; the second time step refers to the time step after the first time step. The first processing module is used to perform self-attention processing on the denoised conditional data to obtain the first self-attention feature of the second time step, obtain the cross-attention feature of the first time step, and generate the second denoised prediction result of the second time step based on the first self-attention feature and the cross-attention feature; the cross-attention feature refers to the fusion feature that includes visual information of the source video data. The first generation module is used to generate repaired video data corresponding to the source video data based on the second denoising prediction result; the repaired video data refers to video data from the source video data after removing the candidate objects and their visual effects.

[0008] The first acquisition module acquires the source video data and the object mask sequence of candidate objects in the source video data, and performs the following steps: Acquire source video data, perform frame segmentation on the source video data to obtain M video frames, extract features from each of the M video frames to obtain the first image feature of each video frame; M is a positive integer. In M video frames, the starting video frame related to the candidate object is obtained. In the starting video frame, the visual cue information of the candidate object is determined. The visual cue information is encoded to obtain the cue encoding feature. The second image features of the starting video frame are obtained, and the cue encoding features and the second image features are fused to generate the initial mask of the candidate object. The starting video frame, visual cue information, and initial mask are combined into mask cue information. Based on the mask cue information and the first image features of each video frame, an object mask sequence of candidate objects is generated.

[0009] The first encoding module performs visual encoding processing on the source video data to obtain latent video features, which are then used to perform the following steps: The source video data is extracted by a variational autoencoder to obtain the mean coding features and variance coding features of the source video data. By sampling the mean-coded features and variance-coded features, the latent video features of the source video data are obtained.

[0010] The first construction module constructs denoising conditional data for the second time step based on the first denoising prediction result, video latent features, and object mask sequence, which is used to perform the following steps: The first denoised prediction result is serialized to obtain a denoised prediction sequence, and the video latent features are serialized to obtain a video latent feature sequence. The denoising prediction sequence, the video latent feature sequence, and the object mask sequence are combined into the denoising conditional data for the second time step.

[0011] The first processing module performs self-attention processing on the denoised conditional data to obtain the first self-attention feature at the second time step, which is used to perform the following steps: Based on the first transformation weight matrix of the self-attention network in the video restoration model, the denoising conditional data is transformed into the first key matrix, and based on the second transformation weight matrix of the self-attention network, the denoising conditional data is transformed into the first value matrix. The first query matrix is ​​generated based on the second time step, the denoised conditional data, and the third transformation weight matrix of the self-attention network. Perform a dot product operation between the first query matrix and the transpose of the first key matrix to obtain the candidate weight matrix; Obtain the number of columns corresponding to the first query matrix, normalize the ratio between the candidate weight matrix and the square root of the number of columns to obtain the attention weight matrix, and obtain the first self-attention feature of the second time step based on the dot product between the attention weight matrix and the first value matrix.

[0012] The first processing module generates a first query matrix based on the second time step, the denoised conditional data, and the third transformation weight matrix of the self-attention network, which is used to perform the following steps: When the second time step is a background update cycle node, the denoised conditional data is converted into the first query matrix according to the third transformation weight matrix of the self-attention network. When the second time step is not a background update cycle node, obtain the background mask of the previous background update cycle node of the second time step, and generate the first query matrix based on the background mask, denoising condition data and the third transformation weight matrix.

[0013] The first processing module obtains the background mask of the previous background update cycle node at the second time step, and generates a first query matrix based on the background mask, denoising condition data, and the third transformation weight matrix, which is used to perform the following steps: Obtain the third denoising prediction result of the previous background update cycle node in the second time step; Obtain the similarity map between the third denoising prediction result and the video latent features, and generate the background mask of the previous background update cycle node based on the similarity map; The denoising conditional data is updated based on the background mask to obtain conditional update data. The conditional update data is then converted into the first query matrix based on the third transformation weight matrix.

[0014] The first processing module obtains the cross-attention features at the first time step and uses them to perform the following steps: Obtain the time-step sequence of the initial noise data; When the first time step is the first time step in the time step sequence, the second self-attention feature of the first time step is obtained, and the second self-attention feature and the video latent feature are cross-attention processed to obtain the cross-attention feature of the first time step. When the first time step is not the first time step in the time step sequence, the attention features already stored in the cache space are identified as the cross-attention features of the first time step.

[0015] The first processing module performs cross-attention processing on the second self-attention features and the video latent features to obtain the cross-attention features at the first time step, which are used to perform the following steps: By using the cross-attention network in the video restoration model, the second self-attention feature is transformed into the second query matrix, and the video latent feature is transformed into the second key matrix and the second value matrix. Based on the second query matrix, the second key matrix, and the second value matrix, the cross-attention features of the first time step are obtained.

[0016] The first processing module generates a second denoising prediction result for the second time step based on the first self-attention feature and the cross-attention feature, which is used to perform the following steps: Obtain the attention weight parameters, and determine the candidate attention features by multiplying the attention weight parameters with the cross-attention features; The candidate attention features and the first self-attention features are added together to obtain the second denoising prediction result at the second time step.

[0017] The first generation module generates repaired video data corresponding to the source video data based on the second denoising prediction result, which is used to perform the following steps: When the second time step is the last time step in the time step sequence, the second denoising prediction result is deserialized to obtain the video restoration features. The video restoration features are decoded to obtain the restored video frame sequence; Perform video restoration on the sequence of video frames to generate restored video data corresponding to the source video data.

[0018] One embodiment of this application provides a video data processing apparatus, including: The second acquisition module is used to acquire training samples and sample noise data. The training samples include sample video data, target video data, and object sample mask sequences of sample objects in the sample video data. The second encoding module is used to perform visual encoding processing on the sample video data to obtain the latent features of the samples and obtain the first sample prediction result at the first time step; the first sample prediction result refers to the prediction result of the sample noise data in the denoising process at the first time step. The second construction module is used to construct the sample condition data for the second time step based on the prediction results of the first sample, the latent features of the sample, and the object sample mask sequence; the second time step refers to the time step after the first time step. The second processing module is used to perform self-attention processing on the sample conditional data through the initial repair model to obtain the self-attention prediction features of the second time step, obtain the cross-attention prediction features of the first time step, and generate the second sample prediction result of the second time step based on the self-attention prediction features and the cross-attention prediction features; the cross-attention prediction features refer to the fusion features that include visual information of the sample video data. The second generation module is used to generate predicted video data corresponding to the sample video data based on the prediction results of the second sample. The parameter correction module is used to correct the network parameters of the initial repair model based on the target video data and the predicted video data to obtain the video repair model. The video repair model is used to remove candidate objects and their visual effects from the source video data based on the object mask sequence of the candidate objects in the source video data.

[0019] The second acquisition module acquires training samples and performs the following steps: Obtain the foreground object set and the background image set. Combine the foreground objects in the foreground object set and the background images in the background image set to obtain K pairs; K is a positive integer. The foreground object and background image in each pairing group are rendered to obtain K sample video frames, and the K sample video frames are combined into sample video data. The sample objects are identified from the foreground objects contained in the sample video data, and the sample objects in the sample video data are removed to obtain the target video data. Obtain the object sample mask sequence of the sample object in the sample video data, and determine the sample video data, target video data, and object sample mask sequence as training samples.

[0020] The parameter correction module corrects the network parameters of the initial restoration model based on the target video data and the predicted video data to obtain a video restoration model, which is used to perform the following steps: Obtain the label movement speed between the sample noise data and the target video data, and obtain the path movement speed between the sample noise data and the predicted video data; The model loss of the initial repair model is determined based on the label movement speed and the path movement speed. Based on the model loss, the network parameters of the initial restoration model are corrected to obtain the video restoration model.

[0021] One aspect of this application provides a computer device, including a memory and a processor. The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method provided in one aspect of this application.

[0022] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in one aspect of this application.

[0023] According to one aspect of this application, a computer program product is provided, which may include a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and executes the computer program, causing the computer device to perform the method provided in the above aspect.

[0024] In this embodiment, source video data, initial noise data, and object mask sequences of candidate objects in the source video data can be obtained. Visual encoding processing is performed on the source video data to obtain latent video features, resulting in a first denoising prediction result for the first time step. Based on the first denoising prediction result, the latent video features, and the object mask sequence, denoising conditional data for the second time step can be constructed. Self-attention processing is applied to the denoising conditional data to obtain a first self-attention feature for the second time step. Through cached cross-attention features and the first self-attention feature, a second denoising prediction result for the second time step can be obtained. The cross-attention feature refers to a fusion feature that includes visual information from the source video data. Based on the second denoising prediction result, repaired video data corresponding to the source video data can be generated. During video repair, visual information from the source video data can be adaptively fused into the cross-attention feature according to the needs of the current repair area, thereby generating better-performing, artifact-free repaired video data, which can improve the removal effect of video objects and visual effects. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 1 ; Figure 3 This is a schematic diagram of a scene for removing video objects and visual effects provided in an embodiment of this application; Figure 4 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 2 ; Figure 5 This is a schematic diagram of the network structure of a video restoration model provided in an embodiment of this application; Figure 6 This is a schematic diagram of the network structure of a self-attention network provided in an embodiment of this application; Figure 7 This is a schematic diagram of the network structure of a cross-attention network provided in an embodiment of this application; Figure 8 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 3 ; Figure 9 This is a schematic diagram of a model training process provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a video data processing device provided in an embodiment of this application. Figure 1 ; Figure 11 This is a schematic diagram of the structure of a video data processing device provided in an embodiment of this application. Figure 2 ; Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] It is understood that in the specific implementation of this application, video data containing user images or video data taken by the user may be involved. When the embodiments of this application are applied to specific products or technologies, permission or consent from relevant institutions or departments or the user himself or himself or himself or her ...

[0029] For ease of understanding, the basic technical concepts involved in the embodiments of this application will be described below: Diffusion model: This is a deep learning model that can gradually recover clear video data from random noise by simulating the forward "noise addition" and reverse "noise removal" processes. The diffusion model includes a forward diffusion stage and a reverse generation stage. In the forward diffusion stage, noise can be gradually added to the source video data until it becomes random noise. In the reverse generation stage, a neural network can be trained to learn to gradually remove noise, reconstructing or generating new video data from the random noise.

[0030] Video editing refers to the technical process of modifying, processing, or recombining digital video content. It goes beyond traditional editing and splicing; specifically, it involves using artificial intelligence models and computer vision methods to perform pixel-level deep manipulations on video frames. This can include identifying specific objects in the video, erasing those objects and their associated visual effects, and automatically filling in gaps using surrounding pixel information to generate visually natural and seamless restored video data.

[0031] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application; the network architecture may include a server 10d and a terminal cluster, and the terminal cluster may include one or more terminal devices, without limiting the number of terminal devices included in the terminal cluster. Figure 1 As shown, the terminal cluster may specifically include terminal device 10a, terminal device 10b, and terminal device 10c, etc.; all terminal devices in the terminal cluster (for example, may include terminal device 10a, terminal device 10b, and terminal device 10c, etc.) can be connected to server 10d via the network so that each terminal device can interact with server 10d through the network connection.

[0032] in, Figure 1 The terminal devices in the terminal cluster shown may include, but are not limited to: smartphones, tablets, laptops, PDAs, desktop computers, wearable devices (such as smartwatches, smart bracelets, etc.), smart voice interaction devices, smart home appliances (such as smart TVs, etc.), in-vehicle devices, aircraft, and other electronic devices. This application does not limit the type of terminal device.

[0033] Figure 1The server 10d shown can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application does not limit the type of server.

[0034] In the embodiments of this application, Figure 1 Each terminal device in the shown cluster can install video editing software. When the video editing software runs on each terminal device, it can interact with... Figure 1 The servers 10d shown interact with each other. The video editing software running on each terminal device can be a standalone software program or an embedded subroutine integrated into a separate software program; this application does not limit this. The video editing software running on each terminal device can include, but is not limited to: professional film and television post-production software, video editing software, video effects and compositing software, 3D animation and rendering software, and other software with functions for removing video objects and visual effects.

[0035] Figure 1 The server 10d shown can be the backend server corresponding to the video editing software installed in each terminal device. If multiple video editing software programs are running on a terminal device (e.g., terminal device 10a), the multiple video editing software programs can correspond to different servers or the same server, and this application does not limit this.

[0036] Please see Figure 2 , Figure 2 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 1 Understandably, this video data processing method is performed by a computer device, which can be a terminal device (e.g., Figure 1 Any terminal device in the set of terminal devices shown), or a server (such as... Figure 1 The server 10d shown is not limited in this application. The video data processing method may include the following steps S101 to S105: Step S101: Obtain source video data and object mask sequences of candidate objects in the source video data, and obtain initial noise data.

[0037] Specifically, computer devices can be equipped with video editing software. When the video editing software runs on the computer device, it acquires the source video data and the object mask sequence of candidate objects in the source video data. Candidate objects can also be called objects to be removed. Objects to be removed refer to target objects in the source video that need to be removed but have not yet been removed, such as a camera tripod that accidentally entered the scene during filming, pedestrians, moving vehicles, reflections on glass, etc.

[0038] An object mask sequence can be a list of masks arranged chronologically. The length of the object mask sequence is the same as the total number of video frames in the source video data. For example, if the source video data includes 15 video frames, meaning the total number of video frames in the source video data is 15, then the length of the object mask sequence corresponding to this source video data is 15. Each video frame in the source video data can correspond to an object mask in the object mask sequence; for example, the i-th video frame in the source video data corresponds to the i-th object mask in the object mask sequence.

[0039] Each object mask in an object mask sequence is a binary matrix or probability matrix with the same size as the video frame, where each pixel can have a value of 0 or 1. Object masks can be used to precisely mark the pixel locations of target objects. For example, the pixel values ​​of the target object to be removed can be set to 1, and the pixel values ​​of pixels not belonging to the target object to be removed can be set to 0.

[0040] In one or more embodiments, a visual segmentation model can be used to segment source video data to obtain a sequence of object masks for candidate objects (also known as objects to be removed) in the source video data. A visual segmentation model refers to a deep learning model that can automatically or semi-automatically separate target objects from video frames contained in video data. Visual segmentation models may include, but are not limited to, U-shaped networks (U-Net), fully convolutional networks (FCN), and Segment Anything Model 2 (SAM-2).

[0041] Initial noise data is obtained, which can refer to random noise. For example, the initial noise data can be noise randomly sampled from a Gaussian distribution or noise randomly sampled from a normal distribution; this application does not limit this. Here, "initial" specifically refers to the starting point of the denoising process. In one or more embodiments, the initial noise data can be subjected to multi-step denoising processing using a video restoration model to obtain restored video data. The video restoration model can be understood as a diffusion model. Multi-step refers to multiple time steps; that is, the denoising process of the initial noise data includes denoising processing steps at multiple time steps.

[0042] For example, the denoising process for initial noisy data may include a denoising process with three time steps. The three time steps are the first time step, the second time step, and the third time step. In other words, the denoising process for initial noisy data includes a denoising process with the first time step, a denoising process with the second time step, and a denoising process with the third time step.

[0043] In one or more embodiments, a denoising process can be performed using a rectified flow framework. Denoising processes based on traditional diffusion models often resemble traversing a winding, meandering path between two points. The core objective of rectified flow is to learn how to straighten this "winding path" into a straight line. Rectified flow can use ordinary differential equations (ODEs) to describe the evolution path of data from an initial state (e.g., random noise) to a target state (e.g., repaired video data).

[0044] In other words, within the rectified flow framework, the denoising process proceeds along a predicted straight path. The denoising process can be defined as mapping the noise distribution back to the data distribution using ordinary differential equations. Compared to the tortuous path of traditional diffusion models, the rectified flow framework can achieve video inpainting in fewer time steps, thus significantly improving the efficiency of generating inpainted video data.

[0045] Step S102: Perform visual encoding processing on the source video data to obtain video latent features and obtain the first denoising prediction result of the first time step; the first denoising prediction result refers to the prediction result of the initial noise data in the denoising processing of the first time step.

[0046] In one or more embodiments, source video data can be visually encoded using a visual encoder to obtain latent video features. A visual encoder is a neural network model that transforms or maps high-dimensional, raw visual data (such as image data or video data) into low-dimensional, dense latent vectors (which can also be understood as feature representations). Visual encoders can include, but are not limited to, 3D variational autoencoders (3D VAEs), 3D convolutional networks, and spatiotemporal graph convolutional networks.

[0047] Video latent features refer to the numerical representation of raw visual data in a low-dimensional "latent space" after compression processing by a visual encoder. The specific process of obtaining video latent features through visual encoding of source video data will be described in detail later.

[0048] Obtain the first denoising prediction result at the first time step. The first denoising prediction result refers to the prediction result of the initial noisy data in the denoising process at the first time step. Assume the denoising process of the initial noisy data includes 10 time steps, which can be arranged sequentially into a time step sequence. For example, when the first time step is the first time step in the time step sequence (or simply the first time step), the first denoising prediction result at the first time step can be considered the prediction result of the initial noisy data in the denoising process at the first time step. When the first time step is the second time step in the time step sequence (or simply the second time step), the first denoising prediction result at the first time step can be considered the prediction result of the initial noisy data in the denoising process at the second time step. ..., when the first time step is the tenth time step in the time step sequence, the first denoising prediction result at the first time step can be considered the prediction result of the initial noisy data in the denoising process at the tenth time step.

[0049] It is understandable that when the first time step is the last time step in the time step sequence (e.g., the 10th time step), the repaired video data of the source video data can be generated based on the first denoising prediction result of the first time step.

[0050] Step S103: Based on the first denoising prediction result, video latent features, and object mask sequence, construct the denoising conditional data for the second time step; the second time step refers to the time step after the first time step.

[0051] Specifically, the first denoising prediction result can be serialized using a patch embedding network to obtain a denoising prediction sequence; the video latent features can then be serialized using a patch embedding network to obtain a video latent feature sequence. Serialization refers to converting high-dimensional latent features into a one-dimensional vector sequence. In other words, the patch embedding network can segment the three-dimensional latent features into a series of patches, and map each patch to a token using linear projection, thus transforming the three-dimensional spatial features into a one-dimensional vector sequence.

[0052] After serialization processing via an image embedding network, the denoising prediction sequence, video latent feature sequence, and object mask sequence can be combined into denoising conditional data for the second time step. Denoising conditional data refers to the input data provided to the model in the denoising process at each time step, which includes conditional information guiding the denoising process.

[0053] For example, the denoising prediction sequence, the video latent feature sequence, and the object mask sequence can be concatenated into [denoising prediction sequence, video latent feature sequence, object mask sequence], and this [denoising prediction sequence, video latent feature sequence, object mask sequence] can be determined as the denoising conditional data for the second time step. As another example, the denoising prediction sequence, the video latent feature sequence, and the object mask sequence can be concatenated into [object mask sequence, denoising prediction sequence, video latent feature sequence], and this [object mask sequence, denoising prediction sequence, video latent feature sequence] can be determined as the denoising conditional data for the second time step; this application does not limit this approach. For ease of understanding, this application describes the example of determining [object mask sequence, denoising prediction sequence, video latent feature sequence] as the denoising conditional data for the second time step.

[0054] Understandably, in the denoising process at each time step, the denoising conditional data for the current time step can be constructed based on the denoising prediction result, video latent features, and object mask sequence from the previous time step. For example, the initial noise denoising process includes 10 time steps. In the first time step, the denoising conditional data for that time step can be constructed based on the initial noise data, video latent features, and object mask sequence. In the second time step, the denoising conditional data for that time step can be constructed based on the denoising prediction result, video latent features, and object mask sequence from the first time step. ... In the tenth time step, the denoising conditional data for that time step can be constructed based on the denoising prediction result, video latent features, and object mask sequence from the ninth time step.

[0055] Step S104: Perform self-attention processing on the denoised conditional data to obtain the first self-attention feature of the second time step, obtain the cross-attention feature of the first time step, and generate the second denoising prediction result of the second time step based on the first self-attention feature and the cross-attention feature; the cross-attention feature refers to the fusion feature that includes visual information of the source video data.

[0056] Specifically, the denoised conditional data can be processed using a self-attention network in the video restoration model to obtain the first self-attention feature at the second time step. Self-attention processing allows each element in the sequence to interact with all other elements in the sequence and dynamically assigns weights based on relevance to compute a new feature containing global context. The specific process of obtaining the first self-attention feature at the second time step by processing the denoised conditional data using a self-attention network in the video restoration model will be described in detail later.

[0057] The specific process of obtaining the cross-attention features of the first time step includes: when the first time step is the first time step in the time step sequence, the second self-attention features of the first time step can be obtained. The second self-attention features and the video latent features are then cross-attention processed by the cross-attention network in the video restoration model to obtain the cross-attention features of the first time step. Furthermore, the cross-attention features obtained in the first time step can be stored in a cache space.

[0058] When the first time step is not the first time step in the time step sequence, the stored attention features in the cache space can be directly identified as the cross-attention features of the first time step. The stored attention features refer to the cross-attention features obtained in the first time step.

[0059] After obtaining the first self-attention feature and the cross-attention feature, the attention weight parameter can be obtained. The product of the attention weight parameter and the cross-attention feature is determined as the candidate attention feature. The candidate attention feature and the first self-attention feature are added together to obtain the second denoising prediction result at the second time step. The value range of the attention weight parameter is [0, 1]. The attention weight parameter can be determined according to the situation. For example, the attention weight parameter can be 0.5, 0.7, etc., and this application does not limit it.

[0060] Understandably, the process of obtaining a denoising prediction result based on the denoising conditional data at each time step using a video restoration model can be called a denoising process. For example, the process of obtaining a denoising prediction result for the first time step based on the denoising conditional data at the first time step using a video restoration model can be called the denoising process for the first time step.

[0061] Step S105: Based on the second denoising prediction result, generate repaired video data corresponding to the source video data; repaired video data refers to video data from the source video data after removing candidate objects and their visual effects.

[0062] Specifically, when the second time step is the last time step in the time step sequence, the second denoising prediction result can be deserialized using an image embedding network to obtain video restoration features. Deserialization refers to transforming a low-dimensional vector sequence into high-dimensional latent features. Decoding the video restoration features yields the restored video frame sequence.

[0063] Specifically, the video restoration features can be upsampled using a decoder in a 3D variational autoencoder (3D VAE) to convert them into pixel space, resulting in a sequence of restored video frames. In the restored video frame sequence, candidate objects in the video frames have been removed or "hollowed out." The regions in the video frames after removing candidate objects are semantically invalid noise, blurry, or distorted content, which can be termed "holes."

[0064] Obtain the object mask sequence. Regions with a value of 1 in the object mask mark the locations of objects that need to be removed / filled. Align the object mask sequence with the repaired video frame sequence; that is, each object mask corresponds to a repaired video frame. For example, object mask 3 corresponds to video repair frame 3, and regions with a value of 1 in object mask 3 can be considered "holes" in video repair frame 3. For each repaired video frame, video repair can be performed on each repaired video frame based on its corresponding object mask, resulting in a frame-by-frame repaired video frame sequence. This frame-by-frame repaired video frame sequence can then be identified as the repaired video data corresponding to the source video data.

[0065] In this embodiment, by introducing an adaptive context adjustment mechanism based on cross-attention, visual information from the source video data is dynamically injected into the denoising process at each time step. This allows the video restoration model to flexibly and accurately extract background cues from the source video data according to the needs of the current restoration area, thereby generating high-quality restored video data that is visually highly consistent, artifact-free, and completely removes candidate objects and visual effects, thus improving the removal effect of video objects and visual effects. Simultaneously, by storing the cross-attention features of the first time step in a cache, the stored attention features (the cross-attention features of the first time step) can be directly reused in subsequent time steps, reducing runtime and improving the removal efficiency of video objects and visual effects.

[0066] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating a scene for removing video objects and visual effects according to an embodiment of this application. Figure 3 As shown, the terminal device held by the object can have video editing software installed, which integrates functions for removing video objects and visual effects. When the object wants to remove a specific object from the source video data (e.g., Figure 3 When candidate object 20a is shown, the device can open the video editing software in the terminal device and upload the source video data containing candidate object 20a and the object mask sequence of candidate object 20a in the source video data to the video encoding software.

[0067] By using the video object removal and visual effects functions in video editing software, a variational autoencoder can be invoked to perform visual encoding processing on the source video data, obtaining the video's latent features. Initial noise data is then acquired, and a video restoration model is invoked to perform denoising processing at multiple time steps, yielding the denoising prediction result for the final time step. By calling a decoder to decode the denoising prediction result of the final time step, the restored video data corresponding to the source video data can be obtained. This restored video data is then returned to the object for display on the terminal device.

[0068] It is understood that the video data processing method proposed in this application embodiment can be integrated into video editing software as a plug-in. For example, a "smart eraser brush" control can be provided in the video editing software, which has the function of removing video objects and visual effects. When the object to be removed (e.g., props that cause a scene (e.g., wires, microphones), staff members who accidentally appear on camera, or unwanted background clutter) is selected in the source video data, the "smart eraser brush" control can automatically remove the target object and perfectly repair the background.

[0069] Please see Figure 4 , Figure 4 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 2 Understandably, this video data processing method is performed by a computer device, which can be a terminal device (e.g., Figure 1 Any terminal device in the set of terminal devices shown), or a server (such as... Figure 1 The server 10d shown is not limited in this application. The video data processing method may include the following steps S201 to S211: Step S201: Obtain the source video data and the object mask sequence of the candidate objects in the source video data, and obtain the initial noise data.

[0070] Specifically, the computer device can be equipped with video editing software. When the video editing software runs on the computer device, it acquires source video data, which can be as shown in equation (1): (1) in, Represents source video data, Represents an RGB image, which is an image representation method based on the three basic color channels of red, green, and blue. This represents the total number of video frames contained in the source video data. This represents the number of color channels in each video frame. Represents the size of a video frame. Represents pixel height. Represents pixel width.

[0071] For ease of understanding, this application uses the visual segmentation model as an example of the Segment All Model (SAM-2). The specific process of segmenting the source video data using the Segment All Model (SAM-2) to obtain the object mask sequence of candidate objects (also known as objects to be removed) in the source video data includes: performing frame-by-frame processing on the source video data to obtain M video frames; extracting features from each of the M video frames to obtain the first image features of each video frame.

[0072] For example, by segmenting the source video data into frames, we obtain 20 video frames, designated as video frame 1, video frame 2, ..., video frame 20. Feature extraction is then performed on each of these 20 video frames to obtain the first image feature of each frame.

[0073] The starting video frame associated with the candidate object is obtained from 20 video frames. For example, video frame 5 is the starting video frame associated with the candidate object. In video frame 5, visual cues for the candidate object are determined. Visual cues may include, but are not limited to, point cues, bounding box cues, and mask cues. Point cues refer to the spatial coordinates of the candidate object. Bounding box cues refer to the region containing the candidate object. Mask cues refer to the object mask containing the candidate object. Encoding the visual cues yields cue encoding features.

[0074] The first image feature of video frame 5 is obtained, and the first image feature of video frame 5 is fused with the cue coding feature to generate the initial mask of the candidate object. The starting video frame, visual cue information, and initial mask are combined into mask cue information. Based on the mask cue information, the first image feature of each of the 20 video frames is fused to obtain the object mask sequence of the candidate object. The object mask sequence can be shown in equation (2): (2) in, Represents the object mask sequence.

[0075] Step S202: Perform visual encoding processing on the source video data to obtain the video's latent features and acquire the first denoising prediction result of the first time step.

[0076] Specifically, feature extraction is performed on the source video data to obtain the mean-coded features and variance-coded features. The mean-coded features and variance-coded features are then sampled to obtain the latent video features of the source video data. The first denoising prediction result for the first time step is then obtained.

[0077] For ease of understanding, this application uses a visual encoder as an example of a variational autoencoder. The specific process of obtaining latent features of the video by visually encoding the source video data using a variational autoencoder includes: performing convolution processing on the source video data to obtain the spatiotemporal features of the source video data; and performing feature mapping on the spatiotemporal features, which can map the spatiotemporal features to mean-coded features and variance-coded features in the latent space.

[0078] After obtaining the mean-coded features and variance-coded features, the mean-coded features and variance-coded features can be sampled through reparameterization to obtain the video latent features of the source video data.

[0079] Sampling the mean-coded features and variance-coded features by reparameterization can be shown in Equation (3): (3) in, The video's latent features represent the source video data. Represents the mean coding feature. Represents the standard deviation coding feature. , Represents variance-coded features. Sampled from a standard normal distribution.

[0080] Step S203: Based on the first denoising prediction result, video latent features, and object mask sequence, construct the denoising conditional data for the second time step.

[0081] The specific implementation process in step S203 can be found in the detailed description in step S103, and will not be repeated here.

[0082] Please see Figure 5 , Figure 5 This is a schematic diagram of the network structure of a video restoration model provided in an embodiment of this application. For example... Figure 5 As shown, the network structure of the video restoration model may include, but is not limited to: an image embedding network and N denoising components, each of which may include a self-attention network and a cross-attention network. This application does not limit the network structure of the video restoration model. This application uses a video restoration model network structure including an image embedding network and a denoising component (including a self-attention network and a feedforward neural network) as an example.

[0083] Assuming the initial noise denoising process comprises 10 time-step denoising steps, in each step, denoising conditional data for the current time step is constructed using the image embedding network in the video inpainting model, based on the denoising prediction result from the previous time step, latent video features, and object mask sequences. Then, using the self-attention and cross-attention networks in the video inpainting model, the denoising prediction result for the current time step is obtained based on the denoising conditional data. After all 10 time-step denoising steps are completed, the denoising prediction result for the 10th time step is obtained. Based on this result, the restored video data from the source video data can be generated.

[0084] For example, in the denoising process at the first time step, denoising conditional data for the first time step can be constructed based on the initial noise data, latent video features, and object mask sequence. Using the self-attention network and cross-attention network in the video inpainting model, the denoising prediction result for the current time step can be obtained based on the denoising conditional data from the first time step.

[0085] In the denoising process of the second time step, denoising conditional data for the second time step can be constructed based on the denoising conditional data of the first time step, the video latent features, and the object mask sequence. Using the self-attention network and cross-attention network in the video inpainting model, the denoising prediction result for the current time step can be obtained based on the denoising conditional data of the second time step.

[0086] Similarly, in the denoising process of the last time step (e.g., the 10th time step), the denoising conditional data for the last time step can be constructed based on the denoising conditional data of the 9th time step, the video latent features, and the object mask sequence. Using the self-attention network and cross-attention network in the video inpainting model, the denoising prediction result for the last time step can be obtained based on the denoising conditional data of the last time step.

[0087] It is understandable that the denoising process at each time step is similar. For ease of understanding, this application will describe the denoising process at the second time step as an example.

[0088] Step S204: Based on the first transformation weight matrix of the self-attention network in the video restoration model, the denoising conditional data is transformed into a first key matrix, and based on the second transformation weight matrix of the self-attention network, the denoising conditional data is transformed into a first value matrix.

[0089] Please see Figure 6 , Figure 6 This is a schematic diagram of the network structure of a self-attention network provided in an embodiment of this application. Figure 6As shown, assuming the self-attention network is a Transformer encoder, the network structure of the Transformer encoder may include, but is not limited to, at least one encoder component, each encoder component consisting of at least one self-attention network layer and at least one feedforward neural network layer. This application does not limit the network structure of the Transformer encoder. This application uses a Transformer encoder including one encoder component (including one self-attention network layer and one feedforward neural network layer) as an example.

[0090] like Figure 6 As shown, based on the first transformation weight matrix of the self-attention network layer in the self-attention network, the denoised conditional data at the second time step is transformed into a first key matrix (which can be denoted as K1). Based on the second transformation weight matrix of the self-attention network layer in the self-attention network, the denoised conditional data is transformed into a first value matrix (which can be denoted as V1). The first transformation weight matrix corresponding to the self-attention network layer is the parameter matrix W. k1 The second transformation weight matrix is ​​the parameter matrix W. v1 The first and second transformation weight matrices can be learned during training. Specifically, the denoised conditional data and the parameter matrix W can be used... k1 Performing a dot product yields the first key matrix K1. The denoising conditional data is then combined with the parameter matrix W in the transformation weight matrix. v1 Performing a dot product yields the first-value matrix V1.

[0091] Step S205: Generate the first query matrix based on the second time step, the denoised conditional data, and the third transformation weight matrix of the self-attention network.

[0092] Specifically, when the second time step is a background update cycle node, the denoised conditional data can be transformed into the first query matrix (which can be denoted as Q1) according to the third transformation weight matrix of the self-attention network. Specifically, the denoised conditional data and the parameter matrix W can be... q1 Performing a dot product yields the first query matrix Q1. Here, the background update period refers to the background update time, which can be determined as needed; for example, it can be 3 time steps, 2 time steps, etc., and this application does not impose any limitations on this. The background update period node refers to the time node of the background update.

[0093] For example, the denoising process for initial noisy data includes 10 time steps. Assuming the background update cycle is set to 3 time steps, then the 1st, 5th, and 9th time steps are background update cycle nodes. The 2nd, 3rd, 4th, 6th, 7th, 8th, and 10th time steps are not background update cycle nodes. For instance, when the second time step is the 1st, 5th, or 9th time step, the denoised conditional data can be transformed into the first query matrix based on the third transformation weight matrix of the self-attention network.

[0094] When the second time step is not a background update cycle node, for example, when the second time step is the 2nd, 3rd, 4th, 6th, 7th, 8th and 10th time steps, the background mask of the previous background update cycle node of the second time step can be obtained, and the first query matrix can be generated based on the background mask, the denoising condition data and the third transformation weight matrix.

[0095] For example, when the second time step is the 2nd, 3rd, or 4th time step, the preceding background update cycle node is the 1st time step. When the second time step is the 6th, 7th, or 8th time step, the preceding background update cycle node is the 5th time step. When the second time step is the 10th time step, the preceding background update cycle node is the 9th time step.

[0096] The specific process of obtaining the background mask of the previous background update cycle node of the second time step, and generating the first query matrix based on the background mask, denoising condition data, and the third transformation weight matrix, includes: obtaining the third denoising prediction result of the previous background update cycle node of the second time step. For example, when the second time step is the third time step, the denoising prediction result of the first time step can be obtained, and the cosine similarity between the video latent features and the denoising prediction result of the first time step can be obtained. The cosine similarity is then used as the similarity map between the denoising prediction result of the first time step and the video latent features.

[0097] The similarity graph can be represented as shown in equation (4): (4) in, Represents a similarity graph. This represents the third denoising prediction result. When the second time step is the third time step, the third denoising prediction result is the same as the denoising prediction result of the first time step. This represents the potential features of the video.

[0098] Thresholding and morphological dilation can transform a similarity map into a background mask from the previous background update cycle. Thresholding is a technique that binarizes an image based on pixel values, thus initially separating foreground objects from the background image. Morphological dilation is a fundamental operation in mathematical morphology used to expand the boundaries of white areas (which can also be understood as foreground objects) in an image.

[0099] The background mask can be as shown in equation (5): (5) in, Represents the background mask. This indicates slight expansion. Represents a similarity graph. , This represents the mean of the similarity map. The background mask identifies regions that remain similar before and after denoising (which can be understood as background regions).

[0100] In the denoised conditional data, the range of tokens belonging to the background region is determined. Features within this range are then replaced with a background mask to obtain the conditional update data. Based on the third transformation weight matrix, the conditional update data is transformed into the first query matrix. Specifically, the conditional update data and the parameter matrix W can be... q1 Dot product yields the first query matrix Q1. The third transformation weight matrix can be learned during training.

[0101] Understandably, based on the background mask and background cache update strategy, in the denoising process at each time step, the tokens belonging to the foreground object in the denoising condition data of each time step can be calculated and updated normally; for the tokens belonging to the background region in the denoising condition data of each time step, they are updated once every background update cycle (e.g., every 3 time steps). That is, when the current time step is not a background update cycle node, the tokens belonging to the background region in the denoising condition data of the current time step can directly replace the background mask of the previous background update cycle node. This reduces redundant calculations for static background regions, accelerates the denoising process, and improves the efficiency of removing video objects and visual effects.

[0102] Step S206: Perform a dot product operation between the first query matrix and the transpose of the first key matrix to obtain the candidate weight matrix.

[0103] like Figure 6As shown, after obtaining the first query matrix, the first key matrix, and the first value matrix, the candidate weight matrix (which can be denoted as ) can be obtained by performing a dot product operation between the transpose of the first query matrix Q1 and the first key matrix K1 through a self-attention network layer in a self-attention network. The candidate weight matrix can be the inner product (also known as dot product) of each row vector in the first query matrix Q1 and the first key matrix K1.

[0104] Step S207: Obtain the number of columns corresponding to the first query matrix, normalize the ratio between the candidate weight matrix and the square root of the number of columns to obtain the attention weight matrix, and obtain the first self-attention feature of the second time step based on the dot product between the attention weight matrix and the first value matrix.

[0105] Specifically, to prevent the inner product from becoming too large, we can obtain the number of columns corresponding to the first query matrix Q1 (the spatial query matrix Q1 and the first key matrix K1 have the same number of columns, which can also be called the vector dimension). The candidate weight matrix is ​​then multiplied by the square root of the number of columns (which can be denoted as...). Normalizing the ratio between the two values ​​yields the attention weight matrix.

[0106] Candidate latent features are obtained by the dot product between the attention weight matrix and the first value matrix V1. The attention weight matrix can be represented as follows: The softmax function is used for normalization. It can be used to calculate the self-attention coefficient of a single feature with respect to other features. Each row in the matrix is ​​softmaxed. The dot product between the attention weight matrix and the first value matrix V1 is determined as the output feature of the self-attention network (which can be represented as O1=). In this case, the output features of the self-attention network layer can be used as candidate latent features.

[0107] like Figure 6 As shown, the candidate latent features are input into the feedforward neural network layer, and the feedforward neural network layer performs feature transformation on the candidate latent features. For example, the hidden weight matrix and bias vector corresponding to the feedforward neural network layer can be obtained. Based on the dot product between the bias vector and the hidden weight matrix and the candidate latent features, the first self-attention feature of the second time step is obtained.

[0108] Step S208: Obtain the time step sequence of the initial noise data. When the first time step is the first time step in the time step sequence, obtain the second self-attention feature of the first time step. Perform cross-attention processing on the second self-attention feature and the video latent feature to obtain the cross-attention feature of the first time step.

[0109] Specifically, to obtain the time step sequence of the initial noisy data, assuming the denoising process of the initial noisy data includes 10 time steps, these 10 time steps can be combined into a time step sequence. For example, if the 10 time steps are time step 1, time step 2, time step 3, ..., time step 10, then the time step sequence is [time step 1, time step 2, time step 3, ..., time step 10]. Time step 1 in the time step sequence can be called the first time step in the time step sequence.

[0110] When the first time step is the first time step (time step 1) in the time step sequence, the second self-attention feature of time step 1 can be obtained. The second self-attention feature and the video latent feature are cross-attention processed to obtain the cross-attention feature of the first time step (time step 1).

[0111] Please see Figure 7 , Figure 7 This is a schematic diagram of the network structure of a cross-attention network provided in an embodiment of this application. Figure 7 As shown, assuming the cross-attention network is a Transformer encoder, the network structure of the Transformer encoder may include, but is not limited to, at least one encoder component, each encoder component consisting of at least one cross-attention network layer and at least one feedforward neural network layer. This application does not limit the network structure of the Transformer encoder. This application uses a Transformer encoder including one encoder component (including one cross-attention network layer and one feedforward neural network layer) as an example.

[0112] like Figure 7 As shown, based on the parameter matrix W in the cross-attention network layer q2 Parameter matrix W k2 and parameter matrix W v2 The second self-attention feature is combined with the parameter matrix W q2 Performing a dot product yields the second query matrix Q2; combining the video latent features with the parameter matrix W k2 Dot product yields the first key matrix K2; the video latent features are then combined with the parameter matrix W. v2 Performing a dot product yields the first value matrix V2. Each query vector in the second query matrix described above can be used to encode the similarity relationship between each feature and other features, which determines the dependency information between that feature and its preceding features. The parameter matrix W in the cross-attention network layer... q2 Parameter matrix W k2 and parameter matrix W v2 Learning can be done during the training process.

[0113] like Figure 7 As shown, after obtaining the second query matrix, the second key matrix, and the second value matrix, the cross-attention features of the first time step can be obtained through a cross-attention network based on the second query matrix, the second key matrix, and the second value matrix. The cross-attention features of the first time step (time step 1) can be stored in the cache space.

[0114] The specific process of obtaining the cross-attention features of the first time step based on the second query matrix, the second key matrix, and the second value matrix is ​​similar to the specific process of obtaining the first self-attention features of the second time step based on the first query matrix, the first key matrix, and the first value matrix. For details, please refer to the detailed description in steps S206 to S207, which will not be repeated here.

[0115] Step S209: When the first time step is not the first time step in the time step sequence, the stored attention features in the cache space are determined as the cross-attention features of the first time step.

[0116] Specifically, when the first time step is not the first time step in the time step sequence, for example, when the first time step is time step 2, time step 3, ..., time step 10, the attention features already stored in the cache space (which can be understood as the cross attention features of time step 1) can be directly identified as the cross attention features of the first time step.

[0117] Understandably, since the cross-attention features are consistent across all time steps, a context-based cache update strategy can be used. In the denoising process of time step 1, latent video features are adaptively introduced. A cross-attention network is then used to perform cross-attention processing on the self-attention features and latent video features of time step 1, resulting in the cross-attention features for time step 1, which are then stored in the cache. In subsequent time steps, the self-attention features of time step 1 from the cache can be directly reused, avoiding the repetitive calculation of cross-attention features in each time step's denoising process. This accelerates the denoising process and improves the efficiency of removing video objects and visual effects.

[0118] Step S210: Generate the second denoising prediction result for the second time step based on the first self-attention feature and the cross-attention feature.

[0119] Specifically, the attention weight parameters are obtained, and the product of the attention weight parameters and the cross-attention features is determined as the candidate attention features. The candidate attention features are added to the first self-attention features to obtain the second denoising prediction result at the second time step.

[0120] Assuming the second time step is denoted as time step t, the second denoising prediction result at time step t can be expressed as shown in equation (6): (6) in, This represents the second denoising prediction result at the second time step. The first self-attention feature representing the second time step. Represents the attention weight parameter. Cross-attention features representing the first time step.

[0121] Step S211: Based on the second denoising prediction result, generate the repaired video data corresponding to the source video data.

[0122] The specific implementation process in step S211 can be found in the detailed description in step S105, and will not be repeated here.

[0123] In this embodiment, by introducing an adaptive context adjustment mechanism based on cross-attention, latent video features can be dynamically injected into the denoising process at each time step. This allows the video restoration model to flexibly and accurately extract background cues from the source video data according to the immediate needs of denoising at each time step, thereby generating visually consistent, artifact-free, and thoroughly removed candidate objects and visual effects restoration video data, improving the removal effect of video objects and visual effects. Simultaneously, based on a background and context caching (BC-Cache) strategy, which includes background cache update strategies and context cache update strategies, background masks and cross-attention features can be adaptively cached. During denoising at each time step, the cached background masks and cross-attention features can be reused, avoiding redundant calculations. While ensuring the removal effect remains unchanged, approximately 2.2 times inference speedup can be achieved, improving the removal efficiency of video objects and visual effects.

[0124] Please see Figure 8 , Figure 8 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 3 Understandably, this video data processing method is performed by a computer device, which can be a terminal device (e.g., Figure 1 Any terminal device in the set of terminal devices shown), or a server (such as... Figure 1 The server 10d shown is not limited in this application. The video data processing method may include the following steps S301 to S306: Step S301: Obtain training samples and sample noise data. The training samples include sample video data, target video data, and object sample mask sequences of sample objects in the sample video data.

[0125] In one or more embodiments, when the training samples are synthetic samples, the specific process of obtaining the training samples includes: obtaining a foreground object set and a background image set. Assume the foreground object set includes 10 foreground objects, namely foreground object 1, foreground object 2, foreground object 3, foreground object 4, ..., foreground object 10. The background image set includes 5 background images, namely background image 1, background image 2, background image 3, background image 4, and background image 5.

[0126] The foreground objects in the foreground object set and the background images in the background image set are combined to obtain K pairing groups. For example, background image 2 can be combined with foreground object 3 and foreground object 4 into one pairing group, or background image 4 can be combined with foreground object 1 into one pairing group; this application does not limit the specific pairing group. Each pairing group can contain at least one foreground object and one background image.

[0127] Assume K is 5, and the five pairing groups are pairing group 1, pairing group 2, pairing group 3, pairing group 4, and pairing group 5. Pairing group 1 includes background image 1 and foreground object 1, foreground object 7, and foreground object 10. Pairing group 2 includes background image 2 and foreground object 1 and foreground object 4. Pairing group 3 includes background image 3 and foreground object 6. Pairing group 4 includes background image 4 and foreground object 3, foreground object 8, and foreground object 10. Pairing group 5 includes background image 5 and foreground object 2, foreground object 5, and foreground object 9.

[0128] The foreground and background images in each pairing group are rendered to obtain K sample video frames. For example, background image 1, foreground object 1, foreground object 7, and foreground object 10 in pairing group 1 are rendered to obtain sample video frame 1. Background image 2, foreground object 1, and foreground object 4 in pairing group 2 are rendered to obtain sample video frame 2. ..., background image 5, foreground object 2, foreground object 5, and foreground object 9 in pairing group 5 are rendered to obtain sample video frame 5.

[0129] Five sample video frames (sample video frame 1, sample video frame 2, ..., sample video frame 5) are combined to form sample video data. The sample object is identified from the foreground objects contained in the sample video data. Assuming the sample object is foreground object 1, it needs to be removed from sample video frames 1 and 2 to obtain the target video data. The object sample mask sequence of the sample object in the sample video data is obtained. The sample video data, target video data, and object sample mask sequence are then used as training samples.

[0130] The specific process of obtaining the object sample mask sequence of the sample object in the sample video data is similar to the specific process of obtaining the object mask sequence of the candidate object in the source video data. For details, please refer to the detailed description in step S201, which will not be repeated here.

[0131] Optionally, training samples can include synthetic samples and real samples. Real samples refer to samples consisting of pixel-aligned pairs of video frames, one with an object and one without, acquired by actually introducing and removing real objects. The sample video data in real samples can consist of video frames with objects. The target video data in real samples can consist of video frames without objects.

[0132] Obtain sample noise data, which refers to random noise obtained by "adding noise" to the target video data. The sample noise data can be represented as shown in equation (7): (7) in, , , Represents the total number of time steps. It is Gaussian noise. The video latent features representing the sample video data.

[0133] Step S302: Perform visual encoding processing on the sample video data to obtain the sample latent features and obtain the first sample prediction result at the first time step; the first sample prediction result refers to the prediction result of the sample noise data in the denoising processing at the first time step.

[0134] Please see Figure 9 , Figure 9 This is a schematic diagram of a model training process provided in an embodiment of this application. For example... Figure 9As shown, the network structure of the initial inpainting model includes, but is not limited to, an image embedding network and N denoising components. Each denoising component may include a self-attention network and a cross-attention network. After the computer device acquires the sample video data, sample noise data, and object sample mask sequence, it inputs these data into the initial inpainting model. The initial inpainting model performs denoising processing at multiple time steps, outputting the sample prediction result for the last time step. Decoding the sample prediction result yields the predicted video data.

[0135] For example, taking the denoising process of the second time step using the initial repair model as an example, the sample video data is visually encoded using a variational autoencoder to obtain the sample latent features, thus obtaining the first sample prediction result of the first time step. The specific process of obtaining the sample latent features by visually encoding the sample video data using a variational autoencoder is similar to the specific process of obtaining the video latent features by visually encoding the source video data using a variational autoencoder. The specific process of obtaining the first sample prediction result of the first time step is similar to the specific process of obtaining the first denoising prediction result of the first time step; please refer to the detailed description in step S202 for details, which will not be repeated here.

[0136] Step S303: Based on the prediction results of the first sample, the latent features of the sample, and the object sample mask sequence, construct the sample condition data for the second time step; the second time step refers to the time step after the first time step.

[0137] like Figure 9 As shown, the image embedding network can serialize the first sample prediction result, sample latent features, and object sample mask sequence separately to obtain the sample denoising sequence corresponding to the first sample prediction result and the sample latent feature sequence corresponding to the sample latent features. Combining the sample denoising sequence, sample latent feature sequence, and object sample mask sequence yields the sample conditional data for the second time step.

[0138] The specific process of constructing the sample conditional data for the second time step based on the first sample prediction result, sample latent features, and object sample mask sequence is similar to the specific process of constructing the denoising conditional data for the second time step based on the first denoising prediction result, video latent features, and object mask sequence. For details, please refer to the detailed description in step S103, which will not be repeated here.

[0139] Step S304: Perform self-attention processing on the sample conditional data through the initial repair model to obtain the self-attention prediction features of the second time step, obtain the cross-attention prediction features of the first time step, and generate the second sample prediction result of the second time step based on the self-attention prediction features and the cross-attention prediction features; the cross-attention prediction features refer to the fusion features that include visual information of the sample video data.

[0140] like Figure 9 As shown, by applying self-attention to the sample conditional data through the self-attention network in the initial repair model, the self-attention prediction features for the second time step can be obtained. When the first time step is the first time step in the time step sequence, the self-attention prediction features and sample latent features of the first time step can be cross-attention processed through the cross-attention network in the initial repair model to obtain the cross-attention prediction features for the first time step. When the first time step is not the first time step in the time step sequence, the attention prediction features stored in the cache space can be used as the cross-attention prediction features for the first time step. Based on the self-attention prediction features and the cross-attention prediction features, the second sample prediction result for the second time step is generated.

[0141] The specific process of performing self-attention processing on the sample conditional data to obtain the self-attention prediction features of the second time step is similar to the specific process of performing self-attention processing on the denoised conditional data to obtain the first self-attention features of the second time step; the specific process of obtaining the cross-attention prediction features of the first time step is similar to the specific process of obtaining the cross-attention features of the first time step; the specific process of generating the second sample prediction result of the second time step based on the self-attention prediction features and the cross-attention prediction features is similar to the specific process of generating the second denoised prediction result of the second time step based on the first self-attention features and the cross-attention features. For details, please refer to the detailed description in steps S205 to S210, which will not be repeated here.

[0142] Step S305: Based on the prediction results of the second sample, generate the predicted video data corresponding to the sample video data.

[0143] like Figure 9 As shown, when the second time step is the last time step in the time step sequence, the prediction result of the second sample can be deserialized to obtain the repaired prediction features. The repaired prediction features are then input into the decoder. By decoding the repaired prediction features, the predicted video data corresponding to the sample video data can be obtained.

[0144] The specific process of generating the predicted video data corresponding to the sample video data based on the second sample prediction result is similar to the specific process of generating the repaired video data corresponding to the source video data based on the second denoising prediction result. For details, please refer to the detailed description in step S105, which will not be repeated here.

[0145] Step S306: Based on the target video data and the predicted video data, the network parameters of the initial repair model are corrected to obtain the video repair model; the video repair model is used to remove candidate objects and their visual effects from the source video data based on the object mask sequence of the candidate objects in the source video data.

[0146] like Figure 9 As shown, the label movement speed between the sample noise data and the target video data is obtained, and the path movement speed between the sample noise data and the predicted video data is obtained. Based on the label movement speed and the path movement speed, the model loss of the initial restoration model is determined. Based on the model loss, the network parameters of the initial restoration model are corrected to obtain the video restoration model.

[0147] In this context, the target video data can be considered as a noise-free video data distribution, specifically the target data distribution of the sample noise data. The tag movement speed can be the flow speed from the sample noise data to the target video data. The path movement speed can refer to the flow speed from the sample noise data to the predicted video data at time step t, and can be a function of time step t and the predicted video data; t is a value uniformly sampled from the range [0,1]. The model loss refers to the loss of the video inpainting model calculated using the flow matching generation paradigm. The model loss can be calculated using the loss function of the flow matching generation paradigm, which can be of a general form, and will not be described in detail in this embodiment.

[0148] The specific process of correcting the network parameters of the initial restoration model based on the model loss to obtain the video restoration model may include: determining the gradient of the initial restoration model based on the model loss, and correcting the network parameters of the initial restoration model based on the gradient; if the model loss reaches the minimum value, the initial restoration model containing the corrected network parameters can be determined as the video restoration model.

[0149] The gradient is a vector representing the point in time where the model loss changes most rapidly (i.e., at its highest rate of change). The gradient helps us find the direction that minimizes the model loss. Gradient calculation refers to using the backpropagation algorithm to calculate the gradient of the model loss with respect to the model parameters. The backpropagation algorithm is a calculation method based on the chain rule.

[0150] After obtaining the gradient, the network parameters of the initial repair model can be updated using gradient descent. Gradient descent is an optimization algorithm based on gradients, which can include, but is not limited to, batch gradient descent (BGD), stochastic gradient descent (SGD), and mini-batch gradient descent (MBGD). This application does not limit the specific gradient descent method used. Batch gradient descent uses the entire training set to calculate the gradient and update the model's network parameters. Stochastic gradient descent uses only one sample or a mini-batch of samples in each iteration to calculate the gradient and update the model's network parameters. Mini-batch gradient descent uses a fixed number of samples in each iteration to calculate the gradient and update the model's network parameters.

[0151] Specifically, based on the model loss, the gradient of the model loss with respect to the output layer in the initial repair model is calculated. Furthermore, the gradient is backpropagated layer by layer to each layer of the neural network in the initial repair model using the chain rule. The gradient of the network parameters of each layer of the neural network is calculated to obtain the gradient of the network parameters of each layer of the neural network in the initial repair model. Based on the gradient, the network parameters in the initial repair model are updated using a gradient descent algorithm (e.g., stochastic gradient descent).

[0152] During the training of the initial repair model, back-propagation (BP) can be performed based on the model loss to calculate the gradient of the model loss with respect to the network parameters of each layer of the neural network in the initial repair model. The network parameters of the initial repair model can be iteratively adjusted according to the gradient. By minimizing the model loss, the network parameters of the initial repair model can be iteratively updated.

[0153] Understandably, the training phase of the initial restoration model can include multiple epochs of iterative training. Each epoch involves inputting at least one training sample into the initial restoration model to perform forward computation and obtain the sample prediction results. During the model training phase, when the initial restoration model has reached the pre-set maximum number of iterations, or when the model loss obtained from the previous iteration is the same as the model loss obtained from the next iteration, the model loss can be considered to have reached its minimum. Training is then stopped, and the model parameters at the minimum loss are saved. At this point, the initial restoration model can be considered the successfully trained video restoration model.

[0154] In this embodiment, by combining a compositing engine with live-action footage, training samples containing accurate foreground objects and visual effects (shadows, reflections) are provided. This allows the initial restoration model to undergo specialized training based on the training samples, mastering the ability to remove visual effects under complex lighting conditions. This improves the initial restoration model's ability to handle visual effects, thereby enhancing the removal effect of video objects and visual effects.

[0155] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a video data processing device provided in an embodiment of this application. Figure 10 As shown, the video data processing device 1 may include: a first acquisition module 101, a first encoding module 102, a first construction module 103, a first processing module 104, and a first generation module 105; The first acquisition module 101 is used to acquire source video data and the object mask sequence of candidate objects in the source video data, and to acquire initial noise data; The first encoding module 102 is used to perform visual encoding processing on the source video data to obtain video latent features and acquire the first denoising prediction result at the first time step; the first denoising prediction result refers to the prediction result of the initial noise data in the denoising processing at the first time step. The first construction module 103 is used to construct the denoising conditional data for the second time step based on the first denoising prediction result, video latent features, and object mask sequence; the second time step refers to the time step after the first time step. The first processing module 104 is used to perform self-attention processing on the denoised conditional data to obtain the first self-attention feature of the second time step, obtain the cross-attention feature of the first time step, and generate the second denoising prediction result of the second time step based on the first self-attention feature and the cross-attention feature; the cross-attention feature refers to the fusion feature that includes visual information of the source video data. The first generation module 105 is used to generate repaired video data corresponding to the source video data based on the second denoising prediction result; the repaired video data refers to video data from the source video data after removing the candidate objects and the visual effects of the candidate objects.

[0156] The first acquisition module 101 acquires the source video data and the object mask sequence of candidate objects in the source video data, and performs the following steps: Acquire source video data, perform frame segmentation on the source video data to obtain M video frames, extract features from each of the M video frames to obtain the first image feature of each video frame; M is a positive integer. In M video frames, the starting video frame related to the candidate object is obtained. In the starting video frame, the visual cue information of the candidate object is determined. The visual cue information is encoded to obtain the cue encoding feature. The second image features of the starting video frame are obtained, and the cue encoding features and the second image features are fused to generate the initial mask of the candidate object. The starting video frame, visual cue information, and initial mask are combined into mask cue information. Based on the mask cue information and the first image features of each video frame, an object mask sequence of candidate objects is generated.

[0157] The first encoding module 102 performs visual encoding processing on the source video data to obtain latent video features, which are used to perform the following steps: The source video data is extracted by a variational autoencoder to obtain the mean coding features and variance coding features of the source video data. By sampling the mean-coded features and variance-coded features, the latent video features of the source video data are obtained.

[0158] The first construction module 103 constructs denoising conditional data for the second time step based on the first denoising prediction result, video latent features, and object mask sequence, for performing the following steps: The first denoised prediction result is serialized to obtain a denoised prediction sequence, and the video latent features are serialized to obtain a video latent feature sequence. The denoising prediction sequence, the video latent feature sequence, and the object mask sequence are combined into the denoising conditional data for the second time step.

[0159] The first processing module 104 performs self-attention processing on the denoised conditional data to obtain the first self-attention feature at the second time step, which is used to perform the following steps: Based on the first transformation weight matrix of the self-attention network in the video restoration model, the denoising conditional data is transformed into the first key matrix, and based on the second transformation weight matrix of the self-attention network, the denoising conditional data is transformed into the first value matrix. The first query matrix is ​​generated based on the second time step, the denoised conditional data, and the third transformation weight matrix of the self-attention network. Perform a dot product operation between the first query matrix and the transpose of the first key matrix to obtain the candidate weight matrix; Obtain the number of columns corresponding to the first query matrix, normalize the ratio between the candidate weight matrix and the square root of the number of columns to obtain the attention weight matrix, and obtain the first self-attention feature of the second time step based on the dot product between the attention weight matrix and the first value matrix.

[0160] The first processing module 104 generates a first query matrix based on the second time step, the denoised conditional data, and the third transformation weight matrix of the self-attention network, for performing the following steps: When the second time step is a background update cycle node, the denoised conditional data is converted into the first query matrix according to the third transformation weight matrix of the self-attention network. When the second time step is not a background update cycle node, obtain the background mask of the previous background update cycle node of the second time step, and generate the first query matrix based on the background mask, denoising condition data and the third transformation weight matrix.

[0161] The first processing module 104 obtains the background mask of the previous background update cycle node of the second time step, and generates a first query matrix based on the background mask, denoising condition data, and third transformation weight matrix, for performing the following steps: Obtain the third denoising prediction result of the previous background update cycle node in the second time step; Obtain the similarity map between the third denoising prediction result and the video latent features, and generate the background mask of the previous background update cycle node based on the similarity map; The denoising conditional data is updated based on the background mask to obtain conditional update data. The conditional update data is then converted into the first query matrix based on the third transformation weight matrix.

[0162] The first processing module 104 obtains the cross-attention features at the first time step and uses them to perform the following steps: Obtain the time-step sequence of the initial noise data; When the first time step is the first time step in the time step sequence, the second self-attention feature of the first time step is obtained, and the second self-attention feature and the video latent feature are cross-attention processed to obtain the cross-attention feature of the first time step. When the first time step is not the first time step in the time step sequence, the attention features already stored in the cache space are identified as the cross-attention features of the first time step.

[0163] The first processing module 104 performs cross-attention processing on the second self-attention feature and the video latent feature to obtain the cross-attention feature at the first time step, which is used to perform the following steps: By using the cross-attention network in the video restoration model, the second self-attention feature is transformed into the second query matrix, and the video latent feature is transformed into the second key matrix and the second value matrix. Based on the second query matrix, the second key matrix, and the second value matrix, the cross-attention features of the first time step are obtained.

[0164] The first processing module 104 generates a second denoising prediction result for the second time step based on the first self-attention feature and the cross-attention feature, which is used to perform the following steps: Obtain the attention weight parameters, and determine the candidate attention features by multiplying the attention weight parameters with the cross-attention features; The candidate attention features and the first self-attention features are added together to obtain the second denoising prediction result at the second time step.

[0165] The first generation module 105 generates repaired video data corresponding to the source video data based on the second denoising prediction result, for use in performing the following steps: When the second time step is the last time step in the time step sequence, the second denoising prediction result is deserialized to obtain the video restoration features. The video restoration features are decoded to obtain the restored video frame sequence; Perform video restoration on the sequence of video frames to generate restored video data corresponding to the source video data.

[0166] According to one embodiment of this application, the foregoing Figure 2 and Figure 4 The data processing method shown can be related to the steps involved by Figure 10 The video data processing device 1 shown is used to execute the various modules. Figure 2 The step S101 shown can be performed by Figure 10 The first acquisition module 101 shown is used to execute this. Figure 2 The step S102 shown can be performed by Figure 10 The first encoding module 102 shown is used to execute this. Figure 2 The step S103 shown can be performed by Figure 10 The first building block 103 shown is used for execution. Figure 2 Step S104 shown can be derived from Figure 10 The first processing module 104 shown is used to execute this. Figure 2 The step S105 shown can be performed by Figure 10 The first generation module 105 shown is used to perform the above.

[0167] According to one embodiment of this application, Figure 10 The various modules in the video data processing device 1 shown can be individually or all combined into one or more modules, or some of the modules can be further divided into at least two functionally smaller units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by at least two units, or the function of at least two modules can be implemented by one module.

[0168] In this embodiment, an adaptive context adjustment mechanism can dynamically query the latent video features of the source video data during denoising at each time step. This allows the denoising process at each time step to accurately extract the texture and lighting information of the occluded areas from the source video data. The generated filling content blends seamlessly with the original background in terms of color, texture, and lighting, eliminating visual seams and blurring, thus improving the removal effect of video objects and visual effects. Furthermore, the background and context caching strategy reduces runtime, further enhancing the efficiency of removing video objects and visual effects.

[0169] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a video data processing device provided in an embodiment of this application. Figure 11 As shown, the video data processing device 2 may include: a second acquisition module 201, a second encoding module 202, a second construction module 203, a second processing module 204, a second generation module 205, and a parameter correction module 206. The second acquisition module 201 is used to acquire training samples and sample noise data. The training samples include sample video data, target video data, and object sample mask sequences of sample objects in the sample video data. The second encoding module 202 is used to perform visual encoding processing on the sample video data to obtain the sample latent features and acquire the first sample prediction result at the first time step; the first sample prediction result refers to the prediction result of the sample noise data in the denoising process at the first time step. The second construction module 203 is used to construct sample condition data for the second time step based on the first sample prediction result, sample latent features, and object sample mask sequence; the second time step refers to the time step after the first time step. The second processing module 204 is used to perform self-attention processing on the sample conditional data through the initial repair model to obtain the self-attention prediction features of the second time step, obtain the cross-attention prediction features of the first time step, and generate the second sample prediction result of the second time step based on the self-attention prediction features and the cross-attention prediction features; the cross-attention prediction features refer to the fusion features that include visual information of the sample video data. The second generation module 205 is used to generate predicted video data corresponding to the sample video data based on the prediction results of the second sample. The parameter correction module 206 is used to correct the network parameters of the initial repair model based on the target video data and the predicted video data to obtain the video repair model. The video repair model is used to remove candidate objects and their visual effects from the source video data based on the object mask sequence of the candidate objects in the source video data.

[0170] The second acquisition module 201 acquires training samples and performs the following steps: Obtain the foreground object set and the background image set. Combine the foreground objects in the foreground object set and the background images in the background image set to obtain K pairs; K is a positive integer. The foreground object and background image in each pairing group are rendered to obtain K sample video frames, and the K sample video frames are combined into sample video data. The sample objects are identified from the foreground objects contained in the sample video data, and the sample objects in the sample video data are removed to obtain the target video data. Obtain the object sample mask sequence of the sample object in the sample video data, and determine the sample video data, target video data, and object sample mask sequence as training samples.

[0171] The parameter correction module 206 corrects the network parameters of the initial restoration model based on the target video data and the predicted video data to obtain a video restoration model, which is used to perform the following steps: Obtain the label movement speed between the sample noise data and the target video data, and obtain the path movement speed between the sample noise data and the predicted video data; The model loss of the initial repair model is determined based on the label movement speed and the path movement speed. Based on the model loss, the network parameters of the initial restoration model are corrected to obtain the video restoration model.

[0172] According to one embodiment of this application, the foregoing Figure 8 The data processing method shown can be related to the steps involved by Figure 11 The video data processing device 2 shown in the diagram executes the various modules. Figure 8 The step S301 shown can be performed by Figure 11 The second acquisition module 201 shown is used to execute this. Figure 8 The step S302 shown can be performed by Figure 11 The second encoding module 202 shown is used to execute this. Figure 8 The step S303 shown can be performed by Figure 11 The second building block 203 shown is used for execution. Figure 8 Step S34 shown can be derived from Figure 11 The second processing module 204 shown is used to execute this. Figure 8 The step S305 shown can be performed by Figure 11 The second generation module 205 shown is used to execute this. Figure 8 The step S306 shown can be performed by Figure 11 The parameter correction module 206 shown is used to perform the above.

[0173] According to one embodiment of this application, Figure 11 The various modules in the video data processing device 2 shown can be individually or all combined into one or more modules, or some of the modules can be further divided into at least two functionally smaller units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by at least two units, or the function of at least two modules can be implemented by one module.

[0174] In this embodiment, the initial restoration model is trained using constructed training samples, enabling the video restoration model to understand complex light and shadow relationships. It can not only remove video objects but also automatically identify and completely erase visual effects that move with the video objects (e.g., dynamic shadows, specular reflections), filling in the correct background lighting effects, thus improving the removal effect of video objects and visual effects.

[0175] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 12 As shown, the computer device 1000 can be a terminal device, for example, the one described above. Figure 1 The terminal device 10a in the corresponding embodiment can also be a server, for example, as described above. Figure 1 The server 10d in the corresponding embodiment will not be limited here. For ease of understanding, this application takes a computer device as an example as the terminal device. The computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The memory 1005 may also optionally be at least one storage device located away from the aforementioned processor 1001. Figure 12 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0176] The network interface 1004 in the computer device 1000 can also provide network communication functions, and the optional user interface 1003 can also include a display screen and a keyboard. Figure 12 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve: Obtain the source video data and the object mask sequence of candidate objects in the source video data, and obtain the initial noise data; Visual encoding is performed on the source video data to obtain the video's latent features, and the first denoising prediction result at the first time step is obtained; the first denoising prediction result refers to the prediction result of the initial noise data in the denoising process at the first time step. Based on the first denoising prediction result, video latent features, and object mask sequence, denoising conditional data for the second time step is constructed; the second time step refers to the time step after the first time step. Self-attention processing is performed on the denoised conditional data to obtain the first self-attention feature of the second time step. The cross-attention feature of the first time step is obtained. Based on the first self-attention feature and the cross-attention feature, the second denoising prediction result of the second time step is generated. The cross-attention feature refers to the fusion feature that includes visual information from the source video data. Based on the second denoising prediction result, repaired video data corresponding to the source video data is generated; repaired video data refers to video data from the source video data after removing candidate objects and their visual effects.

[0177] Or to achieve: Acquire training samples and sample noise data. The training samples include sample video data, target video data, and object sample mask sequences of sample objects in the sample video data. Visual encoding is performed on the sample video data to obtain the sample latent features and to obtain the first sample prediction result at the first time step. The first sample prediction result refers to the prediction result of the sample noise data in the denoising process at the first time step. Based on the prediction results of the first sample, the latent features of the sample, and the object sample mask sequence, the sample condition data for the second time step is constructed; the second time step refers to the time step after the first time step. The sample conditional data is processed by self-attention using the initial repair model to obtain the self-attention prediction features of the second time step, and the cross-attention prediction features of the first time step are obtained. Based on the self-attention prediction features and the cross-attention prediction features, the second sample prediction result of the second time step is generated. The cross-attention prediction features refer to the fusion features that include visual information of the sample video data. Based on the prediction results of the second sample, generate the predicted video data corresponding to the sample video data; Based on the target video data and the predicted video data, the network parameters of the initial restoration model are corrected to obtain the video restoration model. The video restoration model is used to remove candidate objects and their visual effects from the source video data based on the object mask sequence of the candidate objects in the source video data.

[0178] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 2 , Figure 4 , Figure 8 The description of the video data processing method in any of the embodiments can also be performed using the methods described above. Figure 10 The description of the video data processing apparatus 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0179] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned video data processing device 1 or video data processing device 2. The computer program includes computer instructions, and when the processor executes the computer instructions, it can execute the aforementioned... Figure 2 , Figure 4 , Figure 8 The description of the video data processing method in any of the embodiments is already provided, and therefore will not be repeated here. Similarly, the beneficial effects of using the same method will not be repeated here either. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed and executed on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0180] Furthermore, it should be noted that this application also provides a computer program product, which may include a computer program that can be stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium, and the processor can execute the computer program, causing the computer device to perform the aforementioned... Figure 2 , Figure 4 , Figure 8 The description of the video data processing method in any of the embodiments is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0181] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different media content, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0182] It is understood that in the description of the embodiments of this application, "when," "if," and "if" all refer to the computer device making corresponding processing under certain objective circumstances, and are not limited to a time, nor do they require the computer device to make a judgment action, nor do they mean that there are other limitations.

[0183] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0184] The methods and related apparatuses provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0185] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0186] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A video data processing method, characterized in that, include: Obtain source video data and the object mask sequence of candidate objects in the source video data, and obtain initial noise data; The source video data is subjected to visual encoding processing to obtain video latent features, and a first denoising prediction result is obtained at the first time step; the first denoising prediction result refers to the prediction result of the initial noise data in the denoising processing at the first time step. Based on the first denoising prediction result, the video latent features, and the object mask sequence, denoising conditional data for the second time step is constructed; the second time step refers to the time step after the first time step. The denoised conditional data is subjected to self-attention processing to obtain the first self-attention feature of the second time step, and the cross-attention feature of the first time step is obtained. Based on the first self-attention feature and the cross-attention feature, the second denoised prediction result of the second time step is generated. The cross-attention feature refers to the fusion feature that includes the visual information of the source video data. Based on the second denoising prediction result, the repaired video data corresponding to the source video data is generated; The repaired video data refers to video data from the source video data that removes the candidate object and the visual effects of the candidate object.

2. The method according to claim 1, characterized in that, The acquisition of source video data and the object mask sequence of candidate objects in the source video data includes: Acquire source video data, perform frame segmentation on the source video data to obtain M video frames, extract features from each of the M video frames to obtain the first image feature of each video frame; M is a positive integer; From the M video frames, obtain the starting video frame related to the candidate object; in the starting video frame, determine the visual cue information of the candidate object; encode the visual cue information to obtain cue encoding features. The second image feature of the starting video frame is obtained, and the prompt encoding feature and the second image feature are fused to generate the initial mask of the candidate object. The starting video frame, the visual cue information, and the initial mask are combined into mask cue information. Based on the mask cue information and the first image features of each video frame, an object mask sequence of the candidate object is generated.

3. The method according to claim 1, characterized in that, The step of performing visual encoding processing on the source video data to obtain latent video features includes: The source video data is feature extracted by a variational autoencoder to obtain the mean coding feature and variance coding feature of the source video data. The mean coding feature and the variance coding feature are sampled to obtain the video latent features of the source video data.

4. The method according to claim 1, characterized in that, The step of constructing denoising conditional data for the second time step based on the first denoising prediction result, the video latent features, and the object mask sequence includes: The first denoising prediction result is serialized to obtain a denoising prediction sequence, and the video latent features are serialized to obtain a video latent feature sequence. The denoising prediction sequence, the video latent feature sequence, and the object mask sequence are combined into the denoising conditional data for the second time step.

5. The method according to claim 1, characterized in that, The step of performing self-attention processing on the denoised conditional data to obtain the first self-attention feature at the second time step includes: Based on the first transformation weight matrix of the self-attention network in the video restoration model, the denoising conditional data is transformed into a first key matrix, and based on the second transformation weight matrix of the self-attention network, the denoising conditional data is transformed into a first value matrix. A first query matrix is ​​generated based on the second time step, the denoising conditional data, and the third transformation weight matrix of the self-attention network. The candidate weight matrix is ​​obtained by performing a dot product operation between the first query matrix and the transpose of the first key matrix. Obtain the number of columns corresponding to the first query matrix, normalize the ratio between the candidate weight matrix and the square root of the number of columns to obtain the attention weight matrix, and obtain the first self-attention feature of the second time step based on the dot product between the attention weight matrix and the first value matrix.

6. The method according to claim 5, characterized in that, The step of generating the first query matrix based on the second time step, the denoising conditional data, and the third transformation weight matrix of the self-attention network includes: When the second time step is a background update cycle node, the denoising conditional data is converted into a first query matrix according to the third transformation weight matrix of the self-attention network. When the second time step is not the background update cycle node, obtain the background mask of the previous background update cycle node of the second time step, and generate a first query matrix based on the background mask, the denoising condition data and the third transformation weight matrix.

7. The method according to claim 6, characterized in that, The step of obtaining the background mask of the previous background update cycle node of the second time step, and generating a first query matrix based on the background mask, the denoising condition data, and the third transform weight matrix, includes: Obtain the third denoising prediction result of the previous background update cycle node of the second time step; Obtain a similarity map between the third denoising prediction result and the video latent features, and generate a background mask for the previous background update cycle node based on the similarity map; The denoising conditional data is updated according to the background mask to obtain conditional update data. The conditional update data is then converted into a first query matrix according to the third transformation weight matrix.

8. The method according to claim 1, characterized in that, The step of obtaining the cross-attention features at the first time step includes: Obtain the time-step sequence of the initial noise data; When the first time step is the first time step in the time step sequence, the second self-attention feature of the first time step is obtained, and the second self-attention feature and the video latent feature are subjected to cross-attention processing to obtain the cross-attention feature of the first time step. When the first time step is not the first time step in the time step sequence, the attention features already stored in the cache space are determined as the cross-attention features of the first time step.

9. The method according to claim 8, characterized in that, The step of performing cross-attention processing on the second self-attention feature and the video latent feature to obtain the cross-attention feature at the first time step includes: The second self-attention feature is transformed into a second query matrix through the cross-attention network in the video restoration model, and the video latent feature is transformed into a second key matrix and a second value matrix. Based on the second query matrix, the second key matrix, and the second value matrix, the cross-attention features of the first time step are obtained.

10. The method according to claim 1, characterized in that, The step of generating a second denoising prediction result for the second time step based on the first self-attention feature and the cross-attention feature includes: Obtain attention weight parameters, and determine the product between the attention weight parameters and the cross-attention features as candidate attention features; The candidate attention feature and the first self-attention feature are added together to obtain the second denoising prediction result for the second time step.

11. The method according to claim 1, characterized in that, The step of generating repaired video data corresponding to the source video data based on the second denoising prediction result includes: When the second time step is the last time step in the time step sequence, the second denoising prediction result is deserialized to obtain video restoration features. The video restoration features are decoded to obtain a restored video frame sequence; The repaired video frame sequence is subjected to video repair to generate repaired video data corresponding to the source video data.

12. A video data processing method, characterized in that, include: Acquire training samples and sample noise data, wherein the training samples include sample video data, target video data, and object sample mask sequences of sample objects in the sample video data; The sample video data is subjected to visual encoding processing to obtain sample latent features, and the first sample prediction result at the first time step is obtained; the first sample prediction result refers to the prediction result of the sample noise data in the denoising processing at the first time step. Based on the first sample prediction result, the sample latent features, and the object sample mask sequence, sample condition data for the second time step is constructed; the second time step refers to the time step after the first time step. The sample conditional data is processed by self-attention using an initial repair model to obtain the self-attention prediction features of the second time step. The cross-attention prediction features of the first time step are obtained. Based on the self-attention prediction features and the cross-attention prediction features, the second sample prediction result of the second time step is generated. The cross-attention prediction features refer to the fusion features that include visual information of the sample video data. Based on the second sample prediction result, generate the predicted video data corresponding to the sample video data; Based on the target video data and the predicted video data, the network parameters of the initial repair model are corrected to obtain a video repair model; the video repair model is used to remove the candidate objects and their visual effects from the source video data based on the object mask sequence of the candidate objects in the source video data.

13. The method according to claim 12, characterized in that, The acquisition of training samples includes: Obtain a set of foreground objects and a set of background images. Combine the foreground objects in the set of foreground objects and the background images in the set of background images to obtain K pairs; K is a positive integer. The foreground object and background image in each pairing group are rendered to obtain K sample video frames, and the K sample video frames are combined into the sample video data. The sample object is determined from the foreground objects contained in the sample video data, and the sample object is removed from the sample video data to obtain the target video data; Obtain the object sample mask sequence of the sample object in the sample video data, and determine the sample video data, the target video data, and the object sample mask sequence as training samples.

14. The method according to claim 12, characterized in that, The step of correcting the network parameters of the initial restoration model based on the target video data and the predicted video data to obtain the video restoration model includes: Obtain the label movement speed between the sample noise data and the target video data, and obtain the path movement speed between the sample noise data and the predicted video data; The model loss of the initial repair model is determined based on the label movement speed and the path movement speed. Based on the model loss, the network parameters of the initial repair model are corrected to obtain the video repair model.

15. A video data processing apparatus, characterized in that, include: The first acquisition module is used to acquire source video data and the object mask sequence of candidate objects in the source video data, and to acquire initial noise data; The first encoding module is used to perform visual encoding processing on the source video data to obtain video latent features and acquire the first denoising prediction result at the first time step; the first denoising prediction result refers to the prediction result of the initial noise data in the denoising processing at the first time step. The first construction module is used to construct denoising conditional data for the second time step based on the first denoising prediction result, the video latent features, and the object mask sequence; the second time step refers to the time step after the first time step. The first processing module is used to perform self-attention processing on the denoised conditional data to obtain the first self-attention feature of the second time step, obtain the cross-attention feature of the first time step, and generate the second denoised prediction result of the second time step based on the first self-attention feature and the cross-attention feature; the cross-attention feature refers to the fusion feature that includes the visual information of the source video data. The first generation module is used to generate repaired video data corresponding to the source video data based on the second denoising prediction result; The repaired video data refers to video data from the source video data that removes the candidate object and the visual effects of the candidate object.

16. A video data processing apparatus, characterized in that, include: The second acquisition module is used to acquire training samples and sample noise data. The training samples include sample video data, target video data, and object sample mask sequences of sample objects in the sample video data. The second encoding module is used to perform visual encoding processing on the sample video data to obtain sample latent features and acquire the first sample prediction result at the first time step; the first sample prediction result refers to the prediction result of the sample noise data in the denoising processing at the first time step. The second construction module is used to construct sample condition data for the second time step based on the first sample prediction result, the sample latent features, and the object sample mask sequence; the second time step refers to the time step after the first time step. The second processing module is used to perform self-attention processing on the sample conditional data through the initial repair model to obtain the self-attention prediction features of the second time step, obtain the cross-attention prediction features of the first time step, and generate the second sample prediction result of the second time step based on the self-attention prediction features and the cross-attention prediction features; the cross-attention prediction features refer to the fusion features that include the visual information of the sample video data. The second generation module is used to generate predicted video data corresponding to the sample video data based on the second sample prediction result. The parameter correction module is used to correct the network parameters of the initial repair model based on the target video data and the predicted video data to obtain a video repair model; the video repair model is used to remove the candidate objects and their visual effects from the source video data based on the object mask sequence of the candidate objects in the source video data.

17. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to invoke the computer program so that the computer device performs the method according to any one of claims 1 to 11, or performs the method according to any one of claims 12 to 14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1 to 11, or to perform the method of any one of claims 12 to 14.

19. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11, or implements the method of any one of claims 12 to 14.

Images