Diffusion model video local editing method and system based on mask guidance

Abstract

The invention provides a diffusion model video local editing method and system based on mask guidance. The method comprises the following steps: encoding a video sequence of a target video to obtain a first video frame submerged space feature; performing coarse-grained mask labeling on a target editing area in the target video to obtain mask information; determining a space attention weight according to the first video frame potential space feature; determining a second video frame potential space feature according to the first video frame potential space feature, the mask information and the space attention weight; determining a time attention weight according to the second video frame potential space feature; determining a third video frame potential space feature according to the mask information, the second video frame potential space feature and the time attention weight; and decoding the hidden space feature of the third video frame to generate an edited video. According to the method, frame-by-frame accurate masking is not needed, the time-space consistency of local editing of the video can be effectively enhanced, an unedited area is kept unchanged, and a high-quality and stable video editing effect is achieved.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and in particular relates to a method and system for local video editing based on a mask-guided diffusion model. Background Technology

[0002] With the development of computer vision and deep learning technologies, video content generation and editing methods based on diffusion models have gradually become mainstream. In video editing tasks, users often want to replace content only in local areas of a video, such as changing the appearance, material, or shape of an object, without affecting other unedited areas.

[0003] Traditional video editing primarily relies on manual frame-by-frame modification or optical flow-based content propagation. However, these methods are not only complex and time-consuming, but also prone to noticeable visual artifacts and structural distortions in videos with significant motion or occlusion changes. Therefore, achieving high-quality local video editing without requiring manual frame-by-frame processing has significant application value.

[0004] Diffusion models excel in image generation and editing tasks, and some text-based editing methods achieve semantic replacement of video content by fine-tuning image diffusion models. However, these methods typically assume that the region being edited is the primary semantic object in the video. For editing localized, non-primary regions in complex scenes, precise control is often difficult, leading to issues such as edited region offset, semantic mismatch, and accidental modification of unedited areas. Furthermore, when the edited region changes significantly over time, the lack of cross-frame consistency modeling can result in inconsistent texture jitter and shape distortion in the generated results over time.

[0005] To improve the controllability of local region editing, some methods introduce mask models to define the editing area. However, these methods typically require precise masking frame by frame, resulting in high operational costs and difficulty in scalability. Furthermore, most current methods do not explicitly model the spatiotemporal relationship of the masked region within the diffusion model, leading to difficulties in maintaining stability and consistency across the video sequence even though local content can be replaced. Summary of the Invention

[0006] This invention provides a video local editing method and system based on a mask-guided diffusion model, which can enhance the spatiotemporal consistency and semantic expression of the video local editing area without the need for precise frame-by-frame masking, and ensure that the unedited areas in the video are not destroyed.

[0007] In a first aspect, the present invention provides a mask-guided diffusion model-based video local editing method, comprising: The video sequence of the target video is encoded using VAE to obtain the latent space features of the video frames of the target video, which are then used as the first latent space features of the video frames. Coarse-grained mask annotation is performed on the target editing area in the target video to obtain mask information; Determine spatial attention weights based on the latent spatial features of the first video frame; Based on the latent space features of the first video frame, mask information, and spatial attention weights, the spatially enhanced latent space features of the first video frame are determined and used as the latent space features of the second video frame. Determine the temporal attention weights based on the latent spatial features of the second video frame; The latent space features of the second video frame are determined based on the mask information, the latent space features of the second video frame, and the temporal attention weights, and are used as the latent space features of the third video frame. The latent space features of the third video frame are decoded to generate the edited video; in this process, the target editing area is modified while the unedited area remains unchanged.

[0008] Optionally, determining the spatial attention weights based on the latent spatial features of the first video frame includes: The spatial attention weight W is calculated using the following formula. s : ; Where F represents the latent space feature of the first video frame; max(·) represents taking the maximum value in the channel dimension; avg(·) represents taking the mean value in the channel dimension; combined[a,b] represents concatenating the results of a and b in the channel dimension; Conv 7×7 (·) indicates that the multi-channel feature map is fed into the 7×7 convolutional layer and outputs a single-channel feature map; σ(·) represents the sigmoid activation function.

[0009] Optionally, determining the spatially enhanced latent space features of the first video frame based on the latent space features of the first video frame, mask information, and spatial attention weights, to serve as the latent space features of the second video frame, includes: The latent space feature F of the second video frame is calculated using the following formula. s : F s =F⊙(W s ⊙M); Where F represents the latent space feature of the first video frame; ⊙ represents the dot product; W s represents the spatial attention weights; M represents the mask information.

[0010] Optionally, determining the temporal attention weights based on the latent space features of the second video frame includes: Calculate the time attention weight W using the following formula.t : ; Where σ(·) represents the sigmoid activation function; F(·) represents the feature transformation module, which includes two 3D convolutional layers for processing video frames and one activation function layer; AvgPool3d H,W (·) indicates a 3D average pooling operation; F s For the latent space features of the second video frame; MaxPool3d H,W (·) represents a 3D max pooling operation; H is the height dimension of the feature; W is the width dimension of the feature.

[0011] Optionally, determining the spatiotemporally enhanced second video frame latent space features based on mask information, second video frame latent space features, and temporal attention weights as third video frame latent space features includes: The latent space feature F of the third video frame is calculated using the following formula. st : F st =F s ⊙(W t ⊙M); Among them, F s The latent space features of the second video frame; ⊙ represents dot product; W t represents the temporal attention weight; M represents the mask information.

[0012] Optionally, the first aspect also includes: A noise prediction loss function is constructed to optimize the latent space features of the second and third video frames; wherein, the noise prediction loss function is based on the noise prediction task of the diffusion model and minimizes the Euclidean distance between the predicted noise and the real noise.

[0013] Optionally, the construction of the noise prediction loss function to optimize the latent space features of the second and third video frames includes: Construct the expression for the noise prediction loss function Loss: ; Where E represents the expectation; z,ε~N(0,1) are the latent space features of the video frame z and the actual added noise ε following a standard normal distribution; t is the diffusion time step; The predicted noise output by the noise prediction network of the diffusion model; z t c' represents the noise latent feature at time step t; c' represents the text condition. It is the square of the Euclidean norm.

[0014] Secondly, the present invention provides a mask-guided diffusion model-based video local editing system, comprising: The encoding module is used to perform VAE encoding on the video sequence of the target video to obtain the video frame latent space features of the target video, which are used as the first video frame latent space features. The mask annotation module is used to perform coarse-grained mask annotation on the target editing area in the target video to obtain mask information; The first determining module is used to determine spatial attention weights based on the latent spatial features of the first video frame. The second determining module is used to determine the spatially enhanced latent space features of the first video frame based on the latent space features of the first video frame, mask information and spatial attention weights, so as to serve as the latent space features of the second video frame. The third determining module is used to determine the temporal attention weights based on the latent space features of the second video frame. The fourth determination module is used to determine the latent space features of the spatiotemporally enhanced second video frame based on the mask information, the latent space features of the second video frame, and the temporal attention weight, so as to serve as the latent space features of the third video frame. The decoding module is used to decode the latent space features of the third video frame to generate the edited video; in this module, the target editing area is modified while the unedited area remains unchanged.

[0015] Thirdly, the present invention provides a computer device including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the mask-guided diffusion model video local editing method described in the first aspect.

[0016] Fourthly, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the mask-guided diffusion model video local editing method described in the first aspect.

[0017] This invention provides a mask-guided diffusion model-based video local editing method and system. The method enables controllable editing of local areas in a video without the need for precise frame-by-frame masking. It achieves precise positioning of the semantic expression of the edited area through a spatial attention enhancement mechanism and maintains cross-frame consistency through a temporal attention enhancement mechanism. This effectively reduces deformation and texture flicker in local areas and preserves the structure and appearance of unedited areas, thereby obtaining more natural, stable and high-quality video editing results. Attached Figure Description

[0018] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a mask-guided diffusion model-based video local editing method provided in an embodiment of the present invention; Figure 2 A schematic diagram showing red bloodstains on the face of a soldier, provided as an embodiment of the present invention; Figure 3 A schematic diagram of the video frame mask corresponding to the face of a soldier provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the effect of bloodstain color editing provided in an embodiment of the present invention; Figure 5 A schematic diagram of a ship with a dragon head sculpture provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the video frame mask corresponding to the dragon head sculpture provided in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the effect of sculpting after editing, provided in an embodiment of the present invention. Figure 8 This is a schematic diagram of the structure of a mask-guided diffusion model video local editing system provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1 like Figure 1 As shown, this embodiment provides a mask-guided diffusion model-based video local editing method, including: Step 101: Perform VAE encoding on the video sequence of the target video to obtain the latent space features of the video frames of the target video, which are used as the first latent space features of the video frames.

[0022] In this embodiment, a diffusion model is used for video editing, which requires two inputs: one is the latent feature representation of the video frame order (the size is adjusted to a uniform specification, such as 512×512), and the other is the mask information.

[0023] Step 102: Perform coarse-grained mask annotation on the target editing area in the target video to obtain mask information.

[0024] In this embodiment, video data preprocessing and mask annotation are performed on the target editing area according to the editing requirements of the input video. Taking facial expression editing as an example, the face area is annotated with rectangles or polygons, without the need for precise masking frame by frame. The mask covers the motion area of ​​the editing target to obtain mask information.

[0025] Step 103: Determine the spatial attention weights based on the latent spatial features of the first video frame.

[0026] In this step, the latent spatial features of the first video frame are feature-encoded using a spatial attention coding module to obtain spatial attention weights. Specifically, the spatial attention coding module calculates the spatial attention weights W according to the following formula. s : .

[0027] Where F represents the latent space feature of the first video frame; max(F) represents the maximum value of the latent space feature F of the first video frame in the channel dimension; avg(F) represents the mean value of the latent space feature F of the first video frame in the channel dimension; combined[avg(F),max(F)] represents concatenating the results of avg(F) and max(F) in the channel dimension; Conv 7×7 (·) indicates that the concatenated two-channel feature map is fed into a 7×7 convolutional layer to output a single-channel feature map, generating importance weights for each position; σ(·) represents the sigmoid activation function; R is the set of real numbers; B is the batch size; H is the height dimension of the feature; W is the width dimension of the feature; C is the number of channels of the feature; c is the summation index variable; F :,C,:,: This represents the value of the Cth channel in the feature; e is the natural constant; and n is an intermediate variable.

[0028] Step 104: Determine the spatially enhanced latent space features of the first video frame based on the latent space features of the first video frame, mask information, and spatial attention weights, so as to serve as the latent space features of the second video frame.

[0029] In this step, the obtained spatial attention weight W... s Multiplying the spatial attention weights by the mask information M yields the spatial attention weights within the mask. These weights are then weighted with the latent spatial features of the first video frame to obtain the weighted latent spatial features of the video frame. Specifically, the latent spatial features F of the second video frame are calculated according to the following formula. s : F s =F⊙(W s ⊙M).

[0030] Where F represents the latent space feature of the first video frame; ⊙ represents the dot product; W srepresents the spatial attention weights; M represents the mask information.

[0031] In this step, the semantic features within the editing region are enhanced using a mask-guided spatial attention module in the diffusion model. Specifically, the mask-guided spatial attention module comprises a dual-branch processing structure. The average pooling branch performs multi-scale average pooling on the spatial features to capture global contextual information; the max pooling branch performs multi-scale max pooling on the spatial features to highlight local salient features. Subsequently, the output undergoes ChannelAvg and ChannelMax operations separately, and then enters a 7×7 convolutional layer (Conv7×7) for processing. Next, the spatial attention weights are obtained through a sigmoid activation function, and finally, these weights are weighted and fused with the original input features F.

[0032] Step 105: Determine the temporal attention weights based on the latent space features of the second video frame.

[0033] Mask-guided temporal attention constraints augment the spatially enhanced features Fs. To facilitate the module's processing of relationships between frame sequences, the original input tensor has the shape (B, C, H, W), where B represents the total number of input samples, but each sample is actually composed of f frames. To explicitly model the temporal relationships between frames, this embodiment first reorganizes the original input tensor into (B...C, H, W) using a view operation. The shape of (, C, f, H, W), where B =B / f, B Here, f is the actual number of samples, and f is the number of frames per sample. Therefore, the frame dimension f is used as an independent time axis for easier subsequent processing. The processed data is then input into the temporal attention module to obtain the temporal consistency features, i.e., the temporal attention weights.

[0034] For example, the temporal attention weight W is calculated according to the following formula. t : .

[0035] Where σ(·) represents the sigmoid activation function; F(·) represents the feature transformation module, which includes two 3D convolutional layers for processing video frames and one activation function layer; AvgPool3d H,W (·) indicates a 3D average pooling operation in both the height and width dimensions; F s For the latent space features of the second video frame; MaxPool3d H,W(·) represents a 3D max pooling operation on the height and width dimensions; H is the height dimension of the feature; W is the width dimension of the feature; h' is the height of the pooling window; w' is the width of the pooling window; h is the summation index variable of the height dimension; w is the summation index variable of the width dimension; X is the feature variable; Conv3D(·) represents 3D convolution; ReLU(·) represents the linear rectified function.

[0036] Step 106: Determine the latent space features of the second video frame for spatiotemporal enhancement based on the mask information, the latent space features of the second video frame, and the temporal attention weights, so as to serve as the latent space features of the third video frame.

[0037] In this step, the obtained temporal attention weights W t Multiplying the result by the mask information M yields the temporal attention weights within the mask. The processed weights are then combined with the features F. s The weighted latent space features F of the video frames are obtained by performing weighted processing. st .

[0038] For example, the latent space feature F of the third video frame is calculated according to the following formula. st : F st =F s ⊙(W t ⊙M).

[0039] Among them, F s The latent space features of the second video frame; ⊙ represents dot product; W t represents the temporal attention weight; M represents the mask information.

[0040] Specifically, the mask-guided temporal attention module performs a view operation on the input feature X, and then inputs it into MaxPool and AvgPool for temporal pooling, obtaining the pooled features X_maxpool and X_avgpool. X_maxpool and X_avgpool are then processed by a Conv3d 1×1 module and a ReLU activation function, respectively, to obtain the processed features X_conv1_max and X_conv1_avg. X_conv1_max and X_conv1_avg are then added together and input into a Sigmoid activation function to obtain temporal attention weights. Finally, the obtained temporal attention weights are multiplied element-wise with the input feature X after the view operation to obtain the temporally attention-weighted features, thus completing the temporal attention mechanism.

[0041] Step 107: Decode the latent space features of the third video frame to generate the edited video; wherein the target editing area is modified and the unedited area remains unchanged.

[0042] In this step, the latent space features of the third video frame are input into the multi-scale decoder to generate the final video editing result. The decoder adopts a progressive upsampling strategy: first, coarse-grained content is generated in the low-resolution space, then gradually upsampled and detailed information is incorporated, finally outputting the edited video at full resolution.

[0043] For example, the mask-guided diffusion model video local editing method provided in this embodiment further includes: A noise prediction loss function is constructed to optimize the latent space features of the second and third video frames; wherein, the noise prediction loss function is based on the noise prediction task of the diffusion model and minimizes the Euclidean distance between the predicted noise and the real noise.

[0044] Specifically, the algorithms executed by the aforementioned modules are added to the diffusion model, effectively enhancing its video editing capabilities. Based on the training paradigm of the diffusion model, a noise prediction loss function is used to optimize and constrain the mask-guided spatial attention module and the mask-guided temporal attention module. The specific optimization process is as follows: Noise Addition and Prediction Mechanism Forward diffusion process: (1) Gaussian noise is gradually added to the latent features of the original video to generate noisy latent features.

[0045] (2) Noise addition follows a predefined diffusion scheduling strategy to control the noise intensity to increase with time step. Noise prediction task: 1) The diffusion model receives noise latent features, time step information, and text conditions.

[0046] 2) The model needs to predict the added noise components and learn the denoising mapping relationship.

[0047] 3) Align text cues with video content using a cross-attention mechanism.

[0048] Construct the expression for the noise prediction loss function Loss: .

[0049] Where E represents the expectation (of the random variable in parentheses); z,ε~N(0,1) are the latent space features of the video frame z and the real added noise ε following a standard normal distribution; t is the diffusion time step; The predicted noise output by the noise prediction network of the diffusion model; z t c' represents the noise latent feature at time step t; c' represents the text condition. The square of the Euclidean norm measures the difference between predicted noise and actual conditional noise. This difference is used as a parameter to update the mask-guided spatial attention module and the mask-guided temporal attention module through Loss backpropagation, thereby improving the editing effect of the modules.

[0050] Optimization goal: Spatiotemporal consistency learning: (1) Through the noise prediction task, the model implicitly learns the spatiotemporal distribution pattern of video content.

[0051] (2) The time attention mechanism ensures motion continuity and content stability across frames.

[0052] (3) Spatial attention mechanism ensures the natural integration of the editing area and the background within a single frame.

[0053] This embodiment also provides a result image of the video editing, such as... Figure 2 , Figure 3 and Figure 4 As shown, this section simply demonstrates the editing effect of the corresponding frame. Figure 2 The content includes a soldier figure with red bloodstains on his face. Figure 3 For the mask corresponding to the soldier's face, after inputting these two parameters into the mask-guided spatial attention module and the mask-guided temporal attention module, the text can be edited as needed, such as changing the bloodstains on the soldier's face from red to blue. Figure 4 It demonstrates the effect of editing bloodstain color while ensuring that other areas of the face remain unchanged.

[0054] like Figure 5 , Figure 6 and Figure 7 As shown, similarly Figure 5 The content includes a ship with a dragon head sculpture. Figure 6 Create a video frame mask corresponding to the dragon head sculpture, and provide edit text as needed, such as editing the dragon head sculpture to a tiger head sculpture. Figure 7 The sculpture editing effect on display.

[0055] In summary, this embodiment provides a mask-guided diffusion model-based video local editing method. First, coarse-grained mask annotation is applied to the regions to be edited in the input video sequence, and the video is encoded into latent spatial features. Then, the mask information is embedded into the spatial attention module of the diffusion model to enhance the semantic expressiveness of the target editing region, thereby obtaining a local content modification result. Next, a temporal attention module is used to model the association of features of the same mask region in adjacent frames, ensuring that the edited region maintains consistency in shape, texture, and appearance across multiple frames. Finally, the attention-enhanced latent features are decoded to generate a video, resulting in an edited result where content modification is completed in the edited region while the unedited regions retain their original appearance. By constraining the temporal consistency between the edited result and the input video and guiding optimization to preserve the structure of local regions, the stability and visual quality of the final video can be further improved.

[0056] This embodiment proposes a mask-guided diffusion model-based video local editing method that enables controllable editing of local regions in a video without the need for precise frame-by-frame masking. It achieves precise positioning of the semantic expression of the edited region through a spatial attention enhancement mechanism and maintains cross-frame consistency through a temporal attention enhancement mechanism, effectively reducing deformation and texture flicker in local regions. Furthermore, it preserves the structure and appearance of the unedited region, thereby obtaining more natural, stable, and high-quality video editing results.

[0057] Example 2 Based on the same inventive concept as Embodiment 1, this embodiment also provides a mask-guided diffusion model video local editing system. Since the principle of this system in solving the problem is similar to the aforementioned mask-guided diffusion model video local editing method, the implementation of this system can refer to the implementation of the mask-guided diffusion model video local editing method.

[0058] like Figure 8 As shown, a mask-guided diffusion model-based video local editing system includes: The encoding module 10 is used to perform VAE encoding on the video sequence of the target video to obtain the video frame latent space features of the target video, which are used as the first video frame latent space features.

[0059] The mask annotation module 20 is used to perform coarse-grained mask annotation on the target editing area in the target video to obtain mask information.

[0060] The first determining module 30 is used to determine the spatial attention weights based on the latent spatial features of the first video frame.

[0061] The second determining module 40 is used to determine the spatially enhanced latent space features of the first video frame based on the latent space features of the first video frame, mask information and spatial attention weights, so as to serve as the latent space features of the second video frame.

[0062] The third determining module 50 is used to determine the temporal attention weights based on the latent space features of the second video frame.

[0063] The fourth determining module 60 is used to determine the latent space features of the spatiotemporally enhanced second video frame based on the mask information, the latent space features of the second video frame, and the temporal attention weight, so as to serve as the latent space features of the third video frame.

[0064] The decoding module 70 is used to decode the latent space features of the third video frame to generate the edited video; wherein the target editing area is modified while the unedited area remains unchanged.

[0065] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0066] Example 3 This embodiment provides a computer device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the mask-guided diffusion model video local editing method described in Embodiment 1.

[0067] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0068] Example 4 This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the mask-guided diffusion model video local editing method described in Embodiment 1.

[0069] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0070] Example 5 This embodiment provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the steps of the mask-guided diffusion model video local editing method described in Embodiment 1.

[0071] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0072] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems, devices, storage media, and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0073] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0074] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0075] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0076] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0077] The present invention has been described in detail above with reference to specific embodiments and exemplary examples; however, these descriptions should not be construed as limiting the present invention. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and embodiments of the present invention without departing from the spirit and scope of the invention, and all such modifications and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A mask-guided diffusion model-based video local editing method, characterized in that, include: The video sequence of the target video is encoded using VAE to obtain the latent space features of the video frames of the target video, which are then used as the first latent space features of the video frames. Coarse-grained mask annotation is performed on the target editing area in the target video to obtain mask information; Determine spatial attention weights based on the latent spatial features of the first video frame; Based on the latent space features of the first video frame, mask information, and spatial attention weights, the spatially enhanced latent space features of the first video frame are determined and used as the latent space features of the second video frame. Determine the temporal attention weights based on the latent spatial features of the second video frame; The latent space features of the second video frame are determined based on the mask information, the latent space features of the second video frame, and the temporal attention weights, and are used as the latent space features of the third video frame. The latent space features of the third video frame are decoded to generate the edited video; in this process, the target editing area is modified while the unedited area remains unchanged.

2. The diffusion model video local editing method according to claim 1, characterized in that, The step of determining spatial attention weights based on the latent spatial features of the first video frame includes: The spatial attention weight W is calculated using the following formula. s : ； Where F represents the latent space feature of the first video frame; max(·) represents taking the maximum value in the channel dimension; avg(·) represents taking the mean value in the channel dimension; combined[a,b] represents concatenating the results of a and b in the channel dimension; Conv 7×7 (·) indicates that the multi-channel feature map is fed into the 7×7 convolutional layer and outputs a single-channel feature map; σ(·) represents the sigmoid activation function.

3. The diffusion model video local editing method according to claim 1, characterized in that, The step of determining the spatially enhanced latent spatial features of the first video frame based on the latent spatial features of the first video frame, mask information, and spatial attention weights, to serve as the latent spatial features of the second video frame, includes: The latent space feature F of the second video frame is calculated using the following formula. s : F s =F⊙(W s ⊙M); Where F represents the latent space feature of the first video frame; ⊙ represents the dot product; W s represents the spatial attention weights; M represents the mask information.

4. The diffusion model video local editing method according to claim 1, characterized in that, The step of determining the temporal attention weights based on the latent spatial features of the second video frame includes: Calculate the time attention weight W using the following formula. t : ； Where σ(·) represents the sigmoid activation function; F(·) represents the feature transformation module, which includes two 3D convolutional layers for processing video frames and one activation function layer; AvgPool3d H,W (·) indicates a 3D average pooling operation; F s For the latent space features of the second video frame; MaxPool3d H,W (·) represents a 3D max pooling operation; H is the height dimension of the feature; W is the width dimension of the feature.

5. The diffusion model video local editing method according to claim 1, characterized in that, The step of determining the spatiotemporally enhanced second video frame latent space features based on mask information, second video frame latent space features, and temporal attention weights, as the third video frame latent space features, includes: The latent space feature F of the third video frame is calculated using the following formula. st : F st =F s ⊙(W t ⊙M); Among them, F s The latent space features of the second video frame; ⊙ represents dot product; W t represents the temporal attention weight; M represents the mask information.

6. The diffusion model video local editing method according to claim 1, characterized in that, Also includes: A noise prediction loss function is constructed to optimize the latent space features of the second and third video frames; wherein, the noise prediction loss function is based on the noise prediction task of the diffusion model and minimizes the Euclidean distance between the predicted noise and the real noise.

7. The diffusion model video local editing method according to claim 6, characterized in that, The construction of the noise prediction loss function to optimize the latent space features of the second and third video frames includes: Construct the expression for the noise prediction loss function Loss: ； Where E represents the expectation; z,ε~N(0,1) are the latent space features of the video frame z and the actual added noise ε following a standard normal distribution; t is the diffusion time step; The predicted noise output by the noise prediction network of the diffusion model; z t c' represents the noise latent feature at time step t; c' represents the text condition. It is the square of the Euclidean norm.

8. A mask-guided diffusion model-based video local editing system, characterized in that, include: The encoding module is used to perform VAE encoding on the video sequence of the target video to obtain the video frame latent space features of the target video, which are used as the first video frame latent space features. The mask annotation module is used to perform coarse-grained mask annotation on the target editing area in the target video to obtain mask information; The first determining module is used to determine spatial attention weights based on the latent spatial features of the first video frame. The second determining module is used to determine the spatially enhanced latent space features of the first video frame based on the latent space features of the first video frame, mask information and spatial attention weights, so as to serve as the latent space features of the second video frame. The third determining module is used to determine the temporal attention weights based on the latent space features of the second video frame. The fourth determination module is used to determine the latent space features of the spatiotemporally enhanced second video frame based on the mask information, the latent space features of the second video frame, and the temporal attention weight, so as to serve as the latent space features of the third video frame. The decoding module is used to decode the latent space features of the third video frame to generate the edited video; in this module, the target editing area is modified while the unedited area remains unchanged.

9. A computer device, characterized in that, It includes a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the mask-guided diffusion model video local editing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; when executed by a processor, the computer programs implement the steps of the mask-guided diffusion model video local editing method as described in any one of claims 1-7.

Images