Video editing model based on text and image co-editing and construction method thereof

Abstract

The scheme provides a video editing model based on joint editing of text and images and a construction method thereof. The video editing model introduces a light flow guided mask fusion module and a multi-modal feature recognition and segmentation module in a denoising diffusion implicit model. The original video is input into the denoising diffusion implicit model to obtain multiple frames of latent space generated frames through forward diffusion and noise addition. The multiple frames of latent space generated frames are input into the light flow guided mask fusion module to obtain time consistency latent space generated frames through inter-frame feature alignment. The text prompt, the image prompt and the original video are input into the multi-modal feature recognition and segmentation module to obtain a conditional vector through alignment and positioning. The conditional vector and the time consistency latent space generated frames are iteratively denoised to generate a target edited video, and dynamic feature modulation is performed in each denoising process. The video editing model based on joint editing of text and images efficiently edits the video under the joint guidance of the text and image prompts.

Description

TECHNICAL FIELD

[0001] The present application relates to a video editing model, in particular to a video editing model based on joint editing of text and images and a construction method thereof. BACKGROUND

[0002] In recent years, generative AI has been widely recognized in the field of computer vision, mainly due to the emergence of diffusion models that generate high-quality images based on text prompts through a gradual denoising process, demonstrating strong generation capabilities and outputting high-quality results. A typical image editing process includes two key steps: (1) using a pre-trained diffusion model to reverse a real image into latent features; (2) modifying the corresponding image area by controlling attention maps during the denoising process. The generation capability of diffusion models in text-to-image synthesis has been fully verified, and the success of diffusion models in text-to-image synthesis has prompted researchers to explore their application in video generation and editing tasks.

[0003] Currently, most video editing methods in the video production field are based on text prompts to generate target videos. For example, the deterministic DDIM algorithm is usually used to convert videos into noise, and then generate edited versions step by step according to text input. Although these methods have been successful in video editing, relying solely on text prompts makes it difficult to handle complex artistic styles and rare objects (which cannot be accurately described by text), resulting in frequent suboptimal or unexpected outputs. Tune-A-Video, based on diffusion models, introduces a mechanism to adapt the Stable Diffusion model to the video generation task, thereby increasing the diversity of video editing results. However, this method is still limited to text-guided applications, and although it can generate visually rich results, it often exhibits high uncontrollability, such as the content structure of the original video being partially destroyed, the loss of temporal dimension information during the generation process, and poor visual coherence between adjacent frames.

[0004] In summary, the field of text-based video editing has made recent progress. However, this text-only guided video editing method, although partially meeting the demand, still struggles to accurately control object shape and style, and edited videos often exhibit poor visual coherence between adjacent frames. SUMMARY

[0005] The present application aims to provide a video editing model based on joint editing of text and images and a construction method thereof, which improves the quality of video editing through joint guidance of text and image prompts and the introduction of dynamic mixed feature fusion mechanism and optical flow guided mask fusion module.

[0006] To achieve the above purpose, the technical solution provides a construction method of a video editing model based on joint editing of text and images, comprising the following steps:

[0007] The original video, the text prompt and the image prompt and the target video are taken as a training data set pair.

[0008] The video editing model is obtained by inputting a plurality of training data set pairs into a video editing framework for training, wherein the video editing model introduces a light flow guided mask fusion module and a multi-modal feature recognition and segmentation module in the denoising diffusion implicit model; the original video is input into the denoising diffusion implicit model to obtain a plurality of frames of latent space generated frames through forward diffusion and noise addition; the text prompt, the image prompt and the original video are input into the multi-modal feature recognition and segmentation module to obtain a condition vector through alignment and positioning; and the condition vector and the time consistency latent space generated frame are iteratively denoised to generate a target edited video, and dynamic feature modulation is performed in each denoising process.

[0009] In a second aspect, the present scheme provides a video editing method based on text and image co-editing, comprising the following steps: inputting an original video, a text prompt and an image prompt into a video editing model based on text and image co-editing to edit and obtain an edited video.

[0010] Compared with the prior art, the technical scheme has the following characteristics and beneficial effects:

[0011] 1. The video editing model based on text and image co-editing optimizes the design of a multi-modal feature recognition and instance segmentation module: the multi-modal feature recognition and instance segmentation module integrates text, image and video input into a unified semantic space, and uses target detection and segmentation technology to locate editable regions, realizes accurate object positioning and semantic understanding in complex video scenes, and further realizes flexible control of style, shape and content in the video editing process.

[0012] 2. The video editing model based on text and image co-editing introduces a dynamic hybrid feature fusion mechanism, which enhances feature representation through dynamic weighting of attention and convolution cascading modules, attention captures global structure, convolution preserves local details, and the combination of the two improves the editing quality of complex scenes, i.e. the fusion features of the attention and convolution cascading modules overcome the limitations of single feature extraction, making the style and shape modification in the video editing process more natural and accurate.

[0013] 3. The video editing model based on text and image co-editing adopts a light flow guided mask fusion module, which solves the inter-frame discontinuity problem based on light flow and occlusion perception fusion, uses a light flow guided mask fusion mechanism to align adjacent frames using light flow, identifies unreliable areas through occlusion mask recognition, and compensates with motion alignment features to ensure smooth transition and coherence between video frames. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 FIG. 1 is a flowchart of a video editing method based on a text and image co-editing video editing model according to an embodiment of the present application.

[0015] Figure 2 FIG. 2 is a structural schematic diagram of a multi-modal feature recognition and segmentation module.

[0016] Figure 3 FIG. 3 is a structural schematic diagram of a self-attention and convolution cascade module.

[0017] Figure 4 FIG. 4 is a structural schematic diagram of an optical flow guided mask fusion module.

[0018] Figure 5 FIG. 5 is an ablation experiment data graph according to the present scheme.

[0019] Figure 6 FIG. 6 is a hardware structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0021] Those skilled in the art should understand that in the disclosure of the present application, the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the above terms cannot be understood as a limitation of the present application.

[0022] Embodiment one

[0023] The present scheme provides a text and image co-editing based video editing model, which efficiently edits video under the joint guidance of text and image prompts, aiming to generate more accurate semantic representation and realize precise control of object appearance and structure. And to further solve the problem of time flicker and inter-frame inconsistency, an optical flow guided mask fusion module based on text and image co-editing video editing model is introduced, which effectively reconstructs unstable areas and improves the temporal smoothness of video output.

[0024] Specifically, the method for constructing a video editing model based on joint editing of text and images provided by the scheme comprises the following steps:

[0025] The original video, the text prompt, the image prompt, and the target video are taken as a training data set pair.

[0026] The multiple training data set pairs are input into a video editing framework for training to obtain a video editing model, wherein the video editing model introduces a light flow guided mask fusion module and a multi-modal feature recognition and segmentation module in a denoising diffusion implicit model; the original video is input into the denoising diffusion implicit model for forward diffusion and noise addition to obtain multiple frames of latent space generated frames, the multiple frames of latent space generated frames are input into the light flow guided mask fusion module for inter-frame feature alignment to obtain time-consistent latent space generated frames; the text prompt, the image prompt, and the original video are input into the multi-modal feature recognition and segmentation module for alignment and positioning to obtain a condition vector, and the condition vector and the time-consistent latent space generated frames are iteratively denoised to generate a target edited video, and dynamic feature modulation is performed in each denoising process.

[0027] The scheme is improved on the basis of the denoising diffusion implicit model to realize time-controllable and semantic-controllable video editing effects by using multi-modal input, that is, the video editing model is constructed and improved based on the latent diffusion framework to enable the video editing process to be guided by text, images, and original video information.

[0028] As shown in Figure 2 The multi-modal feature recognition and segmentation module of the scheme aims to encode the text prompt, the image prompt, and the original video into a unified semantic space, the text prompt is converted into a semantic embedding to generate a text feature vector by a CLIP text encoder, and the image prompt generates an image feature vector by a CLIP image encoder; to realize the replacement of the content of the original video, each frame of the original video is processed by a detection and segmentation module to locate objects and generate corresponding segmentation masks, and the detected object regions and their masks are encoded into object feature vectors, the text feature vector, the image feature vector, and the object feature vector are integrated for conditional generation, all extracted features (including text and image prompts) are spliced through a fully connected layer to obtain a unified condition vector, ensuring effective integration of multi-modal input.

[0029] Specifically, the text prompt and the image prompt are input into the CLIP encoder of the multi-modal feature recognition and segmentation module to be encoded to obtain the text feature vector and the image feature vector, respectively, the original video is input into the multi-modal feature recognition and segmentation module, and the corresponding object feature vector is generated after the original video sequentially passes through a backbone network, an RPN candidate region network, and region alignment, the object feature vector, the text feature vector, and the image feature vector are jointly input into a multi-layer full connection to obtain a condition vector, and the formula of the multi-layer full connection is as follows:

[0030]

[0031] wherein FC represents a multi-layer connection layer, Concat() represents concatenation, f text represents a text feature vector, f image_prompt represents an image feature vector, f objects represents an object feature vector, represents a condition vector.

[0032] Specifically, the CLIP encoder includes a CLIP text encoder and a CLIP image encoder, the text prompt is input into the CLIP text encoder for semantic embedding coding to obtain a text feature vector, and the image prompt is input into the CLIP image encoder for image coding to obtain an image feature vector.

[0033] Specifically, the backbone network, the RPN candidate region network and the region alignment in the multi-modal feature recognition and segmentation module constitute a detection and segmentation module for the original video, each video frame of the original video is input into the backbone network for feature extraction, then input into the RPN candidate region network for object positioning and generating a corresponding segmentation mask, and the segmentation masks in different video frames are subjected to region alignment to obtain an object feature vector.

[0034] In the iteration denoising process of the denoising diffusion implicit model, dynamic feature modulation is performed, specifically, the denoising diffusion implicit model includes a noise prediction model and a decoder, wherein a self-attention and a depth separable convolution and a cross-attention module are inserted into each down-sampling and up-sampling of the noise prediction model, wherein the self-attention and the depth separable convolution pay attention to the global structural features and the local detail features of the input features to generate fusion features, and the fusion features and the condition vector are input into the cross-attention module for attention to obtain output features, the insertion of the self-attention and the depth separable convolution uses the self-attention mechanism to keep the global structure unchanged, and uses the depth separable convolution to keep the local detail content unchanged, which can be used for adaptive balance of extraction of global structural features and local detail features, which is crucial for tasks such as style transfer and shape editing.

[0035] Specifically, the iteration denoising of the denoising diffusion implicit model is conditioned on the condition vector, and the time-consistent latent space generation frame is denoised to generate a target edited video consistent with the image prompt and the text prompt. The noise prediction model in the denoising diffusion implicit model predicts noise in each time step t based on the condition vector, and the latent feature z t predicts noise, and then denoises based on the noise predicted by the noise prediction model and the condition vector, wherein the formula for predicting noise by the noise prediction model is as follows:

[0036] ;

[0037] where z t is the latent feature of the t-th time step, t represents the time step, represents a conditional vector, represents a predicted noise.

[0038] As shown in Figure 3 , the noise prediction model of the denoising diffusion implicit model is a U-Net architecture, which introduces a self-attention and depth separable convolution and cross-attention module between each down-sampling and up-sampling of the noise prediction model, wherein the self-attention and depth separable convolution pay attention to the global and local features of the input features to generate fusion features, and the fusion features and the conditional vector are input into the cross-attention module for attention attention to obtain output features, wherein the self-attention and depth separable convolution include a parallel self-attention mechanism path and a convolution path, and the temporal consistency latent space of each time step generates the latent feature of the frame as a residual feature input into the self-attention and depth separable convolution, which enters the self-attention mechanism path through three 1*1 convolution layers to generate query feature Q, key feature K and value feature V, respectively. After the global self-attention weight calculation of the query feature Q, the key feature K and the value feature V, the global structure feature is output, and the residual feature input into the self-attention and depth separable convolution enters the convolution path to perform depth separable convolution, and then the local detail feature is output. The global structure feature and the local detail feature are dynamically fused to obtain the fusion feature; the fusion feature is taken as the query feature of the cross-attention module, and the conditional vector is taken as the key feature and value feature of the cross-attention module.

[0039] Specifically, the residual feature respectively enters the self-attention mechanism path through three 1*1 convolution layers to generate the query feature Q, the key feature K and the value feature V, and the formula is as follows:

[0040] ;

[0041] where Q represents the query feature, Z t represents the latent feature of the t-th time step, conv 1*1 ) represents 1*1 convolution; K represents the key feature, and V represents the value feature.

[0042] The formula for calculating the global attention weight A is as follows: ;

[0043] where d represents the feature dimension, Q represents the query feature, K represents the key feature, and softmax() represents the activation function.

[0044] The global structure feature output by the last self-attention path is:

[0045] ;

[0046] Where A represents the global attention weight and V represents the value feature.

[0047] The residual features input into the self-attention and depth-wise separable convolution respectively enter the convolution path for depth-wise separable convolution and output local detail features as follows:

[0048] ;

[0049] where Z t represents the potential features at the t-th time step, and Depthwise() represents depthwise separable convolution.

[0050] The fusion feature obtained by dynamically weighting the local detail features and the global structure features is expressed as:

[0051] Fout=Fself-attention+Fconv;

[0052] in represents the fusion feature, Represents the global structural characteristics, Represents local detail features.

[0053] This scheme introduces self-attention and convolution cascade modules into the noise prediction model of the denoising diffusion implicit model. The self-attention and convolution cascade modules use a dynamic fusion mechanism to enable the balance between global structural features and local detail features to be adaptively adjusted at each time step, which ensures that the generated video frames of the target edited video are accurate in style and spatially coherent. The fused features after fusion by the self-attention and convolution cascade modules are used to update the latent feature z, guiding the diffusion process towards the denoised latent representation z0. The denoising process is further optimized by incorporating conditional information from the conditional vector. This information is injected through the cross-attention mechanism within the U-Net architecture at each time step, which ensures that the generated video frames are consistent with the diffusion process and match the specified style and content instructions encoded in the conditional vector. Finally, the denoised latent representation z0 is passed through the decoder D, which generates the video frames of the target edited video.

[0054] To enhance temporal consistency, the video editing model of this scheme introduces an optical flow-guided mask fusion module into the denoising diffusion implicit model to further enhance the temporal coherence between video frames. This optical flow-guided mask fusion module is based on the inter-frame feature alignment mechanism of optical flow and occlusion mask. It extracts motion alignment features from adjacent frames for the potential features of the current frame and uses bidirectional frame information to supplement unreliable areas.

[0055] Specifically, such as Figure 4As shown, the current video frame, the previous video frame and the next video frame of the original video are input into the optical flow guided mask fusion module in parallel to sequentially perform optical flow estimation, occlusion mask estimation, time-based adjacent feature fusion and occlusion-aware feature fusion to obtain the latent space generated frame of the current frame.

[0056] That is, in the optical flow estimation stage, the optical flow algorithm is used to calculate the potential features of the previous video frame and the potential features of the next video frame.

[0057] In the occlusion mask estimation stage, the occlusion mask in the current video frame is calculated through forward-backward consistency check.

[0058] In the time-based adjacent feature fusion stage, the potential features of the previous video frame and the potential features of the next video frame are weighted and fused.

[0059] In the occlusion-aware feature fusion stage, the fusion result of the weighted fusion based on the occlusion mask, the potential features of the previous video frame and the potential features of the next video frame, and the current video features are fused to obtain the updated features of the current video frame, and the latent space generated frame of the current frame is generated based on the updated features of the current video frame.

[0060] Specifically, the optical flow guided mask fusion module is used in the present scheme, which enables the occluded area to adaptively recover information from adjacent video frames while preserving the high confidence content in the current video frame. For each video frame, the optical flow guided mask fusion module uses the estimated forward and backward optical flow fields to align the adjacent video frames and the current video frame respectively ， The corresponding potential features are aligned based on the confidence of the occlusion mask and selectively fused with the features of the current video frame to obtain the latent space generated frame of the current frame.

[0061] The fusion process of the optical flow guided mask fusion module is defined as follows:

[0062] ;

[0063] Wherein is the predicted occlusion mask, is the potential feature of the previous video frame, is the potential feature of the next video frame, Fuse() is weighted fusion, is the feature of the current video frame, is the updated feature of the current video frame.

[0064] The loss function of the present scheme is based on the mean square error (MSE) between the predicted noise and the real noise, and the specific form is as follows:

[0065] ;

[0066] wherein denotes the actual noise, is the model predicted noise, by minimizing this loss function, the model learns to gradually denoise the signal and recover the effective image from Gaussian noise.

[0067] Embodiment two

[0068] The present scheme provides an application method of the video editing model based on the joint editing of text and image constructed according to embodiment one, that is, the present scheme provides a video editing method based on the joint editing of text and image, comprising the following steps:

[0069] inputting the original video, the text prompt and the image prompt into the video editing model based on the joint editing of text and image for editing to obtain an edited video.

[0070] Figure 1 is an example of the video editing model based on the joint editing of text and image of the present scheme when editing a video, which Figure 1 is intended to edit the original video (a woman running by the river) based on the text prompt (man) and the image prompt (image corresponding to the man), as shown in Figure 1 , the original video is input into the denoising diffusion implicit model to obtain multiple frames of latent space generated frames Z through forward diffusion and noise addition, and the multiple frames of latent space generated frames are input into the optical flow guided mask fusion module to obtain time consistent latent space generated frames Z' through inter-frame feature alignment; the text prompt, the image prompt and the original video are input into the multi-modal feature recognition and segmentation module to obtain a conditional vector through alignment and positioning, and the conditional vector and the time consistent latent space generated frames are iteratively denoised to generate a target edited video, and dynamic feature modulation is performed in each denoising process, and finally an edited video (a man running by the river) is output.

[0071] In addition, in order to evaluate the contribution of each module of the video editing model based on the joint editing of text and image of the present scheme, the present scheme designs an ablation experiment, and designs three cases: (i) not using the image prompt, (ii) not using the feature fusion of the attention and convolution cascade module, (iii) not using the mask fusion module guided by optical flow, and the results are shown in Fig. 5. Figure 5The text prompt input is "a bald eagle with white head and tail is flying in the forest", and the image prompt is "an eagle with a consistent white head and tail". It is observed that, without the image prompt (the third column), the generated appearance of the eagle is close to the correct morphology, but the head and tail exhibit a "white with yellow" color tone, deviating from the specified "white" in the prompt. Without the feature fusion of the attention and convolution cascading module (the fourth column), although the white color of the head and tail is correctly rendered, the structural details of the tail are significantly degraded, which highlights the importance of the image prompt in improving editing fidelity and indicates that the feature fusion of the attention and convolution cascading module enhances both feature expression capability and adaptability. Without the optical flow guided mask fusion module (the fifth column), the edited video exhibits obvious temporal artifacts, such as shaking around the object boundary, especially in areas of fast motion or occlusion. In contrast, the text and image co-editing based video editing model of the present solution (the second column) achieves smoother transitions and better temporal semantic consistency.

[0072] Embodiment Three

[0073] The embodiment also provides an electronic device, referring to Figure 6 , comprising a memory 404 and a processor 402, the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0074] Specifically, the above processor 402 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0075] The memory 404 can include a mass storage device for data or instructions. The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, and possible computer program instructions executed by the processor 402.

[0076] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any of the above application methods of the text and image co-editing based video editing model or the construction method of the text and image co-editing based video editing model.

[0077] Optionally, the electronic device can further include a transmission device 406 connected to the processor 402 and an input / output device 408 connected to the processor 402.

[0078] The transmission device 406 can be configured to receive or send data via a network. Examples of the network can include a wired or wireless network provided by a communication provider of the electronic device. In an example, the transmission device includes a network interface controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In an example, the transmission device 406 can be a radio frequency (RF) module configured to communicate with the Internet in a wireless manner.

[0079] The input / output device 408 is configured to input or output information. In the present embodiment, the input information can be raw videos, text prompts, and image prompts, and the output information can be edited videos.

[0080] Optionally, in the present embodiment, the processor 402 can be configured to perform the following steps by executing a computer program:

[0081] The raw videos, text prompts, and image prompts are input into a video editing model based on joint text and image editing to obtain edited videos.

[0082] It should be noted that the specific examples in the present embodiment can refer to the examples described in the above embodiments and optional implementation manners, which will not be described herein again.

[0083] Generally, various embodiments can be implemented in hardware or special-purpose circuitry, software, logic or any combination thereof. Some aspects of the application can be implemented in hardware, while other aspects can be implemented in firmware or software to be executed by a controller, microprocessor or other computing device, although the application is not limited thereto. While various aspects of the application can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controler or other computing devices, or some combination thereof.

[0084] Embodiments of the application can be implemented by computer software executable by a data processor of the mobile device such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, can be stored in any apparatus-readable data storage medium and they include program instructions to implement certain tasks. The program product can include one or more computer-executable components tangibly embodied in a computer- readable medium, when executed, for implementing one or more embodiments of the present application. The one or more computer-executable components can be one or more of: a process; a function; a routine; a sag; a subroutine; a plug-in; an app; or a program. The one or more computer-executable components can include a computer program that is written in any form of programming language, including code.

[0085] Those skilled in the art should clearly understand that each technical feature in the above embodiments can be combined arbitrarily, and for the sake of brevity, each technical feature in the above embodiments is not described in all possible combinations, however, as long as the combination of technical features does not exist, it should be considered as the scope of the present application.

[0086] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for ordinary skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for constructing a video editing model based on co-editing of text and images, characterized in that, The method comprises the following steps: The method comprises the following steps: The original video, the text prompt and the image prompt and the target video are taken as a training data set pair; The video editing model is obtained by inputting multiple groups of training data set pairs into a video editing framework for training, wherein the video editing model introduces a flow-guided mask fusion module and a multi-modal feature recognition and segmentation module in a denoising diffusion implicit model; the original video is input into the denoising diffusion implicit model to perform forward diffusion and noise addition to obtain multiple frames of latent space generated frames, the multiple frames of latent space generated frames are input into the flow-guided mask fusion module to perform inter-frame feature alignment to obtain time-consistent latent space generated frames; the text prompt and the image prompt are input into a CLIP encoder of the multi-modal feature recognition and segmentation module to perform encoding to obtain a text feature vector and an image feature vector respectively, the original video is input into the multi-modal feature recognition and segmentation module, a backbone network, an RPN candidate region network and region alignment in the multi-modal feature recognition and segmentation module constitute a detection and segmentation module for the original video, each video frame of the original video is input into the backbone network for feature extraction and then input into the RPN candidate region network for object positioning and generation of a corresponding segmentation mask, the segmentation masks in different video frames are aligned in regions to obtain object feature vectors, the object feature vectors, the text feature vector and the image feature vector are input into a multi-layer full connection for splicing to obtain a conditional vector, the conditional vector and the time-consistent latent space generated frames are iteratively denoised to generate a target edited video, and dynamic feature modulation is performed in each denoising process, wherein the iterative denoising of the denoising diffusion implicit model is to denoise the time-consistent latent space generated frames to generate a target edited video consistent with the image prompt and the text prompt under the condition of the conditional vector; The denoising diffusion implicit model comprises a noise prediction model and a decoder, wherein a self-attention and depth separable convolution and a cross-attention module are inserted into each down-sampling and up-sampling of the noise prediction model, wherein the self-attention and depth separable convolution pay attention to global structural features and local detail features of input features to generate fusion features, and the fusion features and the conditional vector are input into the cross-attention module to obtain output features through attention; the self-attention and depth separable convolution comprises a parallel self-attention mechanism path and a convolution path, the latent features of the time-consistent latent space generated frames at each time step are input into the self-attention and depth separable convolution to enter the self-attention mechanism path to generate query features Q, key features K and value features V through three 1x1 convolution layers, the global structural features are output after global self-attention weight calculation of the query features Q, the key features K and the value features V, the residual features input into the self-attention and depth separable convolution enter the convolution path to output the local detail features after depth separable convolution, and the local detail features and the global structural features are dynamically weighted and fused to obtain the fusion features, the fusion features are taken as the query features of the cross-attention module, and the conditional vector is taken as the key features and the value features of the cross-attention module.

2. The method of claim 1, wherein the method of constructing a video editing model based on co-editing of text and images, further comprises: The CLIP encoder comprises a CLIP text encoder and a CLIP image encoder, the text prompt is input into the CLIP text encoder for semantic embedding coding to obtain a text feature vector, and the image prompt is input into the CLIP image encoder for image coding to obtain an image feature vector. 3.The method of claim 1, wherein, The current video frame, the previous video frame and the next video frame of the original video are input into the optical flow guided mask fusion module in parallel to sequentially perform optical flow estimation, occlusion mask estimation, time-based adjacent feature fusion and occlusion-aware feature fusion to obtain the latent space generated frame of the current frame.

4. The method of claim 3, wherein the video editing model is constructed based on co-editing of text and images. The optical flow algorithm is used to calculate the latent feature of the previous video frame and the latent feature of the next video frame. The occlusion mask in the current video frame is calculated through forward-backward consistency checking; the latent feature of the previous video frame and the latent feature of the next video frame are weighted and fused; the fusion result of the weighted fusion based on the occlusion mask, the latent feature of the previous video frame and the latent feature of the next video frame, and the current video feature are fused to obtain the updated feature of the current video frame, and the latent space generated frame of the current frame is generated based on the updated feature of the current video frame.

5. A video editing method based on co-editing of text and images, characterized by, The method comprises the following steps: The original video, the text prompt and the image prompt are input into the video editing model based on the joint editing of text and image to perform editing to obtain an edited video, wherein the video editing model based on the joint editing of text and image is constructed according to the construction method of the video editing model based on the joint editing of text and image according to any one of claims 1 to 4. 6.An electronic device comprising a memory and a processor, the electronic device comprising: The memory stores a computer program, and the processor is configured to run the computer program to execute the construction method of the video editing model based on the joint editing of text and image according to any one of claims 1 to 4 or the video editing method based on the joint editing of text and image according to claim 5. The memory stores a computer program, and the processor is configured to run the computer program to execute the construction method of the video editing model based on the joint editing of text and image according to any one of claims 1 to 4 or the video editing method based on the joint editing of text and image according to claim 5.

Images