Video editing model based on common editing of text and image and construction method thereof

Abstract

The invention provides a video editing model based on text and image common editing and a construction method thereof. The video editing model introduces an optical flow guide mask fusion module and a multi-modal feature recognition and segmentation module into a denoising diffusion implicit model; the method comprises the following steps: inputting an original video into a denoising diffusion implicit model to carry out forward diffusion noise addition to obtain a multi-frame submerged space generation frame, and inputting the multi-frame submerged space generation frame into an optical flow guide mask fusion module to carry out inter-frame feature alignment to obtain a time consistency submerged space generation frame; a text prompt, an image prompt and an original video are input into a multi-modal feature recognition and segmentation module to be aligned and positioned to obtain a condition vector, the condition vector and a time consistency submerged space generation frame are subjected to iterative denoising to generate a target editing video, and dynamic feature modulation is performed in each denoising process; a video editing model based on text and image common editing efficiently edits a video under combined guidance of text and image prompts.

Description

Technical Field

[0001] The present invention relates to a video editing model, and particularly to a video editing model based on joint editing of text and images and a construction method thereof. Background Art

[0002] In recent years, generative AI has been widely recognized in the field of computer vision, mainly due to the emergence of diffusion models. Diffusion models generate high-quality images based on text prompts through a step-by-step denoising process, demonstrating powerful generative capabilities and outputting high-quality results. A typical image editing process involves two key steps: (1) using a pre-trained diffusion model to invert a real image into latent features; (2) modifying corresponding image regions by controlling the attention map during the denoising process. Currently, the generative capabilities of diffusion models in text-to-image synthesis have been fully verified. The success of diffusion models in text-to-image synthesis has prompted researchers to explore their applications in video generation and editing tasks.

[0003] Currently, in the field of video production, most video editing methods generate target videos based on text prompts. For example, the deterministic DDIM algorithm is usually used to convert a video into noise and then gradually generate an edited version according to the text input. Although these methods have achieved success in video editing, it is difficult to handle complex artistic styles and rare objects (which are often difficult to accurately describe in text) relying solely on text prompts, resulting in frequent suboptimal or unexpected outputs. Tune-A-Video, on the other hand, introduces a mechanism to adapt the Stable Diffusion model to video generation tasks based on diffusion models, thereby increasing the diversity of video editing results. However, this method is still limited to text-guided applications. Although it can generate visually rich results, it often exhibits high uncontrollability. For example, the content structure of the original video may be partially damaged, and temporal dimension information is lost during the generation process, resulting in poor visual coherence between adjacent frames.

[0004] In summary, the latest progress has been made in the field of text-based video editing. However, this text-guided video editing method, although partially meeting the requirements, still has difficulty in precisely controlling the shape and style of objects, and the edited videos are prone to problems such as poor visual coherence between adjacent frames. Summary of the Invention

[0005] The object of the present invention is to provide a video editing model based on joint editing of text and images and a construction method thereof, which improves the quality of video editing through the joint guidance of text and image prompts and the introduction of a dynamic hybrid feature fusion mechanism and an optical flow-guided mask fusion module.

[0006] To achieve the above object, the present technical solution provides a construction method of a video editing model based on joint editing of text and images, including the following steps: Use the original video, text prompt, image prompt, and target video as a training data set pair; Input multiple groups of training data set pairs into the video editing framework for training to obtain a video editing model. The video editing model introduces an optical flow-guided mask fusion module and a multi-modal feature recognition and segmentation module into the denoising diffusion implicit model; the original video is input into the denoising diffusion implicit model for forward diffusion and noise addition to obtain multiple frames of latent space generated frames, and the multiple frames of latent space generated frames are input into the optical flow-guided mask fusion module for inter-frame feature alignment to obtain temporally consistent latent space generated frames; the text prompt, image prompt, and original video are input into the multi-modal feature recognition and segmentation module for alignment and positioning to obtain a conditional vector, and the conditional vector and the temporally consistent latent space generated frames are iteratively denoised to generate the target edited video, and dynamic feature modulation is performed during each denoising process.

[0007] In a second aspect, the present solution provides a video editing method based on joint editing of text and images, including the following steps: Input the original video, text prompt, and image prompt into the video editing model based on joint editing of text and images for editing to obtain the edited video.

[0008] Compared with the prior art, the present technical solution has the following characteristics and beneficial effects: 1. The video editing model based on joint editing of text and images optimizes the design of the multi-modal feature recognition and instance segmentation module: The multi-modal feature recognition and instance segmentation module integrates the input of text, images, and videos into a unified semantic space, and uses object detection and segmentation technology to locate editable regions, realizing precise object localization and semantic understanding in complex video scenes, and further realizing flexible control of style, shape, and content during the video editing process.

[0009] 2. The video editing model based on joint editing of text and images introduces a dynamic hybrid feature fusion mechanism, enhancing the feature representation through dynamic weighting of the attention and convolutional cascade modules. The attention captures the global structure, and the convolution retains local details. The combination of the two improves the editing quality of complex scenes, that is, the fusion features of the attention and convolutional cascade modules overcome the limitations of single feature extraction, making the style and shape modification during the video editing process more natural and accurate.

[0010] 3. The video editing model based on joint editing of text and images adopts an optical flow-guided mask fusion module, which solves the problem of inter-frame discontinuity based on optical flow and occlusion-aware fusion. The optical flow-guided mask fusion mechanism uses optical flow to align adjacent frames, identifies unreliable regions through occlusion masks, and compensates with motion-aligned features to ensure smooth transition and coherence between video frames. Description of the Drawings

[0011] Figure 1It is a flowchart of a video editing method for a video editing model based on joint text and image editing according to an embodiment of the present application.

[0012] Figure 2 It is a schematic structural diagram of a multimodal feature recognition and segmentation module.

[0013] Figure 3 It is a schematic structural diagram of a self-attention and convolution cascade module.

[0014] Figure 4 It is a schematic structural diagram of an optical flow-guided mask fusion module.

[0015] Figure 5 It is a graph of ablation experiment data according to this solution.

[0016] Figure 6 It is a schematic hardware structure diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0017] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0018] Those skilled in the art should understand that in the disclosure of the present invention, the orientation or positional relationships indicated by the terms "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting the present invention.

[0019] Embodiment 1 This solution provides a video editing model based on joint text and image editing. The video editing model based on joint text and image editing efficiently edits videos under the joint guidance of text and image prompts, aiming to generate more accurate semantic representations and achieve precise control over the appearance and structure of objects. And to further solve the problems of temporal flicker and inter-frame inconsistency, an optical flow-guided mask fusion module is introduced into the video editing model based on joint text and image editing, and the unstable regions are effectively reconstructed and the temporal smoothness of the video output is improved by fusion.

[0020] Specifically, the construction method of the video editing model based on joint text and image editing provided by this solution includes the following steps: Use the original video, text prompt, image prompt, and target video as training data set pairs; Input multiple groups of training data set pairs into the video editing framework for training to obtain a video editing model, where the video editing model introduces an optical flow-guided mask fusion module and a multi-modal feature recognition and segmentation module into the denoising diffusion implicit model; the original video is input into the denoising diffusion implicit model for forward diffusion and noise addition to obtain multiple frames of latent space generated frames, and the multiple frames of latent space generated frames are input into the optical flow-guided mask fusion module for inter-frame feature alignment to obtain temporally consistent latent space generated frames; the text prompt, image prompt, and original video are input into the multi-modal feature recognition and segmentation module for alignment and positioning to obtain a conditional vector, and the conditional vector and the temporally consistent latent space generated frames are iteratively denoised to generate the target edited video, and dynamic feature modulation is performed during each denoising process.

[0021] This solution is modified based on the denoising diffusion implicit model to utilize multi-modal input to achieve temporally controllable and semantically controllable video editing effects, that is, the video editing model is constructed and improved based on the latent diffusion framework so that the video editing process can be guided by text, image, and original video information.

[0022] As Figure 2 shown, the multi-modal feature recognition and segmentation module of this solution aims to encode the text prompt, image prompt, and original video into a unified semantic space. The text prompt is converted into a semantic embedding to generate a text feature vector through the CLIP text encoder, and the image prompt generates an image feature vector through the CLIP image encoder; to replace the content of the original video, each video frame of the original video is processed by the detection and segmentation module to locate objects and generate corresponding segmentation masks, and the detected object regions and their masks are encoded as object feature vectors. The text feature vector, image feature vector, and object feature vector are integrated for conditional generation, and all the extracted features (including text and image prompts) are concatenated through a fully connected layer to obtain a unified conditional vector to ensure the effective integration of multi-modal input.

[0023] Specifically, the text prompt and image prompt are input into the CLIP encoder of the multi-modal feature recognition and segmentation module for encoding respectively to obtain a text feature vector and an image feature vector. The original video is input into the multi-modal feature recognition and segmentation module, and after passing through the backbone network, RPN candidate region network, and region alignment in sequence, the corresponding object feature vector is generated. The object feature vector, text feature vector, and image feature vector are jointly input into a multi-layer fully connected layer for concatenation to obtain a conditional vector. The formula for the multi-layer fully connected layer is as follows:

[0024] where FC represents the multi-layer connection layer, Concat() represents concatenation, f textDenote the text feature vector, f image_prompt Denote the image feature vector, f objects Denote the object feature vector, Denote the conditional vector.

[0025] Specifically, the CLIP encoder includes a CLIP text encoder and a CLIP image encoder. The text prompt is input into the CLIP text encoder for semantic embedding encoding to obtain the text feature vector, and the image prompt is input into the CLIP image encoder for image encoding to obtain the image feature vector.

[0026] Specifically, the backbone network, RPN candidate region network, and region alignment in the multi-modal feature recognition and segmentation module constitute the detection and segmentation module for the original video. Each video frame of the original video is input into the backbone network for feature extraction and then input into the RPN candidate region network for object localization and generate the corresponding segmentation mask. The segmentation masks in different video frames are aligned in regions to obtain the object feature vector.

[0027] In this solution, dynamic feature modulation is performed during the iterative denoising process of the denoising diffusion implicit model. Specifically, the denoising diffusion implicit model includes a noise prediction model and a decoder. In each downsampling and upsampling of the noise prediction model, self-attention, depthwise separable convolution, and cross-attention modules are inserted. Among them, self-attention and depthwise separable convolution focus on the global structural features and local detail features of the input features to generate fused features. The fused features and the conditional vector are input into the cross-attention module for attention focus to obtain the output features. The insertion of self-attention and depthwise separable convolution uses the self-attention mechanism to keep the global structure unchanged and uses depthwise separable convolution to keep the local detail content unchanged, which can be used to adaptively balance the extraction of global structural features and local detail features, which is crucial for tasks such as style transfer and shape editing.

[0028] Specifically, the iterative denoising of the denoising diffusion implicit model is conditioned on the conditional vector to denoise the temporally consistent latent space generated frames to generate the target edited video consistent with the image prompt and the text prompt. The noise prediction model in the denoising diffusion implicit model conditions on the conditional vector at each time step t to predict the noise for the latent feature z t in the temporally consistent latent space generated frames, and then denoises based on the noise predicted by the noise prediction model and the conditional vector. The formula for the noise prediction model to predict noise is as follows: ; where z t is the latent feature at the t-th time step, t represents the time step, denotes the conditional vector, denotes the predicted noise.

[0029] As Figure 3 shown, the noise prediction model of the denoising diffusion implicit model is a U-Net architecture. Self-attention, depthwise separable convolution, and cross-attention modules are introduced between each downsampling and upsampling of the noise prediction model. Among them, self-attention and depthwise separable convolution focus on global and local features of the input features to generate fused features. The fused features and the conditional vector are input into the cross-attention module for attention to obtain output features. The self-attention and depthwise separable convolution include parallel self-attention mechanism paths and convolution paths. The latent features of the frame generated by the temporal consistency latent space at each time step are used as residual features and input into the self-attention and depthwise separable convolution, and then enter the self-attention mechanism path respectively. Query feature Q, key feature K, and value feature V are generated through three 1×1 convolution layers. After calculating the global self-attention weights of the query feature Q, key feature K, and value feature V, the global structural features are output. The residual features input into the self-attention and depthwise separable convolution enter the convolution path respectively to perform depthwise separable convolution and then output local detail features. The local detail features and the global structural features are dynamically weighted and fused to obtain fused features; the fused features are used as the query features of the cross-attention module, and the conditional vector is used as the key feature and value feature of the cross-attention module.

[0030] Specifically, the formulas for generating query feature Q, key feature K, and value feature V by the residual features entering the self-attention mechanism path through three 1×1 convolution layers are as follows: ; where Q represents the query feature, Z t represents the latent feature at the t-th time step, and conv 1*1 () represents a 1*1 convolution; K represents the key feature, and V represents the value feature; The formula for calculating the global attention weight A is as follows: ; where d represents the feature dimension, Q represents the query feature, K represents the key feature, and softmax() represents the activation function.

[0031] Finally, the global structural features output by the self-attention path are: ; where A represents the global attention weight and V represents the value feature.

[0032] The residual features input into the self-attention and depthwise separable convolution enter the convolution path respectively to perform depthwise separable convolution and then output local detail features, which are expressed as: ; where Z tDenote the latent feature at the t-th time step, and Depthwise() represents depthwise separable convolution.

[0033] After dynamically weighting and fusing the local detail features and global structure features, the fused feature is represented as: Fout = Fself-attention + Fconv; where denotes the fused feature, denotes the global structure feature, denotes the local detail feature.

[0034] In this scheme, a self-attention and convolution cascade module is introduced into the noise prediction model of the denoising diffusion implicit model. The self-attention and convolution cascade module adopts a dynamic fusion mechanism to enable the balance between the global structure features and local detail features to be adaptively adjusted at each time step. This ensures that the video frames of the generated target edited video are accurate in style and coherent in space. The fused feature after being fused by the self-attention and convolution cascade module is used to update the latent feature z, guiding the diffusion process towards the denoised latent representation z0, and is further optimized during the denoising process by incorporating conditional information from the conditional vector. This information is injected through the cross-attention mechanism within the U-Net architecture at each time step, which ensures that the generated video frames are consistent with the diffusion process and match the specified style and content instructions encoded in the conditional vector. Finally, the denoised latent representation z0 is passed through the decoder D, and the decoder generates the video frames of the target edited video.

[0035] To enhance temporal consistency, the video editing model of this scheme introduces an optical flow-guided mask fusion module in the denoising diffusion implicit model to further enhance the temporal coherence between video frames. The optical flow-guided mask fusion module is based on the inter-frame feature alignment mechanism of optical flow and occlusion masks, extracts motion-aligned features from the latent features of the current frame from adjacent frames, and uses bidirectional frame information to supplement unreliable regions. Specifically, as Figure 4 shown, the current video frame, the previous video frame, and the next video frame of the original video are input into the optical flow-guided mask fusion module in parallel, and after performing optical flow estimation, occlusion mask estimation, time-based adjacent feature fusion, and occlusion-aware feature fusion in sequence, the latent space generated frame of the current frame is obtained.

[0036] That is, in the optical flow estimation stage of this scheme: the optical flow algorithm is used to calculate the latent features of the previous video frame and the latent features of the next video frame; in the occlusion mask estimation stage: the occlusion mask in the current video frame is calculated through forward-backward consistency checking; In the time-based adjacent feature fusion stage, the latent features of the previous video frame and the latent features of the next video frame are weighted and fused. In the occlusion-aware feature fusion stage, based on the occlusion mask, the fusion result of the weighted fusion of the latent features of the previous video frame and the latent features of the next video frame, and the current video features are fused to obtain the updated features of the current video frame, and the latent space generation frame of the current frame is generated based on the updated features of the current video frame.

[0037] Specifically, this solution adopts an optical flow-guided mask fusion module. This mechanism enables the occluded area to adaptively recover information from adjacent video frames while retaining the high-confidence content in the current video frame. For each video frame, the optical flow-guided mask fusion module uses the estimated forward and backward optical flow fields to align the adjacent video frames to the current video frame respectively ， The corresponding latent features will be aligned based on the confidence of the occlusion mask and selectively fused with the features of the current video frame to obtain the latent space generation frame of the current frame.

[0038] The fusion process of the optical flow-guided mask fusion module is defined as follows: ; where is the predicted occlusion mask, is the latent feature of the previous video frame, is the latent feature of the next video frame, Fuse() is weighted fusion, is the feature of the current video frame, is the updated feature of the current video frame.

[0039] The loss function of this solution is the mean square error (MSE) between the predicted noise and the real noise, and the specific form is as follows: ; where represents the actual noise, is the noise predicted by the model. By minimizing this loss function, the model learns to gradually denoise the signal and recover the effective image from Gaussian noise.

[0040] Embodiment 2 This solution provides an application method of a video editing model based on co-editing of text and image constructed according to Embodiment 1, that is, this solution provides a video editing method based on co-editing of text and image, including the following steps: Input the original video, text prompt, and image prompt into the video editing model based on co-editing of text and image for editing to obtain an edited video.

[0041] Figure 1This is an example illustration of the video editing model based on joint text and image editing in this solution. When performing video editing, this Figure 1 aims to edit the original video (a woman running by the river) based on a text prompt (man) and an image prompt (an image corresponding to the man). As Figure 1 shown, the original video is input into the denoising diffusion implicit model for forward diffusion and noise addition to obtain multiple frames of latent space generated frames Z. The multiple frames of latent space generated frames are input into the optical flow-guided mask fusion module for inter-frame feature alignment to obtain temporally consistent latent space generated frames Z'. The text prompt, image prompt, and original video are input into the multi-modal feature recognition and segmentation module for alignment and positioning to obtain a conditional vector. The conditional vector and the temporally consistent latent space generated frames are iteratively denoised to generate the target edited video, and dynamic feature modulation is performed during each denoising process, and finally the edited video (a man running by the river) is output.

[0042] In addition, to evaluate the contributions of the various modules of the video editing model based on joint text and image editing in this solution, this solution designed ablation experiments with three scenarios: (i) not using the image prompt, (ii) not using the feature fusion of the attention and convolutional cascade modules, and (iii) not using the optical flow-guided mask fusion module. The results are shown in Figure 5. Figure 5 The text prompt input in it is "a bald eagle with white head and tail is flying in the forest", and the image prompt is "an eagle with a consistent white head and tail". It is observed that in the absence of the image prompt (the third column), the appearance of the generated eagle is close to the correct form, but the head and tail show a "yellowish-white" hue, deviating from the "white" specified in the prompt. In the absence of the feature fusion of the attention and convolutional cascade modules (the fourth column), although the white of the head and tail is correctly rendered, the structural details of the tail are significantly degraded. These results highlight the importance of the image prompt in improving editing fidelity and indicate that the feature fusion of the attention and convolutional cascade modules enhances both the feature expression ability and adaptability. In the absence of the optical flow-guided mask fusion module (the fifth column), obvious temporal artifacts will appear in the edited video, such as jitter around the object boundary, especially in areas of fast movement or occlusion. In contrast, the video editing model based on joint text and image editing in this solution (the second column) achieves smoother transitions and better temporal semantic consistency.

[0043] Embodiment 3 This embodiment also provides an electronic device. Refer to Figure 6, including a memory 404 and a processor 402. A computer program is stored in the memory 404, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0044] Specifically, the above-mentioned processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0045] Among them, the memory 404 may include a mass storage 404 for data or instructions. The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402.

[0046] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the above application methods of the video editing model based on joint text and image editing or the construction method of the video editing model based on joint text and image editing.

[0047] Optionally, the above electronic device may further include a transmission device 406 and an input / output device 408. Among them, the transmission device 406 is connected to the above-mentioned processor 402, and the input / output device 408 is connected to the above-mentioned processor 402.

[0048] The transmission device 406 may be used to receive or send data via a network. Specific examples of the above network may include wired or wireless networks provided by a communication provider of the electronic device. In one example, the transmission device includes a network interface controller (NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one example, the transmission device 406 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] The input / output device 408 is used to input or output information. In this embodiment, the input information may be an original video, a text prompt, an image prompt, etc., and the output information may be an edited video, etc.

[0050] Optionally, in this embodiment, the above-mentioned processor 402 may be configured to execute the following steps through a computer program: Input the original video, text prompt, and image prompt into the video editing model based on joint text and image editing for editing to obtain an edited video.

[0051] It should be noted that for the specific examples in this embodiment, reference can be made to the examples described in the above embodiments and optional implementation manners, and they will not be elaborated herein.

[0052] Generally, various embodiments can be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the present invention can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, a microprocessor, or other computing devices. However, the present invention is not limited thereto. Although various aspects of the present invention can be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, as a non-limiting example, the blocks, devices, systems, technologies, or methods described herein can be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or a controller, or other computing devices, or some combination thereof.

[0053] Embodiments of the present invention can be implemented by computer software, which can be executed by a data processor of a mobile device, such as in a processor entity, or implemented by hardware, or implemented by a combination of software and hardware. A computer software or program (also referred to as a program product), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product can include one or more computer-executable components configured to perform the embodiments when the program runs. The one or more computer-executable components can be at least one software code or a part thereof. Additionally, in this regard, it should be noted that any block in the logical flow in the figure can represent a program step, or an interconnected logical circuit, block, and function, or a combination of a program step and a logical circuit, block, and function. The software can be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs. The physical media is a non-transitory medium.

[0054] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0055] The above embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for constructing a video editing model based on co - editing of text and images, characterized in that, Comprising the following steps: Comprising the following steps: Taking the original video, text prompt, image prompt, and target video as a training data set pair; Inputting multiple groups of training data set pairs into a video editing framework for training to obtain a video editing model, where the video editing model introduces an optical flow-guided mask fusion module and a multi-modal feature recognition and segmentation module into the denoising diffusion implicit model; the original video is input into the denoising diffusion implicit model for forward diffusion and noise addition to obtain multiple frames of latent space generated frames, and the multiple frames of latent space generated frames are input into the optical flow-guided mask fusion module for inter-frame feature alignment to obtain temporally consistent latent space generated frames; the text prompt, image prompt, and original video are input into the multi-modal feature recognition and segmentation module for alignment and positioning to obtain a conditional vector, and the conditional vector and the temporally consistent latent space generated frames are iteratively denoised to generate the target edited video, and dynamic feature modulation is performed during each denoising process.

2. The method for constructing a video editing model based on co-editing of text and images according to claim 1, wherein The text prompt and image prompt are input into the CLIP encoder of the multi-modal feature recognition and segmentation module to be encoded respectively to obtain a text feature vector and an image feature vector. The original video is input into the multi-modal feature recognition and segmentation module, and after passing through a backbone network, an RPN candidate region network, and region alignment in sequence, a corresponding object feature vector is generated. The object feature vector, text feature vector, and image feature vector are jointly input into a multi-layer fully connected layer for splicing to obtain a conditional vector.

3. The method for constructing a video editing model based on co-editing of text and images according to claim 2, wherein The CLIP encoder includes a CLIP text encoder and a CLIP image encoder. The text prompt is input into the CLIP text encoder for semantic embedding encoding to obtain a text feature vector, and the image prompt is input into the CLIP image encoder for image encoding to obtain an image feature vector.

4. The method for constructing a video editing model based on co-editing of text and images according to claim 2, wherein The backbone network, RPN candidate region network, and region alignment in the multi-modal feature recognition and segmentation module constitute a detection and segmentation module for the original video. Each video frame of the original video is input into the backbone network for feature extraction and then input into the RPN candidate region network for object localization and generation of corresponding segmentation masks. The segmentation masks in different video frames are region-aligned to obtain an object feature vector.

5. The method for constructing a video editing model based on co - editing of text and images according to claim 1, wherein, The denoising diffusion implicit model includes a noise prediction model and a decoder. Each downsampling and upsampling in the noise prediction model inserts a self-attention, depthwise separable convolution, and cross-attention module. The self-attention and depthwise separable convolution focus on the global structural features and local detail features of the input features to generate a fused feature, and the fused feature and the conditional vector are input into the cross-attention module for attention to obtain an output feature.

6. The method for constructing a video editing model based on co - editing of text and images according to claim 5, wherein, The self-attention and depthwise separable convolution includes a parallel self-attention mechanism path and a convolution path. The temporal consistency latent space at each time step generates the latent features of the frame as residual features input into the self-attention and depthwise separable convolution. They respectively enter the self-attention mechanism path and generate query feature Q, key feature K, and value feature V through three 1×1 convolution layers. After calculating the global self-attention weights for query feature Q, key feature K, and value feature V, the global structural features are output. The residual features input into the self-attention and depthwise separable convolution respectively enter the convolution path, and after depthwise separable convolution, the local detail features are output. The local detail features and the global structural features are dynamically weighted and fused to obtain the fused features. The fused features serve as the query features of the cross-attention module, and the conditional vector serves as the key feature and value feature of the cross-attention module.

7. The method for constructing a video editing model based on co - editing of text and images according to claim 1, wherein, The current video frame, the previous video frame, and the next video frame of the original video are input into the optical flow-guided mask fusion module in parallel, and after performing optical flow estimation, occlusion mask estimation, time-based adjacent feature fusion, and occlusion-aware feature fusion in sequence, the latent space generation frame of the current frame is obtained.

8. The method for constructing a video editing model based on co - editing of text and images according to claim 7, wherein, The optical flow algorithm is used to calculate the latent features of the previous video frame and the latent features of the next video frame; The occlusion mask in the current video frame is calculated through forward-backward consistency check; the latent features of the previous video frame and the latent features of the next video frame are weighted and fused; based on the occlusion mask, the fusion result of the weighted fusion of the latent features of the previous video frame and the latent features of the next video frame, and the current video features are fused to obtain the updated features of the current video frame, and the latent space generation frame of the current frame is generated based on the updated features of the current video frame.

9. A video editing method based on co - editing of text and images, characterized in that, It includes the following steps: The original video, text prompt, and image prompt are input into the video editing model for co-editing based on text and image to obtain the edited video, where the video editing model for co-editing based on text and image is constructed according to the construction method of the video editing model for co-editing based on text and image described in any one of claims 1 to 8.

10. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is set to run the computer program to execute the construction method of the video editing model for co-editing based on text and image described in any one of claims 1 to 8 or the video editing method for co-editing based on text and image described in claim 9.

Images