Intelligent video dynamic editing method based on diffusion model

Abstract

The invention discloses an intelligent video dynamic editing method based on a diffusion model. The method comprises the following steps: performing semantic analysis on a video editing instruction by using a large language model to obtain a modified object and modified content; detecting and segmenting the video by using the object detection model and the segmentation model to obtain a target object image, and generating an edge mask set and a main body mask set; inputting the modified object / content and the image into a CLIP model to obtain a CLIP latent vector; inputting the video into a U-Net encoder to generate a submerged space representation of the video; inputting the main body mask set, the CLIP latent vector and the latent space representation into a first content control module to generate a first initial latent vector, and performing latent space optimization to obtain a first latent vector; inputting the edge mask set and the first latent vector into a second content control module to obtain a second initial latent vector, obtaining a second latent vector by using latent space optimization, and decoding by using a U-Net decoder to obtain a final video; according to the invention, flexible and efficient video editing solutions can be provided for users in different fields.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video editing, and in particular to an intelligent video dynamic editing method based on a diffusion model. BACKGROUND

[0002] In modern image and video generation technology, deep learning models, especially generative adversarial networks (GAN), diffusion models, and large language models (LLM), have achieved widespread application. The progress of these technologies has made it possible to automatically generate high-quality images and videos. However, although these technologies perform well in certain application scenarios, there are still some technical bottlenecks when dealing with complex and diverse video editing tasks.

[0003] Specifically, traditional video editing tools usually rely on manual operation, requiring a lot of time and experience to adjust each detail. Especially when dealing with complex backgrounds, multi-target scenes, and requiring multiple rounds of editing iterations, it is often difficult to ensure the consistency of details and visual naturalness of the edited content. In addition, traditional tools have weak understanding and execution capabilities for complex user instructions, especially natural language instructions, making it difficult to meet the precise needs of users, with the following problems: 1. Fine editing problem in complex scenes: In video editing, especially when the video contains complex backgrounds or multi-target scenes, traditional methods are prone to edge distortion and object misplacement problems. This problem often leads to obvious visual incoordination between the edited content and the original video, affecting the overall quality of the final video; 2. Semantic understanding and consistency problem of edited content: In the video editing task, users' editing instructions are usually input in the form of natural language, and traditional editing tools often have difficulty accurately understanding the specific meaning of these instructions, leading to inconsistencies between the final editing results and user expectations; Detail optimization and visual coherence problem in multiple rounds of calculation: Traditional editing methods usually rely on one-time calculation, and when dealing with complex scenes, they tend to overlook some details, leading to inconsistencies in the generated content. Especially when dealing with dynamic video content, there may be problems such as frame skipping and misplacement; Therefore, how to realize a video editing method or system that can intelligently understand user intent and maintain high-quality output in complex scenarios is a problem that needs to be solved in the current technical field. SUMMARY

[0004] The present application provides an intelligent video dynamic editing method based on a diffusion model to overcome the above technical problems.

[0005] To achieve the above purpose, the technical solution of the present application is: An intelligent video dynamic editing method based on a diffusion model, comprising: S1: obtaining an original video to be edited and a video editing instruction; S2: introducing a large language model, an object detection model, and a segmentation model, performing semantic analysis on the video editing instruction using the large language model to obtain a modification object and a modification content, detecting the original video using the object detection model to identify the position of the modification object, and segmenting the modification object in the video according to the position of the modification object using the segmentation model to obtain a target object image and generate a time sequence edge mask set and a time sequence main mask set; S3: introducing a CLIP model, inputting the modification object, the modification content, and the target object image into the CLIP model for processing to obtain a CLIP latent vector corresponding to the video editing instruction; introducing a U-Net encoder, inputting the original video into the U-Net encoder for encoding to generate a latent space representation of the video; S4: introducing a first content control module, inputting the time sequence main mask set, the CLIP latent vector, and the latent space representation of the video into the first content control module for conditional fusion and constraint to generate a first starting latent vector; S5: introducing a latent space, inputting the first starting latent vector into the latent space, and optimizing the first starting latent vector under the joint constraint of the CLIP latent vector and the latent space representation of the video to obtain a first latent vector; S6: introducing a second content control module, inputting the time sequence edge mask set and the first latent vector into the second content control module for editing to obtain a second starting latent vector; inputting the second starting latent vector into the latent space for optimization to obtain a second latent vector; S7: introducing a U-Net decoder, inputting the second latent vector into the U-Net decoder for decoding to output a final video.

[0006] Further, the segmentation model is used to segment the modification object in the video according to the position of the modification object to obtain a target object image, and a time sequence edge mask set and a time sequence main mask set are generated, including: S21, decomposing the original video into single-frame images; S22, initializing the segmentation model and introducing a target guide condition, i.e., the position of the modification object; S23, using the segmentation model to segment each frame of image to generate an edge mask and a main mask, as shown in formulas (1) and (2), (1) (2) wherein, the edge mask obtained by segmentation is used to optimize boundary details; The subject mask obtained by segmentation, used to define the subject area; The boundary area focusing on the segmented object, The main content focusing on the segmented object, The input video frame, i.e. the decomposed single-frame image; S24, adjusting the inter-frame consistency of the edge mask and the subject mask; S25, temporally unifying the two types of masks in sequence according to the frame sequence, outputting a time sequence edge mask set and a time sequence subject mask set.

[0007] Further, the CLIP model is introduced, and the modified object, the modified content and the target object image are input into the CLIP model for processing to obtain a CLIP latent vector corresponding to the video editing instruction, including: S31, inputting the modified object and the modified content as text into the CLIP text encoder in the CLIP model to generate a text feature vector, as shown in formula (3), (3) Wherein, The text feature vector is represented; S32, inputting the target object image obtained by the object detection and segmentation model as an image into the CLIP image encoder to generate an image feature vector, as shown in formula (4), (4) Wherein, The image feature vector is represented, The segmented target object image is represented; S33, judging the matching degree of the text feature vector and the image feature vector, as shown in formula (5), (5) Wherein, The matching degree between the two feature vectors is represented; S34, if the matching degree meets the set matching threshold, the matched text feature vector and image feature vector are fused to obtain a fused CLIP latent vector; if the matching degree does not meet the set matching threshold, the model parameters of the CLIP model are changed until the matching degree meets the set matching threshold, and the matched text feature vector and image feature vector are fused to obtain a fused CLIP latent vector.

[0008] Further, the first content control module includes a Depth ControlNet model, a PoseControlNet model and a Color ControlNet model connected in sequence.

[0009] Further, the subject mask, the CLIP latent vector, and the latent space representation of the video are input into the first content control module for conditional fusion and constraint to generate a first starting latent vector, including: S41, in the content control module, the CLIP latent vector and the latent space representation of the video are fused to generate a new latent space representation, as shown in formula (6), (6) Wherein, represents the new latent space representation after fusion, represents the CLIP latent vector, represents the latent space representation of the video, represents the fusion function; S42, in the content control module, the new latent space representation is optimized using the subject mask to generate a first starting latent vector, as shown in formula (7), (7) Wherein, represents the first starting latent vector, represents the global consistency optimization function. Global optimization.

[0010] Further, the second content control module includes a Line Art ControlNet model and a TileControlNet model connected in sequence.

[0011] Further, the second starting latent vector is input into the latent space for optimization to obtain a second latent vector, including: The inter-frame consistency of the second starting latent vector is adjusted using a time dimension smoothing mechanism, as shown in formula (8), (8) Wherein, is the time smoothing weight, used to adjust the smoothing degree of the front and rear frames; represents the second starting latent vector; The adjusted second starting latent vector is input into the latent space for optimization to obtain a second latent vector, as shown in formula (9), (9) Wherein, represents the local detail optimization function, represents the global consistency optimization function, represents the second latent vector.

[0012] Further, the second latent vector is input into the U-Net decoder for decoding to output a final video, including: The second latent vector is input to a U-Net decoder for decoding to output a final video, as shown in formula (10), (10) wherein, represents a final video frame sequence, represents a U-Net decoder.

[0013] Advantages: The present application provides an intelligent video dynamic editing method based on a diffusion model, which has the following advantages: 1. Significantly improve editing accuracy and output quality: The present application solves the problem of loss of traditional methods in complex background editing content by fine processing of object detection and segmentation; this method can accurately separate the target object in a multi-object scene and perform independent editing operation, ensuring that the final generated video is visually natural and smooth, avoiding common edge distortion and object misplacement problems, and through the optimization mechanism of multiple rounds of latent space calculation, the detail expressiveness of the generated content can be gradually improved in the video editing process, ensuring the quality and coherence of each frame of picture; 2. Improve intelligence and user operation experience: Combined with the deep semantic understanding ability of the large language model, the method can accurately analyze and execute complex natural language instructions, so that users can realize high-quality video editing without deep technical knowledge; through the combination of CLIP coding and latent space calculation, the editing process can quickly respond to user needs and generate editing results highly consistent with user expectations; in addition, the automatic adjustment capability of the latent space greatly reduces the need for manual adjustment, making the editing process more intelligent and convenient, and improving the user experience; 3. Multiple rounds of latent space calculation ensure details and consistency: The present application uses a multiple rounds of latent space calculation mechanism to gradually optimize the generated content, significantly improving the detail expressiveness and overall consistency in complex scenes; the multiple iteration calculation design allows the edited image to be gradually improved in each round of calculation, ensuring that the output content reaches the ideal state in terms of visual effect, especially when processing dynamic video content, it can effectively avoid frame skipping and misplacement problems, ensuring the visual consistency and coherence of the final video; 4. Wide application prospect and commercial value: Because the present application can effectively handle complex editing tasks, it is suitable for film special effects, advertising creation, virtual reality and other high-demand fields, the high intelligence and efficient editing process of the present application provides powerful tool support for professional video production, greatly improving the efficiency and quality of creation; at the same time, the multi-module cooperation and innovative design make it have wide commercial application potential, which can provide flexible and efficient video editing solutions for users in different fields. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0015] Figure 1 A flow chart of an intelligent video dynamic editing method based on a diffusion model provided by the present application is shown in the figure. Figure 2 A data processing flow chart of an intelligent video dynamic editing method based on a diffusion model provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0016] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0017] The present embodiment provides an intelligent video dynamic editing method based on a diffusion model, as shown in the figure, which includes: Figure 1 S1: obtaining an original video to be edited and a video editing instruction; S2: introducing a large language model, an object detection model and a segmentation model, using the large language model to perform semantic analysis on the video editing instruction to obtain a modification object and a modification content; using the object detection model to detect the original video to identify the position of the modification object; using the segmentation model to segment the modification object in the video according to the position of the modification object to obtain a target object image, and generating a time sequence edge mask set and a time sequence main body mask set; S3: introducing a CLIP model, inputting the modification object, the modification content and the target object image into the CLIP model for processing to obtain a CLIP latent vector corresponding to the video editing instruction; introducing a U-Net encoder, inputting the original video into the U-Net encoder for encoding to generate a latent space representation of the video; S4: introducing a first content control module, inputting the time sequence main body mask set, the CLIP latent vector and the latent space representation of the video into the first content control module for conditional fusion and constraint to generate a first starting latent vector; ​S5: Introduce a latent space, input the first initial latent vector into the latent space, and optimize the first initial latent vector under the joint constraints of the CLIP latent vector and the latent space representation of the video to obtain the first latent vector; S6: Introduce a second content control module, input the time series edge mask set and the first latent vector into the second content control module for editing to obtain a second starting latent vector; input the second starting latent vector into the latent space for optimization to obtain a second latent vector; S7: Introduce the U-Net decoder, input the second latent vector into the U-Net decoder for decoding, and output the final video.

[0018] Specifically, such as Figure 2 As shown, the process first obtains the original video to be edited and the video editing instructions; the video editing instructions are input into a large language model for semantic analysis to obtain the object to be modified and the content to be modified; an object detection model is used to detect the original video and identify the location of the object to be modified; a segmentation model is used to segment the object to be modified in the video according to its location to obtain the target object image, and a time series edge mask set and a time series subject mask set are generated to ensure that the edges and subjects can be processed independently and precisely in subsequent editing processes, avoiding interference or distortion in complex backgrounds; Secondly, the modified object, modified content, and target object image are input into the CLIP model for processing to obtain CLIP latent vectors corresponding to the video editing instructions. The original video is input into the U-Net encoder for encoding to generate the latent space representation of the video. The temporal series subject mask set, CLIP latent vectors, and the video's latent space representation are input into the first content control module for conditional fusion and constraints to generate an optimized latent space, i.e., the first initial latent vector. The latent space is then introduced, and the first initial latent vector is input into it. Under the joint constraints of the CLIP latent vector and the video's latent space representation, the first initial latent vector is optimized to obtain the first latent vector. The CLIP encoding and U-Net encoding processes are the preparatory stages for vectors to enter the latent space for optimization, providing high-quality latent space feature representations for subsequent editing. Through cross-modal feature gating fusion and multi-objective progressive optimization, semantically accurate and temporally coherent latent space representation generation is achieved, providing high-quality initial conditions for subsequent video generation / editing tasks. Video editing can achieve large-scale content adjustment and replacement in the initial calculation stage, laying a solid foundation for subsequent detailed optimization. Next, the time series edge mask set and the first latent vector are input into the second content control module for editing to obtain the second starting latent vector; the second starting latent vector is input into the latent space for optimization to obtain the second latent vector; this significantly improves the video's detail and visual consistency, ultimately achieving a highly realistic editing effect; Finally, the second latent vector is input to the U-Net decoder for decoding, and the final video is output.

[0019] In specific embodiments, the scheme for obtaining the original video to be edited and the video editing instruction is: In this scheme, the user determines the original video that needs to be edited and the instruction that wants to edit, for example, the user uploads a video of a man riding a horse and inputs the instruction: "replace the man with a knight wearing medieval armor." In specific embodiments, a large language model, an object detection model, and a segmentation model are introduced, the large language model is used for semantic analysis of the video editing instruction to obtain the modification object and the modification content; the object detection model is used for detection of the original video to identify the location of the modification object; and the segmentation model is used to segment the modification object in the video according to the location of the modification object to obtain the target object image, and a scheme for generating a time sequence edge mask set and a time sequence main body mask set is: 1. Introduce a large language model, the large language model used in this scheme is ChatGPT, and ChatGPT is used for semantic analysis of the input instruction, for example, it identifies that the object to be modified is "man" and the content to be modified is "wearing medieval armor"; the analysis result is used for subsequent object detection and segmentation steps; 2. Introduce an object detection and segmentation model, the object detection used in this scheme is GroundingDino, and the segmentation model is SAM model; GroundingDino is used to detect the location of "man" in the video, and SAM model is used to finely segment "man" in the video to generate two different types of masks: edge mask and main body mask. The edge mask is used for repairing the edge details in the subsequent processing process, and the main body mask is used for modifying the main content in the video, i.e., replacing "man" with a knight wearing armor. The specific processing steps of the segmentation model are as follows: S21, decompose the original video into single-frame images; S22, initialize the segmentation model, and introduce the target guide condition, i.e., the location of the modification object; S23, use the segmentation model to segment each frame of image to generate edge mask and main body mask, as shown in formulas (11) and (12), (11) (12) wherein, represents the edge mask obtained by segmentation, which is used to optimize the boundary details; represents the main body mask obtained by segmentation, which is used to clarify the main body area; represents the boundary area focusing on the segmentation object, representing the main content focusing on the segmented object, representing the input video frame, i.e., the decomposed single-frame image; S24, adjusting the inter-frame consistency of the edge mask and the main body mask to ensure the consistency of the time sequence; S25, performing time sequence unification on the adjusted two types of masks according to the frame sequence, and outputting a time sequence edge mask set and a time sequence main body mask set, specifically including: S251: frame-by-frame standardization and alignment: After time sequence correction, each frame of main body mask and edge mask is subjected to threshold cleaning, hole filling and edge smoothing operation, and the processed mask is ensured to correspond to the frame image one by one (consistent in width, height and coordinate system); S252: According to the time sequence correction result of S24, a stable time index is established for each group of masks (main body mask and edge mask corresponding to the frame image), ensuring that there is no cross-frame jump and ensuring that the subsequent content control and latent space stage can use "stable" mask sequence; S253: Finally, two ordered sequences, i.e., a time sequence edge mask set and a time sequence main body mask set, are output in the order of frame sequence t=1…T.

[0020] In this scheme, object detection model and segmentation model are used to generate edge mask and main body mask for subsequent video processing. The edge mask is used to maintain and repair the edge details of the video object, while the main body mask is used to mark the area that needs to be replaced in the video. This scheme ensures that the edge and main body parts can be independently and finely processed in the subsequent editing process, avoiding interference or distortion in complex backgrounds.

[0021] In specific embodiments, a CLIP model is introduced, and the modification object, the modification content and the target object image are input into the CLIP model for processing to obtain a CLIP latent vector corresponding to the video editing instruction; a U-Net encoder is introduced, and the original video is input into the U-Net encoder for encoding to generate a latent space representation of the video, which is: S31, inputting the modification object and the modification content as text into the CLIP text encoder in the CLIP model to generate a text feature vector, as shown in formula (13), (13) wherein, represents the text feature vector; these vectors are the representation of the instruction in a high-dimensional space, which contains the user's expectation of video editing and represents the visual features mentioned in the instruction, such as the features of "medieval armor"; S32, input the target object image obtained by the object detection and segmentation model into the CLIP image encoder as image input, and generate an image feature vector, as shown in formula (14), (14) wherein, represents the image feature vector, represents the segmented target object image; S33, judge the matching degree of the text feature vector and the image feature vector, as shown in formula (15), (15) wherein, represents the matching degree between the two feature vectors; S34, if the matching degree meets the set matching threshold, the matched text feature vector and image feature vector are fused to obtain a fused CLIP latent vector; if the matching degree does not meet the set matching threshold, the model parameters of the CLIP model are changed until the matching degree meets the set matching threshold, and the matched text feature vector and image feature vector are fused to obtain a fused CLIP latent vector; S35, introduce a U-Net encoder, input the original video into the U-Net encoder for encoding to generate a latent space representation of the video; In the scheme, the CLIP encoding and U-Net encoding processes are the preparation stage of vector entering the latent space optimization, which provides high-quality latent space feature representation for subsequent editing.

[0022] In specific embodiments, a first content control module is introduced, and the time sequence subject mask set, the CLIP latent vector and the latent space representation of the video are input into the first content control module for conditional fusion and constraint to generate a first starting latent vector. The scheme is: The first content control module includes a Depth ControlNet model, a Pose ControlNet model and a Color ControlNet model connected in sequence; The time sequence subject mask set, the CLIP latent vector and the latent space representation of the video are input into the first content control module for conditional fusion and constraint, and the specific steps are as follows: S41, fuse the CLIP latent vector and the latent space representation of the video in the content control module to generate a new latent space representation, as shown in formula (16), (16) wherein, represents the new latent space representation after fusion, represents the CLIP latent vector, a latent space representation of the video, representing a fusion function; The specific fusion process includes: S411: input the time sequence subject mask set into the Depth ControlNet model for processing, output a depth condition, input the depth condition and the CLIP latent vector into the Pose ControlNet model, output a pose condition, input the latent space representation of the video and the pose condition into the Color ControlNet model, output a color condition; S412: weight and fuse the depth condition, the pose condition and the color condition to obtain a new fused latent space representation; S42, optimize the new latent space representation using the subject mask in the content control module to generate a first starting latent vector, as shown in formula (17), (17) wherein, represents the first starting latent vector, represents a global consistency optimization function.

[0023] In the scheme, a plurality of ControlNet models are used to form a first content control module. The scheme realizes the generation of a semantic accurate and time sequence coherent latent space representation through cross-modal feature gating fusion and multi-target progressive optimization, and provides a high-quality initial condition for subsequent video generation / editing tasks.

[0024] In specific embodiments, a latent space is introduced, and the first starting latent vector is input into the latent space. Under the joint constraint of the CLIP latent vector and the latent space representation of the video, the first starting latent vector is optimized to obtain a first latent vector. The scheme is: S51, divide the original latent space into regions using the subject mask: the region covered by the mask is identified as a "target editing area", and the remaining region is identified as a "maintaining area"; S52, inject the first starting latent vector generated by the first content control module into the target editing area only, to replace or enhance the latent space features of the area; the maintaining area continues to use the original latent space features to maintain the background coherence; Through the above selection-injection process, the accurate separation of the editing range and the non-editing range is realized before the first round of optimization in the latent space, so as to ensure that the subsequent calculation can complete the overall replacement and pose alignment of the target object, and will not damage the background consistency of the original video; S53, inject the CLIP latent vector and the latent space representation of the video into the latent space as a global semantic condition: In the first round of optimization in the latent space, the CLIP latent vector is input into the diffusion network together with the first starting latent vector, and the latent space features are constrained through cross attention to ensure that the visual content generated in the target editing area is completely consistent with the user's instructions in terms of image, clothing, and other semantic levels.

[0025] This scheme mainly adjusts the overall visual effect of the video preliminarily to make it more consistent with the user's instruction requirements; the innovation of this link lies in that through multi-model cooperation, video editing can realize large-scale content adjustment and replacement in the preliminary calculation stage, laying a solid foundation for subsequent detailed optimization.

[0026] In specific embodiments, a second content control module is introduced, and the time sequence edge mask set and the first latent vector are input into the second content control module for editing to obtain a second starting latent vector; the second starting latent vector is input into the latent space for optimization to obtain a second latent vector. The second content control module includes a Line Art ControlNet model and a TileControlNet model connected in sequence; The time sequence edge mask set and the first latent vector are input into the second content control module for editing to obtain a second starting latent vector, and the specific steps include: S61, downsample and feature extract the edge masks in the time sequence edge mask set, channel splice and fuse the extracted features with the first latent vector, and input them into the Line Art ControlNet model for processing to obtain edge enhancement features; S62, block locally process the first latent vector, input the processed vector and the edge enhancement features into the Tile ControlNet model for processing, and output a second starting latent vector; S63, input the second starting latent vector into the latent space for optimization to obtain a second latent vector, which specifically includes: Adjust the inter-frame consistency of the second starting latent vector using a time dimension smoothing mechanism, as shown in equation (18), (18) Wherein, is a time smoothing weight used to adjust the smoothing degree of the previous and subsequent frames; represents the second starting latent vector; Input the adjusted second starting latent vector into the latent space for optimization to obtain a second latent vector, as shown in equation (19), (19) Wherein, represents a local detail optimization function, denotes a global consistency optimization function, denotes a second latent vector.

[0027] In this scheme, the second round of optimization is performed in the latent space, and the specific optimization is as follows: 1. Target object detail restoration: (1) Feature enhancement: further enhance the texture features of the target object (such as the light and shadow effect of the armor, the reflection effect), ensure the details are exquisite; (2) Action fluency: adjust the action features of the target object (such as the body posture when the knight rides a horse), ensure that each frame of action is natural and smooth; 2. Edge area fine processing: Again, the edge area is detected and optimized: repair possible boundary breakage problems; adjust the color and brightness of the boundary transition to ensure that the edge and background are more naturally integrated.

[0028] 3. Global consistency optimization (1) Background and target integration: match the light, shadow and color of the target object and background area to avoid the target object appearing inconsistent. Ensure that the shadow and reflection of the target object conform to the lighting conditions of the background.

[0029] (2) Multi-frame consistency adjustment: analyze the latent space representation between adjacent frames in the time sequence to check whether there are target object jumping or texture inconsistency problems; This round of optimization focuses on the restoration of details and the consistency of content, ensuring that the generated knight image in the video maintains consistent action and posture with the original "man", and the medieval armor worn in each frame naturally integrates with the surrounding environment; Each round of calculation can significantly improve the detail performance and visual consistency of the video, and finally achieve a highly realistic editing effect; This innovative mechanism effectively solves the problems of insufficient detail optimization and visual inconsistency in traditional editing methods, and through multiple rounds of calculation, the generated content is continuously refined and optimized, and the final output result is not only a preliminary generation, but also a high-quality video after multiple fine adjustments; This method performs well in handling complex scenes and high-demand editing tasks, significantly improving the detail performance and overall quality of the video.

[0030] In specific embodiments, a U-Net decoder is introduced, and the second latent vector is input into the U-Net decoder for decoding, and the scheme for outputting the final video is: The second latent vector is input into the U-Net decoder for decoding, and the final video is output, as shown in equation (20), (20) wherein, denotes the final video frame sequence, represents a U-Net decoder.

[0031] After all editing and optimization in the latent space, the latent representation is reconverted into actual video frames by the U-Net decoding module. The decoding process not only restores the visual content of the video, but also ensures the temporal continuity and visual consistency between video frames. The U-Net decoder decodes the high-dimensional information in the latent space, so that the final output video content can be seamlessly integrated with the original video in terms of vision, maintaining high-quality visual effects.

[0032] In summary, through this multi-module collaborative, multi-round calculation and optimization working principle, the application realizes efficient processing of complex video editing tasks, significantly improving the quality and consistency of video generation. This method has wide application potential in the fields of movie special effects, advertising creation, virtual reality, etc. due to its editing ability in complex scenes, and can meet the demand for highly refined control of video content.

[0033] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A method for intelligent video dynamic editing based on a diffusion model, characterized in that, include: S1: Obtain the original video to be edited and the video editing instructions; S2: Introduce a large language model, an object detection model, and a segmentation model. Use the large language model to perform semantic analysis on the video editing instructions to obtain the modified object and the modified content. The original video is used to detect and identify the location of the modified object; The segmentation model is used to segment the modified object in the video based on the location of the modified object, obtain the target object image, and generate a time series edge mask set and a time series main body mask set; S3: Introduce the CLIP model, input the modified object, modified content, and target object image into the CLIP model for processing to obtain the CLIP latent vector corresponding to the video editing instructions; Introduce the U-Net encoder, input the original video into the U-Net encoder for encoding to generate the latent space representation of the video; S4: Introduce the first content control module, input the time series main mask set, CLIP latent vector and the latent space representation of the video into the first content control module for conditional fusion and constraints, and generate the first initial latent vector; S5: Introduce a latent space, input the first initial latent vector into the latent space, and optimize the first initial latent vector under the joint constraints of the CLIP latent vector and the latent space representation of the video to obtain the first latent vector; S6: Introduce a second content control module, input the time series edge mask set and the first latent vector into the second content control module for editing to obtain a second starting latent vector; input the second starting latent vector into the latent space for optimization to obtain a second latent vector; S7: Introduce the U-Net decoder, input the second latent vector into the U-Net decoder for decoding, and output the final video.

2. The intelligent video dynamic editing method based on a diffusion model according to claim 1, characterized in that, The segmentation model is used to segment the modified object in the video based on its location, obtaining the target object image, and generating a time-series edge mask set and a time-series subject mask set, including: S21. Decompose the original video into single-frame images; S22. Initialize the segmentation model and introduce target guiding conditions, i.e., modify the position of the object; S23. Use a segmentation model to segment each frame of the image, generating edge masks and subject masks, as shown in formulas (1) and (2). （1） （2） in, This represents the edge mask obtained from segmentation, used to optimize boundary details; This represents the main mask obtained from the segmentation, used to define the main subject area; This indicates a focus on segmenting the boundary region of the object. This indicates a focus on the main content of the segmented object. This represents the input video frame, i.e., the decomposed single-frame image; S24. Adjust the inter-frame consistency of the edge mask and the main mask; S25. The two types of masks are adjusted and time-series unified according to frame order, and the time-series edge mask set and the time-series main mask set are output.

3. The intelligent video dynamic editing method based on a diffusion model according to claim 2, characterized in that, The CLIP model is introduced, where the object to be modified, the content to be modified, and the target object image are input into the CLIP model for processing to obtain CLIP latent vectors corresponding to video editing instructions, including: S31. Input the modified object and the modified content as text into the CLIP text encoder in the CLIP model to generate text feature vectors, as shown in formula (3). （3） in, Represents the text feature vector; S32. The target object image obtained by the object detection and segmentation model is used as the image input and fed into the CLIP image encoder to generate the image feature vector, as shown in formula (4). （4） in, Represents the image feature vector. Represents the segmented target object image; S33. Determine the degree of matching between the text feature vector and the image feature vector, as shown in formula (5). （5） in, This indicates the degree of matching between two feature vectors; S34. If the matching degree meets the set matching threshold, the matched text feature vector and image feature vector are fused to obtain the fused CLIP latent vector; if the matching degree does not meet the set matching threshold, the model parameters of the CLIP model are changed until the matching degree meets the set matching threshold, and the matched text feature vector and image feature vector are fused to obtain the fused CLIP latent vector.

4. The intelligent video dynamic editing method based on a diffusion model according to claim 1, characterized in that, The first content control module includes the Depth ControlNet model, the Pose ControlNet model, and the ColorControlNet model, which are connected in sequence.

5. The intelligent video dynamic editing method based on a diffusion model according to claim 1, characterized in that, The main mask, CLIP latent vector, and latent space representation of the video are input into the first content control module for conditional fusion and constraints to generate the first initial latent vector, including: S41. In the content control module, the CLIP latent vector and the latent space representation of the video are fused to generate a new latent space representation, as shown in formula (6). （6） in, This represents the new latent space after fusion. Represents the CLIP latent vector. Representing the latent space of a video, Indicates the fusion function; S42. In the content control module, the main mask is used to optimize the new latent space representation and generate the first initial latent vector, as shown in formula (7). （7） in, Denotes the first initial latent vector. This represents the globally consistent optimization function.

6. The intelligent video dynamic editing method based on a diffusion model according to claim 1, characterized in that, The second content control module includes a Line Art ControlNet model and a Tile ControlNet model connected in sequence.

7. The intelligent video dynamic editing method based on a diffusion model according to claim 5, characterized in that, The second initial latent vector is input into the latent space for optimization to obtain the second latent vector, which includes: The inter-frame consistency of the second initial latent vector is adjusted using a time-dimensional smoothing mechanism, as shown in Equation (8). （8） in, It is a temporal smoothing weight, used to adjust the smoothness between consecutive frames; Indicates the second initial latent vector; The adjusted second initial latent vector is input into the latent space for optimization to obtain the second latent vector, as shown in formula (9). （9） in, This represents a local detail optimization function. This represents the global consistency optimization function. This represents the second latent vector.

8. The intelligent video dynamic editing method based on a diffusion model according to claim 7, characterized in that, The second latent vector is input into the U-Net decoder for decoding, and the final video is output, including: The second latent vector is input into the U-Net decoder for decoding, and the final video is output as shown in formula (10). （10） in, This represents the final video frame sequence. This indicates the U-Net decoder.

Images