Multi-object video editing method based on semantic perception content alignment

Abstract

According to the multi-object video editing method based on semantic perception content alignment provided by the invention, a precise semantic-level editing effect with consistent time sequence can be realized in a multi-object video without training a diffusion model. Effective semantic alignment is realized through two core mechanisms of semantic adaptive modulation and semantic prior modeling. Specifically, coupling semantic features among multiple objects are effectively decoupled through an attention regulation and control mechanism of region perception and text perception, so that the model can accurately position the spatial position and semantic attribute of each object; meanwhile, cross-modal semantic consistency supervision is constructed based on a pre-trained visual language model, and the diffusion sampling process is guided to keep global semantic consistency with text editing prompt. Experimental results show that the method is remarkably superior to the prior art in the aspects of semantic alignment, editing precision and visual consistency, can be widely applied to the fields of film and television production, human-computer interaction, digital media creation and the like, and has excellent practical value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to image processing and video synthesis technology. Specifically, the present application relates to a multi-object video editing method based on semantic perception content alignment, aiming to realize multi-object semantic accurate alignment, consistent structure and appearance, and time sequence stable high-quality video editing effect by introducing semantic adaptive modulation and semantic prior modeling mechanism in the video editing process, so as to be applicable to multi-object semantic editing tasks of any input video without additional training. BACKGROUND

[0002] With the breakthrough progress of diffusion models in the field of image generation, its application in video generation and video editing tasks has also made remarkable progress. Currently, text-driven video editing can be roughly divided into two technical routes: Fine-tuning method and Training-free method.

[0003] Early Fine-tuning editing methods usually introduce spatio-temporal modeling modules in U-Net, and fine-tune the model using large-scale training data to improve the generation quality and cross-frame consistency in single-object scenes. However, such methods are highly dependent on large-scale text-video datasets and are accompanied by huge computational costs, making it difficult to popularize in practical applications.

[0004] To reduce data requirements, Tune-A-Video extends the latent space diffusion model to the spatio-temporal domain, implicitly models the video motion of single objects through one-shot fine-tuning, and to some extent improves the temporal consistency. On this basis, EI 2 Further introduces spectral normalization to enhance the spatio-temporal feature interaction ability of single objects, thereby improving the generation coherence and overall quality of the video. In addition, VMC captures the motion trajectory of single objects in the video by designing a motion distillation loss target, thereby achieving more efficient motion preservation. CCEdit divides the modeling process into three parallel branches of structure, appearance and style, and introduces a temporal modeling module to strengthen the cross-frame dependency, thereby achieving high-quality and time-consistent editing effect.

[0005] Although a series of one-shot methods have greatly reduced the dependence on massive training data, their high computational cost and lack of multi-object spatial relationship perception ability make it difficult for current fine-tuning methods to handle more complex multi-object video editing scenarios.

[0006] Compared with the fine-tuning method, the training-free video editing does not need to perform any additional optimization on the model parameters, significantly reduces the computational cost, and has wider practical application potential. For example, FateZero maintains the overall appearance consistency of a single object by attention mixing; Ground-A-Video introduces a depth map and an optical flow field as conditional information to maintain the structural consistency of a single object across frames. FLATTEN proposes an optical flow guided attention mechanism, which effectively improves the visual consistency of the edited video.

[0007] TokenFlow realizes cross-frame information transmission through linear combination of features based on single-object time sequence correspondence, but is prone to introduce video blur artifacts. To solve this problem, VidToMe adopts a cross-frame self-attention token aggregation strategy to reduce temporal redundancy and improve generation quality. Subsequently, VideoGrain realizes control over video content by designing a spatio-temporal attention mechanism, further realizing multi-granularity video editing capabilities. In addition, some researches based on continuous editing paradigm introduce ordinary differential equations into the editing process to maintain the structural consistency of a single object.

[0008] Although the above methods improve the visual quality of video editing to some extent, due to the lack of explicit modeling of the spatial relationship of multiple objects, it is still difficult to provide stable and accurate semantic control in the diffusion sampling process, often causing object confusion, semantic leakage and other problems. At the same time, these methods also have obvious shortcomings in cross-frame structural consistency and object detail preservation, which easily lead to semantic misplacement, structure damage and temporal inconsistency, and the core problem of precise multi-object semantic editing has not been effectively solved. SUMMARY

[0009] To solve the above problems, the present application provides a multi-object video editing method based on semantic-aware content alignment, which comprises the following steps:

[0010] 1. A multi-object video editing method based on semantic-aware content alignment, characterized in that it comprises the following steps:

[0011] S1, constructing a video latent space representation and performing spatio-temporal feature inversion, latent space encoding and DDIM inversion of the video to be edited to obtain initial noise latent variables containing video motion and structure information;

[0012] S2, constructing a region-aware attention modulation based on spatial decoupling of self-attention, so that the model can effectively separate the features of multiple objects by enhancing the focus of regional internal features and suppressing inter-regional interference;

[0013] S3, construct instance positioning of cross attention based on text-aware attention modulation, so that the model can focus on enhancing the features of the text semantic related area and suppress the interference of irrelevant areas, and accurately align the text semantics and instance area;

[0014] S4, introduce semantic prior modeling, construct cross-modal semantic consistency supervision through visual language model to constrain the semantic alignment in the diffusion sampling process;

[0015] S5, perform diffusion sampling and generate the final edited video, so that the output video is consistent with the edited text prompt in appearance, structure, semantics and temporal consistency.

[0016] 2. The step S1 specifically comprises the following steps:

[0017] S11, map the input video frame to the latent space representation through the latent space encoder ;

[0018] S12, adopt DDIM inversion to project to the corresponding noise latent variable so that the subsequent editing process can be controllable generation while maintaining the overall structure and motion consistency of the video.

[0019] 3. The step S2 specifically comprises the following steps:

[0020] S21, in the t-step denoising process, calculate the self-attention similarity matrix:

[0021] (1)

[0022] Wherein and are the visual feature query Query and key Key respectively;

[0023] S22, introduce region-aware modulation templates to distinguish intra-regional and inter-regional semantics, which is expressed as:

[0024] (2)

[0025] (3)

[0026] Wherein and represent the spatial index of the query Query and the key Key respectively; and measure the deviation of the attention value relative to its maximum and minimum value respectively, which is used to realize the intra-regional consistency enhancement and inter-regional interference suppression;

[0027] S23, construct Query-Key condition mapping, limit attention to the area range of the current instance, and introduce fuzzy membership function to continuous process the area boundary, so that the attention realizes smooth transition at the edge of the instance, which is in the form of:

[0028] (4)

[0029] (5)

[0030] S24, based on the regional membership relationship, execute regional zoom-in and regional suppression, and obtain the final regional adaptive attention:

[0031] (6)

[0032] (7)

[0033] wherein is a normalization operation;

[0034] 4. The step S3 specifically comprises the following steps:

[0035] S31, in the t-step denoising process, calculate the cross-attention similarity matrix:

[0036] (8)

[0037] wherein and are visual feature query Query and text embedding key Key respectively;

[0038] S32, construct a text-aware modulation template for distinguishing attention responses related to and unrelated to the text, which is expressed as follows:

[0039] (9)

[0040] (10)

[0041] wherein and measure the deviation between the attention value and its upper and lower bounds, respectively, for strengthening the visual area aligned with the target text and suppressing irrelevant text semantics;

[0042] S33, design Query-Key cross-modal alignment mapping, and continuous process it through fuzzy membership function to ensure the smoothness of cross-modal mapping, which can be expressed as:

[0043] (11)

[0044] (12)

[0045] wherein represents a visual area related to the text mark ;

[0046] S34, modifying the cross-attention similarity map by combining , and to achieve text adaptive attention modulation and obtain the final text-aware attention:

[0047] (13)

[0048] (14)

[0049] wherein is a normalization operation.

[0050] 5. The step S4 specifically comprises the following steps:

[0051] S41, for a given text prompt y, the generated video content x is expected to satisfy both the generation prior distribution p(x) of the pre-trained diffusion model and be consistent with y at the semantic level, therefore, the editing process is modeled as a posterior inference problem:

[0052] (15)

[0053] And the posterior maximization can be expressed as:

[0054] (16)

[0055] S42, input the editing text prompt y into the text encoder of the pre-trained visual language model to obtain the text semantic embedding, and at the same time, input the visual content corresponding to the current generated video frame or intermediate latent variable into the same visual language model to obtain its semantic description embedding, which can be expressed as:

[0056] (17)

[0057] (18)

[0058] S43, based on the text semantic embedding and the generated semantic embedding, a semantic likelihood term is constructed to measure the semantic matching degree between the generated content and the text target, and the cosine similarity between the two is used as a consistency measure, and then the semantic likelihood term is constructed in the form of an exponential, which is expressed as follows:

[0059] (19)

[0060] S44, Design a semantic consistency loss function based on the semantic likelihood term, and use it as a semantic supervision signal for the diffusion sampling process. The smaller the loss, the more the generated content conforms to the text requirements. During the diffusion inverse sampling process, update the latent variables according to the gradient of the loss to gradually strengthen semantic alignment, as follows:

[0061] (20)

[0062] in The loss function is used to progressively update the latent space noise.

[0063] 6. Step S5 specifically includes the following steps:

[0064] S51, integrate the sampled video frames that have undergone the above semantic modulation and semantic prior optimization, and combine them according to the time order of the original video to generate the final edited video;

[0065] S52, output the final edited video, so that it achieves high-quality performance in terms of semantic consistency, object structure stability, appearance fidelity and temporal continuity, realizing high-precision multi-object semantic editing that can be applied to any input video without additional training.

[0066] Beneficial Effects: Compared with existing technologies, this invention provides a multi-object video editing method based on semantically aware content alignment, which can achieve high-quality video editing with semantic accuracy, structural stability, temporal consistency, and appearance fidelity, and has the following beneficial effects:

[0067] 1. Significantly improves multi-object semantic alignment capabilities

[0068] This invention constructs a region-aware attention modulation mechanism to explicitly spatially decouple the self-attention of the diffusion model, enabling clear differentiation of the visual features of each object in the attention space. This avoids the problems of mixed multi-object representations and severe semantic coupling in traditional methods. This mechanism effectively improves the model's ability to understand the spatial relationships of multiple objects, achieving accurate object-level localization and precise execution of editing instructions, and significantly enhancing the semantic controllability of multi-object editing.

[0069] 2. Enhance cross-modal text-visual semantic consistency

[0070] This invention designs a text-aware attention modulation module that enhances the response of relevant text regions and suppresses interference from irrelevant regions, achieving a precise correspondence between text semantics and object regions. This mechanism effectively solves common problems in traditional methods such as text semantic leakage, attention shift, and attribute misbinding, enabling the editing results to maintain higher consistency with the target text. It is particularly suitable for fine-grained editing tasks under complex text instructions.

[0071] 3. Introduce semantic prior modeling to improve editing stability and consistency.

[0072] This invention constructs a cross-modal semantic prior based on a visual language model. By maximizing the consistency between textual semantics and generated semantics, it constrains the diffusion sampling process, ensuring that the sampling trajectory continuously converges towards a direction consistent with the textual semantics at each step. This mechanism effectively alleviates common problems in traditional diffusion editing, such as semantic drift and expression instability, and significantly improves the semantic stability and consistency of the generated video.

[0073] 4. Maintain video structural integrity and temporal consistency.

[0074] This invention combines spatial modulation, text modulation, and semantic prior knowledge in the diffusion inverse sampling process, effectively improving the consistency of video in terms of structural details, object appearance, and cross-frame temporal sequence. This results in a more natural, coherent, stable, and reliable visual presentation in the final generated video. Compared with traditional methods, this invention demonstrates stronger generalization ability and stability in multi-scene, cross-attribute, and multi-category video editing tasks.

[0075] 5. Possesses broad practical application value.

[0076] This invention can be widely applied in film and television production, virtual digital human generation, video advertising customization, game content editing, interactive narrative generation, and intelligent short video editing, providing content creators, special effects artists, and intelligent media platforms with powerful video semantic editing capabilities. Its high-precision, multi-object controllable editing features will significantly improve production efficiency and user experience, and have broad industry application prospects. Attached Figure Description

[0077] Figure 1 This is a flowchart of a multi-object video editing method based on semantically aware content alignment.

[0078] Figure 2 This is a visual and qualitative comparison of the results of this invention with other methods.

[0079] Figure 3 and Figure 4 This is a qualitative comparison of the results of this invention with other methods.

[0080] Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 10 and Figure 11 This is a visual comparison diagram of the present invention and other methods.

[0081] Figure 12 and Figure 13This is an example of a high-quality video editing result generated by the method of the present invention under conditions of multiple categories of objects and multiple types of text descriptions. Detailed Implementation

[0082] This invention presents a multi-object video editing method based on semantically aware content alignment. By innovatively introducing a semantic adaptive modulation mechanism and a semantic prior modeling strategy, it effectively solves the problems of semantic confusion, object misalignment, and difficulty in maintaining structural stability and cross-frame consistency in existing multi-object video editing methods. While maintaining high visual fidelity, this method significantly improves the accuracy of multi-object semantic control and the temporal continuity of editing results, enabling the generation of edited video content with precise semantic alignment, clear structure, and consistent appearance. The specific implementation process is as follows.

[0083] The invention will be further described below with reference to specific embodiments.

[0084] Example 1:

[0085] like Figure 1 As shown, this method mainly includes two core modules: (a) a semantic adaptive modulation module, which consists of region-aware attention modulation and text-aware attention modulation. By dynamically modulating the self-attention and cross-attention within the diffusion model, it achieves effective decoupling of spatial semantics among multiple objects and accurately corresponds text semantic instructions with target object regions, thereby improving the accuracy of object-level semantic localization; (b) a semantic prior modeling module, which uses a pre-trained visual language model to construct cross-modal semantic consistency supervision. During the diffusion sampling process, it continuously constrains the generated content to converge toward the target text semantics, so that the edited video is consistent with the edited text in terms of overall semantics, object attributes, and behavioral expression.

[0086] Figure 2 This paper demonstrates the visual results and qualitative comparisons of the present invention with other video editing methods in multi-object semantic editing tasks. The present invention can achieve accurate object-level semantic replacement, more consistent structure preservation, and more natural appearance rendering.

[0087] like Figure 3 and Figure 4 As shown, a qualitative comparison of the present invention with other methods under different editing instructions, different object categories, and different scenarios is further presented. The present invention can effectively avoid semantic confusion, attention leakage, and object misalignment in multi-object scenarios, significantly improving the stability and accuracy of editing.

[0088] like Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 10 andFigure 11 The diagram illustrates a comparison of the visual effects of this invention in multi-object editing tasks, including various editing types such as multi-object editing, single-object editing, attribute modification, and local editing. This invention significantly outperforms existing methods in maintaining semantic consistency, structural stability, and temporal coherence.

[0089] like Figure 12 and Figure 13 As shown, this invention can generate high-quality video results under various object types and text description conditions. This demonstrates that the invention possesses excellent stability and generalization ability.

[0090] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0091] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multi-object video editing method based on semantically aware content alignment, characterized in that, Includes the following steps: S1, construct the video latent space representation and perform spatiotemporal feature inversion, perform latent space encoding and DDIM inversion on the video to be edited to obtain the initial noise latent variables containing video motion and structural information; S2, constructing a spatial decoupling of self-attention based on region-aware attention modulation, enabling the model to effectively separate features of multiple objects by enhancing feature focusing within regions and suppressing interference between regions; S3, constructs a text-aware attention modulation-based cross-attention instance localization, enabling the model to accurately align text semantics with instance regions by enhancing the focus on semantically relevant regions and suppressing interference from irrelevant regions; S4 introduces semantic prior modeling and constructs cross-modal semantic consistency supervision through visual language models to constrain semantic alignment during the diffusion sampling process; S5 performs diffusion sampling and generates the final edited video, ensuring that the output video is aligned with the editing text prompts in terms of appearance, structure, semantics, and temporal consistency.

2. Step S1 specifically includes the following steps: S11, the input video frame is mapped to the latent space representation through the latent space encoder. ; S12, using DDIM inversion to... Projected onto the corresponding noise latent variable This allows for controlled generation of subsequent editing processes while maintaining consistency in the overall structure and motion of the video.

3. Step S2 specifically includes the following steps: S21, during the denoising process at step t, calculate the self-attention similarity matrix: (1) in and These are respectively the visual feature query and the key; S22, a region-aware modulation template is introduced to distinguish semantics within and between regions, and its expression is: (2) (3) in These represent the spatial indexes for the query and the key, respectively. and The deviation of the attention value from its maximum and minimum values ​​are measured respectively to achieve intra-regional consistency enhancement and inter-regional interference suppression. S23, construct the Query-Key conditional mapping to limit attention to the region of the current instance, and introduce a fuzzy membership function to make the region boundary continuous, so that the attention can smoothly transition at the edge of the instance. Its form is as follows: (4) (5) S24, based on region affiliation, performs region convergence and region suppression, and obtains the final region adaptive attention: (6) (7) in This is a normalization operation.

4. Step S3 specifically includes the following steps: S31, in the denoising process at step t, calculate the cross-attention similarity matrix: (8) in and These are respectively the visual feature query and the text embedding key; S32, construct a text-aware modulation template to distinguish between text-related and text-irrelevant attention responses, expressed as follows: (9) (10) in and The deviation between the attention value and its upper and lower bounds is measured separately to enhance the visual region aligned with the target text and suppress irrelevant text semantics. S33, Design a Query-Key cross-modal alignment map and use a fuzzy membership function to make it continuous to ensure the smoothness of the cross-modal mapping: its expression can be represented as: (11) (12) in Representation and text tags Related visual areas; S34, through combination , and This is used to modify the cross-attention similarity graph, achieve text-adaptive attention modulation, and obtain the final text-aware attention: (13) (14) in This is a normalization operation.

5. Step S4 specifically includes the following steps: S41, for a given text prompt y, the expected generated video content x simultaneously satisfies the prior distribution p(x) of the pre-trained diffusion model and is semantically consistent with y. Therefore, the editing process is modeled as a posterior inference problem: (15) The posterior maximization can be expressed as: (16) S42, the editable text prompt y is input into the text encoder of the pre-trained visual language model to obtain the text semantic embedding. Simultaneously, the visual content corresponding to the currently generated video frame or intermediate latent variables is input into the same visual language model to obtain its semantic description embedding. This can be represented as: (17) (18) S43, Based on the text semantic embedding and the generated semantic embedding, a semantic likelihood term is constructed to measure the degree of semantic matching between the generated content and the text target. The cosine similarity between the two is used as a consistency measure. Then, the semantic likelihood term is constructed in exponential form, and its expression is as follows: (19) S44, Design a semantic consistency loss function based on the semantic likelihood term, and use it as a semantic supervision signal for the diffusion sampling process. The smaller the loss, the more the generated content conforms to the text requirements. During the diffusion inverse sampling process, update the latent variables according to the gradient of the loss to gradually strengthen semantic alignment, as follows: (20) in The loss function is used to progressively update the latent space noise.

6. Step S5 specifically includes the following steps: S51, integrate the sampled video frames that have undergone the above semantic modulation and semantic prior optimization, and combine them according to the time order of the original video to generate the final edited video; S52, output the final edited video, so that it achieves high-quality performance in terms of semantic consistency, object structure stability, appearance fidelity and temporal continuity, realizing high-precision multi-object semantic editing that can be applied to any input video without additional training.

Images