Device and method for video editing based on noise dilution

Abstract

A noise dilution-based video editing method by at least one processor comprises: separating initial latent noise of an input video into visual noise and Gaussian noise using an adaptive spectrum filter designed to extract video compositions according to the frequency distribution of the input video; determining an editing area to be edited in the input video using the input video and input text; generating visual noise diluted from the visual noise of the editing area through a Gaussian noise mixture with the visual noise; and generating diluted latent noise by recombining the Gaussian noise separated from the initial latent noise and the diluted visual noise.

Description

Technology Field

[0001] The present disclosure relates to video editing. Background Technology

[0002] Diffusion model-based video editing technology can perform editing with high fidelity and precise text alignment on the input video. However, because it performs editing while maintaining the structural layouts of the input video, it is limited to rigid editing such as style transfer and object overlay.

[0003] Therefore, in the case of non-rigid editing, such as changing object movement, it fails to properly generate video that meets user requirements or produces unnatural motion. As such, conventional video editing techniques have limitations in edits that require structural changes, such as changing a walking object into a running object. The problem to be solved

[0004] The present disclosure relates to a noise dilution-based video editing device and method.

[0005] The present disclosure relates to an apparatus and method for generating a video aligned to a target prompt without constraints on visual structures by mitigating visual structures contained in the initial latent noise of an input video through noise dilution. means of solving the problem

[0006] A noise dilution-based video editing method by at least one processor according to some embodiments comprises: separating initial latent noise of an input video into visual noise and Gaussian noise using an adaptive spectrum filter designed to extract video compositions according to the frequency distribution of the input video; determining an editing area to be edited in the input video using the input video and input text; generating visual noise diluted from the visual noise of the editing area through a Gaussian noise mixture with the visual noise; and generating diluted latent noise by recombining the Gaussian noise separated from the initial latent noise and the diluted visual noise.

[0007] The above editing area can be determined by an editing guide mask obtained through cross-attention between the input text and the input video.

[0008] The step of determining the above editing area may generate an editing guide mask for each editing reference word extracted from the above input text, and combine the editing guide masks of all editing reference words to generate an editing guide mask that guides the above editing area.

[0009] The step of determining the above editing area can be combined by adjusting the influence of the mask for rigid editing and the mask for non-rigid editing among the editing guide masks of the above-mentioned entire editing reference words.

[0010] The step of generating the diluted visual noise can generate the diluted visual noise through the Gaussian noise mixing that mixes the Gaussian noise added to the editing area and the visual noise added to the non-editing area.

[0011] The above method may further include the step of generating an edited video by performing noise removal on the diluted potential noise according to the input text.

[0012] The above method may further include the step of providing the diluted potential noise to a backward process of diffusion model-based video editing.

[0013] A noise dilution-based video editing method by at least one processor according to some embodiments comprises: acquiring an original video and a target prompt text; generating initial latent noise by injecting Gaussian noise into the original video; separating the initial latent noise into visual noise and Gaussian noise using an adaptive spectrum filter for the original video; determining an editing area to be edited in the original video using the original video and the target prompt text; generating diluted visual noise by diluting the visual noise in the editing area; generating diluted latent noise by recombining the Gaussian noise separated from the initial latent noise and the diluted visual noise; and generating an edited video by performing noise removal on the diluted latent noise according to the target prompt text.

[0014] The above adaptive spectrum filter can be designed to extract video components according to the frequency distribution of the original video.

[0015] The above editing area can be determined by an editing guide mask obtained through cross-attention between the target prompt text and the original video.

[0016] The step of determining the above editing area may generate an editing guide mask for each editing reference word extracted from the above target prompt text, and combine the editing guide masks of all editing reference words to generate an editing guide mask that guides the above editing area.

[0017] The step of determining the above editing area can be combined by adjusting the influence of the mask for rigid editing and the mask for non-rigid editing among the editing guide masks of the above-mentioned entire editing reference words.

[0018] The step of generating the diluted visual noise can generate the diluted visual noise by diluting the visual noise of the editing area through Gaussian noise mixing and maintaining the visual noise of the non-editing area.

[0019] An apparatus according to some embodiments comprises at least one memory and at least one processor that executes instructions stored in the at least one memory, wherein the processor separates the initial latent noise of the input video into visual noise and Gaussian noise using an adaptive spectrum filter designed to extract video compositions according to the frequency distribution of the input video, determines an editing area to be edited in the input video using the input video and input text, generates visual noise with the visual noise of the editing area diluted through a Gaussian noise mixture with the visual noise, and generates diluted latent noise by recombining the Gaussian noise separated from the initial latent noise and the diluted visual noise.

[0020] A computer program stored on a computer-readable recording medium according to some embodiments comprises instructions that cause at least one processor to execute the steps of: separating initial latent noise of an input video into visual noise and Gaussian noise using an adaptive spectrum filter designed to extract video compositions according to the frequency distribution of the input video; determining an editing area to be edited in the input video using the input video and input text; generating diluted visual noise from the visual noise of the editing area through a Gaussian noise mixture with the visual noise; and recombining the Gaussian noise separated from the initial latent noise and the diluted visual noise to generate diluted latent noise. Effects of the invention

[0021] According to the present disclosure, a video aligned to a target prompt can be generated without constraints on the visual structure through noise dilution.

[0022] According to the present disclosure, non-rigid edits, such as changes in motion, can be performed precisely and dynamically, and rigid edits that maintain visual structure, such as style transformations, can also be performed much more naturally and creatively.

[0023] According to the present disclosure, a user can create an edited video as desired.

[0024] The performance of existing video editing systems can be improved by applying the Dilutional Noise Initialization (DNI) framework of the present disclosure to various video editing systems. Brief explanation of the drawing

[0025] Figure 1 is a diagram illustrating diffusion model-based video editing. Figure 2 is a diagram exemplifying the editing result by conventional video editing technology. FIG. 3 is a drawing illustrating a noise dilution-based video editing device according to one embodiment. FIG. 4 is a diagram illustrating the spectral characteristics of initial potential noise according to one embodiment. FIG. 5 is a diagram illustrating noise separation according to one embodiment. FIG. 6 is a diagram illustrating noise dilution according to one embodiment. Figure 7 is a non-rigid body editing result with a diluted noise initialization (DNI) framework applied according to one embodiment. Figure 8 is a rigid body editing result with a diluted noise initialization (DNI) framework applied according to one embodiment. FIGS. 9 and FIGS. 10 are examples of text alignment sensitivity analysis according to an editing guide mask according to one embodiment. FIG. 11 is a flowchart of a noise dilution method according to one embodiment. FIG. 12 is a flowchart of a noise dilution-based video editing method according to one embodiment. FIG. 13 is a hardware configuration diagram of a computing device in one embodiment. Specific details for implementing the invention

[0026] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0027] In the description, drawing symbols and names are provided for convenience of explanation and are not limited to drawing symbols or names.

[0028] In the description, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "…part," "…unit," and "module" as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0029] The device of the present disclosure is a computing device configured and connected such that at least one processor can perform the operation of the present disclosure by executing instructions. A computer program includes instructions described to enable a processor to perform the operation of the present disclosure and may be stored on a non-transitory computer-readable storage medium. The computer program may be downloaded over a network or sold as a product.

[0030] Figure 1 is a diagram illustrating diffusion model-based video editing, and Figure 2 is a diagram exemplarily showing the editing result by conventional video editing technology.

[0031] Referring to FIG. 1, video editing aims to modify specific attributes of an input video (original video) according to user requirements. Recently, a technique that allows editing of the input video with high fidelity and precise text alignment using a diffusion model instead of a generative model has gained popularity. A diffusion model-based video editing method (10) consists of a forward process that adds noise and a backward process that removes noise to produce a desired output.

[0032] A diffusion model-based video editing method (10) can prepare latent noise (z) by adding Gaussian noise to an input video through a noise initialization process (11), and generate an edited video by performing denoising on the latent noise according to the input text through an editing process (12).

[0033] As such, the diffusion model-based video editing method (10) allows the user to edit the video into the scene they want, but it is limited to rigid editing such as style transfer and object overlay. Therefore, in the case of non-rigid editing such as changing object movement, it fails to properly generate a video suitable for user requirements or generates unnatural motion.

[0034] Referring to Figure 2, when the target prompt "Astronaut is jumping" is entered, the target jumping motion is not generated (unchanged motion), or an unnatural motion is generated by mixing the original video's walking motion with the target jumping motion.

[0035] Referring again to Fig. 1, it can be seen that the quality degradation of non-rigid edits, such as changes in object movement, is caused by latent noise. That is, latent noise is created by gradually adding Gaussian noise to the input video, but the visual structure / video layout of the input video still remains in the latent noise that should be assumed to be Gaussian noise. The visual structure remaining in the latent noise can be identified through frequency filtering. While the visual structure remaining in the latent noise facilitates rigid edits, it conversely acts as an obstacle that restricts non-rigid edits.

[0036] To address these problems, the present disclosure describes a method for generating a video edited according to input text without constraints on the input video layout by mitigating visual structures contained in latent noise through noise dilution.

[0037] FIG. 3 is a drawing illustrating a noise dilution-based video editing device according to one embodiment.

[0038] Referring to FIG. 3, a noise dilution-based video editing device (simply referred to as the device) (100) may be implemented to inject noise into an input video and remove the noise according to an input text to generate an edited video. In this case, the device (100) is implemented to perform video editing using dilutional latent noise (z*) obtained through noise dilution, instead of performing video editing using the initial latent noise (z) as is. The device (100) may be a computing device comprising a memory that stores instructions that cause the present disclosure to be performed, and at least one processor implemented to execute the instructions to perform the method of the present disclosure. A computer program includes instructions that cause at least one processor to perform the present disclosure.

[0039] In the present disclosure, noise dilution refers to generating diluted latent noise (z*) that is closer to Gaussian noise than to the initial latent noise (z). There may be various noise dilution methods; however, if Gaussian noise is additionally injected into the entire video area, the visual structure of the entire video is diluted, and consequently, the quality of the edited video generated by removing latent noise may be degraded. Therefore, the present disclosure identifies the area to be edited, particularly the non-rigid editing area, and performs noise dilution on the editing area. Since diluted latent noise (z*) with a relaxed visual structure of the input video is used, editing corresponding to the input text can be performed dynamically without being constrained by the structure of the input video. A framework for generating diluted latent noise (z*) from the initial latent noise (z) may be called a Dilutional Noise Initialization (DNI) framework.

[0040] The device (100) may include a noise initialization module (110) for injecting noise into an input video, a noise disentanglement module (120), a noise dilution module (130), and an editing module (140). Each module is a functional module distinguished to explain the operation of the device (100), and does not necessarily need to be physically or logically separated. The noise initialization module (110) and the editing module (140) may correspond to the forward process and backward process of a diffusion model, and may learn to generate an edited video through noise addition and noise removal according to the training method of the diffusion model. The noise disentanglement module (120) and the noise dilution module (130) are dilutive noise initialization (DNI) frameworks and can be applied to an existing video editing system implemented as in FIG. 1.

[0041] The noise initialization module (110) is implemented to generate initial latent noise (z) of the input video by adding Gaussian noise to the input video. The initial latent noise (z) can be obtained through an inverse noise removal method such as DDIM (denoising diffusion implicit model) inversion. A frame converted into initial latent noise can be called an initial latent noise frame.

[0042] The noise removal module (120) removes the initial latent noise (z) of the input video as visual noise (visual noise, z) related to the visual structure. v ) and Gaussian noise (Gaussian noise, z gIt is implemented to be separated into ). The initial latent noise frame can be separated into a visual noise frame containing visual noise and a Gaussian noise frame containing Gaussian noise. The initial latent noise (z) created by gradually injecting Gaussian noise into the input video is close to Gaussian noise, but since input video components remain, it is necessary to separate the visual noise to be diluted from the initial latent noise (z). The separated visual noise (z v ) is diluted to be close to Gaussian noise in the noise dilution module (130), and then the separated Gaussian noise (z g It is recombined with ).

[0043] The noise removal module (120) can separate visual noise and Gaussian noise contained in the initial latent noise (z) by using an adaptive spectral filter (ASF) for the input video. The adaptive spectral filter can be designed to extract input video components by capturing the frequency range of the input video according to the frequency distribution of the input video.

[0044] The noise dilution module (130) is visual noise (z v A noise dilution procedure is performed for the input video. At this time, by diluting the visual noise in the editing area of ​​the input video, the editing module (140) can dynamically modify the editing area, and at the same time, the visual compositions of the non-editing area can be maintained to ensure video quality. To this end, the noise dilution module (130) can determine the editing area in the input video based on the input text and dilute the visual noise in the editing area by adding Gaussian noise (ε) to the editing area of ​​the visual noise frame.

[0045] The edit region is determined by the semantic relationship between the input video and the input text, and can be determined through an attention map acquired based on cross-attention between the input text and the input video. The cross-attention model can be designed using various models, such as UNet. The edit region can be determined based on the similarity between input text features and video features, and this similarity can be calculated at the grid level by dividing each frame.

[0046] The editing area is the editing guidance mask m edit It can be guided by. Since there are editing reference words that require structural changes and words that do not among the words in the input text, the final editing guide mask can be generated by appropriately combining the editing guide masks obtained from each word.

[0047] The noise dilution module (130) identifies an editing area (e.g., the astronaut's leg area) where structural changes are required by input text (e.g., Astronaut is jumping) and can dilute the visual noise in the editing area through Gaussian noise mixing. Through visual noise dilution, the input video structure is relaxed, allowing dynamic changes to the movement of the editing area to be performed.

[0048] There can be various methods for noise dilution for the editing area. For example, as in Equation 1, the editing guide mask m edit and the inverted non-edited guide mask (1-m edit Gaussian noise blending for visual noise can be performed using ). Editing guide mask m edit Gaussian noise (ε) is added to the edited area identified by, and visual noise (z) is added to the non-edited area. v) can be maintained as is. That is, if Gaussian noise (ε) is filled through the edit guide mask and visual noise is filled through the non-edit guide mask, adaptively diluted visual noise z d can be printed.

[0049]

[0050] Or, visual noise (z d Visual noise in the editing area can also be diluted by adding Gaussian noise (ε) only to the editing area of ​​).

[0051] Gaussian noise (z) separated from the noise decompression module (120) g ) and adaptively diluted visual noise (z) in the noise dilution module (130) d ) are combined to synthesize diluted latent noise (z*). That is, Gaussian noise (z) separated from the initial latent noise. g ) and diluted visual noise (z d ) is recombined and utilized in video editing. Dilution potential noise (z*) serves to preserve the visual structure of the non-editing area while reducing the rigidity of the visual structure of the editing area.

[0052] The editing module (140) can generate an edited video by removing noise from diluted latent noise according to the input text. The editing module (140) can be implemented based on a diffusion model, and, as in Equation 2, receives diluted latent noise (z*) in which the visual noise of the editing area is diluted instead of the initial latent noise (z), and removes noise according to the input text T to generate an edited video V edit It can output.

[0053]

[0054] Recent video editing technologies are classified into tuning-based models and tuning-free models, and the Diluted Noise Initialization (DNI) framework of the present disclosure can be applied regardless of the model. Since tuning-based models train a diffusion model based on input video, overfitting may occur. Therefore, if the training video is overly adhered to when inferring edits, unnatural edits may result. In such tuning-based models, the use of diluted latent noise can reduce excessive reliance on the initial visual structure and adapt better to target prompts, thereby improving naturalness, fidelity, and consistency. Tuning-free models offer excellent consistency but are less responsive to target prompts and are suitable only for rigid body editing because they rely on the input video structure of the initial latent noise. In such tuning-free models, the use of diluted latent noise can adaptively reduce reliance on the input video structure and improve editing efficiency.

[0055] FIG. 4 is a diagram illustrating the spectral characteristics of initial potential noise according to one embodiment, and FIG. 5 is a diagram illustrating noise separation according to one embodiment.

[0056] Referring to FIG. 4, if we examine the spectral characteristics of the initial latent noise (z), the low-frequency band of the initial latent noise (z) exhibits a distribution similar to that of the input video (z0) in both spatial and temporal frequencies. Therefore, the input video composition can be obtained by applying a low-pass filter. However, the low-pass filter loses high-frequency components of the input video and causes it to correspond to different frequencies for each input video. Accordingly, the present disclosure is characterized by designing an adaptive spectral filter (ASF) that can adaptively respond to frequency changes of the input video to separate the input video compositions. The adaptive spectral filter can be constructed based on the frequency spectrum of the input video (z0) to appropriately capture the frequency range of the input video.

[0057] The noise removal module (120) can generate an adaptive spectrum filter according to the input video. The noise removal module (120) can generate an adaptive spectrum filter by obtaining the frequency distribution of the input video and normalizing the frequency distribution of the input video to a value between 0 and 1. The frequency distribution of the input video can be obtained through a Discrete Fourier Transform (DTF). An adaptive spectrum filter F that varies for each input video s It can be defined as in mathematical formula 3.

[0058]

[0059] In Equation 3, z0 is the input video, and W, H, L, and C are the width, height, length, and channels of the input video. DFT 3D is the 3-dimensional Discrete Fourier Transform, and Norm min-max is a min-max normalization function for scaling between 0 and 1.

[0060] Adaptive spectrum filter F that filters frequency components of input video s Using this, the initial latent noise (z) is the visual noise (z) as in Equations 4 and 5. v ) and Gaussian noise (z g It can be separated into ). In Equations 4 and 5, IDFT 3D is the inverse DFT 3D And ⊙ is element-wise multiplication.

[0061]

[0062]

[0063] Referring to FIG. 5, the noise decompression module (120) removes the visual noise (z) of each frame in the frequency domain of the initial latent noise (z). v ) and Gaussian noise(z g ) can be separated.

[0064] The noise removal module (120) applies a 3D Discrete Fourier Transform (DFT) to the initial latent noise (z) of each frame 3D DFT of noise frequency transformed into the frequency domain by applying 3D (z) is obtained (S110).

[0065] The noise removal module (120) is a noise frequency DFT 3D Adaptive spectrum filter F that filters frequency components of the input video at (z) s Visual noise in the frequency domain is separated by multiplying (S120). The noise decomposition module (120) applies a 3D inverse discrete Fourier transform IDFT to the visual noise in the frequency domain. 3D By applying visual noise (z v ) can be obtained (S121). Adaptive spectrum filter F s It is generated to filter input video components based on the frequency distribution of the input video.

[0066] The noise removal module (120) is a noise frequency DFT3D (z) (1-F s The Gaussian noise in the frequency domain is separated by multiplying by ) (S130). The noise decomposition module (120) separates the Gaussian noise in the frequency domain using IDFT 3D Applying Gaussian noise (z g You can obtain ).

[0067] FIG. 6 is a diagram illustrating noise dilution according to one embodiment.

[0068] Referring to FIG. 6, the noise dilution module (130) may include an editing guidance module (131) that determines an editing area in the input video that requires editing, and a noise blending module (132).

[0069] The editing guide module (131) uses input text and input video to determine an editing area in the input video that requires editing based on the input text, and an editing guide mask m that guides the editing area. edit It can generate. For example, if the input text is 'Astronaut is jumping', the editing guide module (131) can select the editing reference words, 'Astronaut', 'jump' from the input text and generate an editing guide mask for the editing reference words.

[0070] In the case of non-rigid editing based on input text, motion / pose correction is required due to the predicate 'jumping,' but general object segmentation models have limitations in identifying non-rigid editing areas due to noun-based detection. Therefore, the editing guidance module (131) can determine the editing area using an attention map obtained through cross-attention between the input text and the input video. The editing guidance module (131) can select an editing reference word, for example, 'jumping,' from the input text and obtain the word features of the editing reference word. The editing guidance module (131) can obtain video features consisting of the video length (L), the number of grids (P) into which the video is segmented, and the dimension (D). The editing guidance module (131) can obtain an attention map containing similarity information at the grid level by multiplying the video features and the word features (cosine product). That is, an area in the video similar to the editing reference word can be represented as an attention map.

[0071] Meanwhile, among the words in the input text, there are edit reference words that require a structural change and words that do not. For example, if the edit reference word is a noun or an adjective, the attention map m provided from the down block of UNet rgd This can be used to provide clear boundaries for the edit object (astronaut) for rigid modification. If the edit reference word is a predicate, the blurry attention map m provided from the intermediate block of UNet. non-rgd This can be used to provide a non-rigid modification region. m rgd It can be called a rigid body editing guide mask that does not require changes in visual structure, and m non-rgd It can be called a non-rigid editing guide mask that requires a change in visual structure.

[0072] The editing guidance module (131) adjusts the size of the attention maps to fit the latent space (W × H) and combines the editing guidance masks obtained by each word as in Equation 6 to obtain a final editing guidance mask m edit It can generate. The editing guide module (131) determines the importance of rigid editing or non-rigid editing in video editing based on the input text, and can adjust the influence of each editing guide mask by α and β determined based on this.

[0073]

[0074] In Equation 6, α and β are hyperparameters between 0 and 1 that control the influence of the editing guide mask, and the closer to 1, the greater the Gaussian noise and the more the influence of the input video can be suppressed through noise mixing. α enables effective modification within the visual structure, while β supports dynamic editing beyond the visual structure.

[0075] The noise mixing module (132) is an editing guide mask m edit Visual noise in the editing area can be diluted by adding Gaussian noise (ε) to the editing area specified by. Editing guide mask m edit Gaussian noise (ε) is added to the edited areas identified by, and visual noise is added to the non-edited areas, resulting in diluted visual noise z d can be printed.

[0076] Diluted visual noise (z d ) is Gaussian noise (z separated from initial latent noise (z) g Diluted latent noise (z*) is synthesized by adding to ). Instead of the initial latent noise (z), diluted latent noise (z*), which is diluted with visual noise of the non-rigid editing area, is input into the editing module (140).

[0077] FIG. 7 is a non-rigid body editing result with a diluted noise initialization (DNI) framework applied according to one embodiment, and FIG. 8 is a rigid body editing result with a diluted noise initialization (DNI) framework applied according to one embodiment.

[0078] Referring to FIG. 7, we examine the results of applying a dilutive noise initialization (DNI) framework, which includes a noise decompression module (120) and a noise dilution module (130), to recent editing systems such as TAV (Tune-A-Video) and VP2P (Video-P2P).

[0079] When the target prompt for the input video is 'A man gives a thumps-up on the snowboard', it can be seen that motion editing, such as a thumbs-up, is selectively performed in the edited video with the Dilutive Noise Initialization (DNI) framework applied to the TAV (TAV+DNI) compared to the edited video of the TAV, depending on the visibility of the snowboarder's hand. The Dilutive Noise Initialization (DNI) framework can improve the editing performance of the TAV by recognizing the editing area (snowboarder's hand) based on the editing reference words 'gives' and 'thumbs-up', and providing diluted latent noise, which dilutes the visual noise of the editing area, to the editing process of the TAV.

[0080] When the target prompt for the input video is 'A bird spreads wings', it can be seen that motion editing, such as the movement of a bird spreading its wings, is faithfully performed in the edited video with the Dilutive Noise Initialization (DNI) framework applied to VP2P (VP2P+DNI) compared to the edited video of VP2P (where the tree in the original video and the modified wings are improperly fused). The Dilutive Noise Initialization (DNI) framework can improve the editing performance of VP2P by recognizing the bird's wing area, which is the editing area, based on the editing reference words 'spreads' and 'wings', and providing diluted latent noise, which dilutes the visual noise of the editing area, to the editing process of VP2P.

[0081] As such, it can be seen that applying the Dilutive Noise Initialization (DNI) framework to existing editing systems can improve non-rigid editing performance for object movement.

[0082] Referring to FIG. 8, we examine the results of applying a dilutive noise initialization (DNI) framework, which includes a noise decomposition module (120) and a noise dilution module (130), to recent editing systems such as TF (TokenFlow) and FZ (FateZero).

[0083] For rigid body editing with the input video goal prompt 'Shiny silver robot is walking on the now', it can be observed that compared to the TF edited video, the TF+DNI edited video, which applies the Dilution Noise Initialization (DNI) framework to TF, not only changes the human appearance into a robot but also adds a snow effect to the sky. Visualizing the noise dilution area in the input frame reveals that the modification range exceeds the initially set boundary. This indicates that the editing based on dilution latent noise connects smoothly with the surrounding area, allowing the editing effect to naturally extend outward and enhancing the overall effect of the edit. Here, the editing reference words are 'shiny', 'silver', 'robot', and 'snow'.

[0084] For rigid body editing where the target prompt for the input video is 'A boat is sailing beside a land full of trees', it can be confirmed that the editing is performed faithfully in the video with the Dilutive Noise Initialization (DNI) framework applied to FZ (FZ+DNI) compared to the video with FZ. Here, the editing reference words are 'land' and 'trees'.

[0085] FIGS. 9 and FIGS. 10 are examples of text alignment sensitivity analysis according to an editing guide mask according to one embodiment.

[0086] Referring to Figures 9 and 10, the editing guide mask m edit As in mathematical equation 6, rigid body editing guide mask m rgd and non-rigid body editing guide mask m non-rgd It is generated by controlling the influence of . α enables effective modification within the visual structure, and β supports dynamic editing beyond the visual structure.

[0087] Referring to Fig. 9, we examine rigid body editing for the input text 'Astronaut walks through red sunset'. 'Red' and 'sunset' from the input text are selected as editing reference words, and an editing guide mask can be generated. Fixing β=0, the rigid body editing guide mask m for the editing reference word 'sunset' rgd Looking at this, as α increases, the influence of the rigid body editing guide mask increases, and as a result, the editing area expands, and at the same time, more effects can be added to the video. In this way, editing suitability can be improved by an appropriate change in α (e.g., ~0.7).

[0088] Referring to Fig. 10, we examine non-rigid editing for 'Astronaut gives a thumbs-up under the moon'. 'gives', 'thumbs-up', and 'Astronaut' from the input text are selected as edit reference words to generate an edit guide mask. Non-rigid edit guide mask m for the edit reference words 'gives' and 'thumbs-up'. non-rgd Non-rigid editing can be enhanced through this. With the influence of the rigid editing guide mask fixed at α=0.2, as β increases, the influence of the non-rigid editing guide mask increases, and consequently, the non-rigid editing area expands. It can be observed that 'thumbs-up', which was not synthesized at β=0.3, is synthesized at β=0.9.

[0089] In this way, as the noise dilution area is expanded by extending the non-rigid editing area, the visual structure of the input video is relaxed, making it possible to freely synthesize the action corresponding to the input text (e.g., giving a thumbs up) from the object's original action (e.g., walking).

[0090] In this way, editing guide mask m edit is rigid body editing guide mask mrgd and non-rigid body editing guide mask m non-rgd It is generated by appropriately combining them, and the quality of the edited video can be determined by hyperparameters α and β.

[0091] The editing guide module (131) determines the importance of rigid editing or non-rigid editing based on the input text, and can adjust the influence of each editing guide mask by α and β determined based on this. For example, since a stable editing effect is guaranteed in the case of rigid editing at 0.3 < α < 0.7 and in the case of non-rigid editing at β > 0.6, α and β can be determined according to the editing attributes required in the input text.

[0092] FIG. 11 is a flowchart of a noise dilution method according to one embodiment.

[0093] Referring to FIG. 11, the device (100) generates an adaptive spectrum filter that extracts video components according to the frequency distribution of the input video (S210). The device (100) can generate an adaptive spectrum filter by obtaining the frequency distribution of the input video through a Discrete Fourier Transform (DTF) and normalizing the frequency distribution of the input video to a value between 0 and 1.

[0094] The device (100) separates the initial latent noise of the input video into visual noise and Gaussian noise using an adaptive spectrum filter (S220). The initial latent noise can be generated by injecting Gaussian noise into the input video. The initial latent noise can be obtained in the forward process of diffusion model-based video editing.

[0095] The device (100) determines an editing area to be edited in the input video using the input video and input text (S230). The editing area can be determined using the semantic relationship or similarity between the input video and the input text, for example, by utilizing an attention map obtainable from a cross-attention model. The editing area can be guided by utilizing an attention map between the input text and the input video as an editing guide mask. An editing guide mask can be generated for each editing reference word extracted from the input text, and an editing guide mask that guides the editing area can be generated by combining the editing guide masks of all editing reference words. For example, an attention map provided from the down block of UNet can be used as a rigid editing guide mask for noun or adjective words, and a blurry attention map provided from the middle block of UNet can be used as a non-rigid editing guide mask for predicate words. The device (100) can combine the influence of a rigid editing guide mask and a non-rigid editing guide mask according to the editing attribute (rigid editing or non-rigid editing) determined by the input text.

[0096] The device (100) generates visual noise with diluted visual noise in the editing area through Gaussian noise mixing with separated visual noise (S240). Diluted visual noise can be generated by mixing Gaussian noise added to the editing area through an editing area mask and visual noise added to the non-editing area through a non-editing area mask. This allows the editing area to be dynamically modified while simultaneously ensuring video quality based on the visual compositions of the non-editing area.

[0097] The device (100) generates diluted latent noise by recombining Gaussian noise separated from the initial latent noise and diluted visual noise (S250). The diluted latent noise can be provided as a backward process for diffusion model-based video editing. An edited video corresponding to the input text can be generated through noise removal on the diluted latent noise.

[0098] FIG. 12 is a flowchart of a noise dilution-based video editing method according to one embodiment.

[0099] Referring to FIG. 12, the device (100) acquires the original video and the target prompt text (S310).

[0100] The device (100) generates initial latent noise by injecting Gaussian noise into the original video (S320).

[0101] The device (100) uses an adaptive spectrum filter on the original video to separate the initial latent noise into visual noise and Gaussian noise (S330).

[0102] The device (100) determines an editing area to be edited in the original video using the original video and the target prompt text (S340). The editing area can be determined through an attention map obtained based on cross-attention between the target prompt text and the original video. The editing area can be guided by an editing guide mask. The editing guide mask can be generated by summing editing guide masks obtained based on editing reference words, and the device (100) can sum the editing guide masks by adjusting the influence of the rigid editing guide mask and the non-rigid editing guide mask according to the editing attribute (rigid editing or non-rigid editing).

[0103] The device (100) dilutes visual noise in the editing area to generate adaptively diluted visual noise according to the editing area (S350). Diluted visual noise can be generated by mixing Gaussian noise added to the editing area through an editing area mask and visual noise added to the non-editing area through a non-editing area mask. This allows the editing area to be dynamically modified while simultaneously ensuring video quality based on the visual compositions of the non-editing area.

[0104] The device (100) generates diluted latent noise by recombining Gaussian noise separated from the initial latent noise and diluted visual noise (S360).

[0105] The device (100) generates an edited video by performing noise removal on diluted potential noise according to the target prompt text (S370).

[0106] FIG. 13 is a hardware configuration diagram of a computing device in one embodiment.

[0107] Referring to FIG. 13, the video editing device (100) may be a computing device (200) comprising a memory that stores instructions for performing the present disclosure and at least one processor implemented to perform the method of the present disclosure by executing the instructions.

[0108] A computing device (200) may include one or more processors (210), a memory (230) for loading a computer program executed by the processor (210), a storage device (250) for storing the computer program and various data, a communication interface (270), and a bus (290) connecting them. In addition, the computing device (200) may further include various components. A computer program may include instructions that cause the processor (210) to perform a method / operation according to various embodiments of the present disclosure when loaded into the memory (230). That is, the processor (210) may perform a method / operation according to various embodiments of the present disclosure by executing the instructions. An instruction refers to a series of computer-readable instructions grouped by function, which are components of a computer program and are executed by the processor.

[0109] The processor (210) controls the overall operation of each component of the computing device (200). The processor (210) may be configured to include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any form of processor well known in the art of the present disclosure. Additionally, the processor (210) may perform operations for at least one application or program for performing a method / operation according to various embodiments of the present disclosure.

[0110] The memory (230) stores various data, commands and / or information. The memory (230) may load one or more programs from the storage device (250) to perform a method / operation according to various embodiments of the present disclosure. The memory (230) may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0111] The storage device (250) may store a program non-temporarily. The storage device (250) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which this disclosure belongs.

[0112] The communication interface (270) supports wired and wireless communication of the computing device (200). To this end, the communication interface (270) may be configured to include a communication module well known in the art of the present disclosure.

[0113] The bus (290) provides communication functions between components of the computing device (200). The bus (290) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0114] Thus, according to the present disclosure, a video aligned to a target prompt can be generated without constraints on the visual structure through noise dilution.

[0115] According to the present disclosure, non-rigid edits, such as changes in motion, can be performed precisely and dynamically, and rigid edits that maintain visual structure, such as style transformations, can also be performed much more naturally and creatively.

[0116] According to the present disclosure, a user can create an edited video as desired.

[0117] The performance of existing video editing systems can be improved by applying the Dilutional Noise Initialization (DNI) framework of the present disclosure to various video editing systems.

[0118] The embodiments of the present disclosure described above are not implemented only through devices and methods, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present disclosure or a recording medium on which such program is recorded.

[0119] Although embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concepts of the present disclosure as defined in the following claims also fall within the scope of the present disclosure.

Claims

Claim 1 A noise dilution-based video editing method by at least one processor, comprising: separating initial latent noise of an input video into visual noise and Gaussian noise using an adaptive spectrum filter designed to extract video compositions according to the frequency distribution of the input video; determining an editing area to be edited in the input video using the input video and input text; generating visual noise in which the visual noise of the editing area is diluted through a Gaussian noise mixture with the visual noise; and generating diluted latent noise by recombining the Gaussian noise separated from the initial latent noise and the diluted visual noise. Claim 2 A method according to claim 1, wherein the editing area is determined by an editing guide mask obtained through cross-attention between the input text and the input video. Claim 3 In paragraph 2, the step of determining the editing area comprises generating an editing guide mask for each editing reference word extracted from the input text, and combining the editing guide masks of all editing reference words to generate an editing guide mask that guides the editing area. Claim 4 In paragraph 3, the step of determining the editing area is a method of combining the influence of a mask for rigid editing and a mask for non-rigid editing among the editing guide masks of the entire editing reference words. Claim 5 In claim 1, the step of generating the diluted visual noise is a method of generating the diluted visual noise through a Gaussian noise mixture that mixes the Gaussian noise added to the editing area and the visual noise added to the non-editing area. Claim 6 A method according to claim 1, further comprising the step of generating an edited video by performing noise removal on the diluted potential noise according to the input text. Claim 7 A method according to claim 1, further comprising the step of providing the diluted potential noise to a backward process of diffusion model-based video editing. Claim 8 A noise dilution-based video editing method by at least one processor, comprising: acquiring an original video and a target prompt text; generating initial latent noise by injecting Gaussian noise into the original video; separating the initial latent noise into visual noise and Gaussian noise using an adaptive spectral filter for the original video; determining an editing area to be edited in the original video using the original video and the target prompt text; generating diluted visual noise by diluting the visual noise in the editing area; generating diluted latent noise by recombining the Gaussian noise separated from the initial latent noise and the diluted visual noise; and generating an edited video by performing noise removal on the diluted latent noise according to the target prompt text. Claim 9 In claim 8, the method wherein the adaptive spectrum filter is designed to extract video compositions according to the frequency distribution of the original video. Claim 10 In claim 8, the editing area is determined by an editing guidance mask obtained through cross-attention between the target prompt text and the original video. Claim 11 In claim 10, the step of determining the editing area comprises generating an editing guide mask for each editing reference word extracted from the target prompt text, and combining the editing guide masks of all editing reference words to generate an editing guide mask that guides the editing area. Claim 12 In claim 11, the step of determining the editing area is a method of combining the influence of a mask for rigid editing and a mask for non-rigid editing among the editing guide masks of the entire editing reference words. Claim 13 In claim 8, the step of generating the diluted visual noise generates the diluted visual noise by diluting the visual noise of the editing area through Gaussian noise mixing and maintaining the visual noise of the non-editing area. Claim 14 An apparatus comprising at least one memory and at least one processor that executes instructions stored in said at least one memory, wherein the processor separates the initial latent noise of said input video into visual noise and Gaussian noise using an adaptive spectrum filter designed to extract video compositions according to the frequency distribution of said input video, determines an editing area to be edited in said input video using said input video and input text, generates visual noise with the visual noise of said editing area diluted through a Gaussian noise mixture with said visual noise, and generates diluted latent noise by recombining the Gaussian noise separated from said initial latent noise and said diluted visual noise. Claim 15 A computer program stored on a computer-readable recording medium, comprising instructions that cause at least one processor to execute the steps of: separating initial latent noise of an input video into visual noise and Gaussian noise using an adaptive spectrum filter designed to extract video compositions according to the frequency distribution of the input video; determining an editing area to be edited in the input video using the input video and input text; generating diluted visual noise from the visual noise of the editing area through a Gaussian noise mixture with the visual noise; and recombining the Gaussian noise separated from the initial latent noise and the diluted visual noise to generate diluted latent noise.

Images