Text-driven zero-sample 6-DOF video editing method and system

Abstract

The present invention discloses a text-driven zero-sample 6-DOF video editing method and system, which belongs to the field of computer vision processing. The method of the present invention includes: processing multi-view source video frame by frame based on a diffusion model, encoding each frame of original image by an encoder; injecting text representation in the inversion process of the UNet network, storing the cross-attention map and self-attention map of each noise-added time step; updating the text-free control vector, outputting the edited image hidden code by the designed UNet editor and decoding it by the decoder, and outputting the target image; using the pixel-level mask as a weight, weighting the target image and the original image to form an edited video; training the four-dimensional representation network again, using the rendering perspective and timestamp sequence as input, and finally outputting the reconstructed 6-DOF video. Through the method of the present invention, users can achieve flexible operation of 6-DOF dynamic scenes by interacting with text.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision processing, and in particular relates to a text-driven zero-sample 6-DOF video editing method and system. Background Art

[0002] Immersive reality technology has significantly changed the way users interact with media content, providing a higher level of immersion in virtual environments. The core of the immersive system is 6-degrees-of-freedom (6-DoF) video technology, which allows users to dynamically adjust their perspective and position in the virtual space. Although this technology has shown considerable practical value in fields such as online education and computer games, most virtual environments and interaction modes are currently preset and fixed, which greatly limits its scope of application. Therefore, there is an urgent need for a method to edit 6-DoF training videos in dynamic scenes, especially to accurately modify target objects in motion.

[0003] In the field of dynamic scenes, the study of dynamic neural radiance fields (NeRFs) has been able to transform multi-view videos into realistic dynamic 3D scenes. Given the excellent performance of diffusion models in image editing tasks, it is possible to transfer text-driven image editing operations to 6-DOF video editing tasks. However, there are three key challenges in applying these powerful techniques to 6-DOF video editing: (1) Contextual semantic consistency: For a single-frame image, popular image editing methods often cause unexpected changes to the background or other non-reference context, resulting in mismatched elements in the image or artifacts in the synthesized new view. (2) Temporal consistency: For a set of frames, it is necessary to maintain the consistency of the edited object's appearance while maintaining its motion information, especially considering that 6-DOF videos are larger in size and have more frames than the commonly used video editing tasks. (3) Cross-view consistency: For a set of frames captured simultaneously from different viewpoints, ensuring the cross-view consistency of the edited objects in a 6-DOF environment is also a challenge because the pre-trained diffusion model does not learn the geometric priors of 3D objects. Although existing methods can solve the problem of cross-view consistency by performing editing during the diffusion process, they are less effective in 6-DOF dynamic scenes.

[0004] The technical background of text-driven zero-shot 6-DOF video editing methods can be summarized as follows: (1) 6-DOF video: 6-DOF video is an emerging technology that enables users to explore new perspectives in videos. Video systems use multi-view camera devices to capture 360-degree panoramas and adopt depth-based reprojection variants to synthesize the views of each video frame. Some methods improve visual quality by optimizing multi-spherical images that change over time, and some methods use multi-view camera devices to shoot 6-DOF videos and synthesize dynamic 3D scenes. (2) Text-driven video editing: Videos can be edited in many ways. In the video stylization editing task, samples are relied on as style guidance, frames are processed individually through image style transfer methods, and temporal consistency is learned in the post-processing stage. In the video local editing task, layer map-based methods implement video editing by flattening texture maps. So far, no existing video editor is capable of performing video editing tasks. (3) 6-DOF video consistency: Video editing tasks need to consider the contextual semantic consistency, temporal consistency, and cross-view consistency of the editing results. Contextual semantic consistency requires that the single-frame image of the edited video maintains the background outside the editing area unchanged, ensuring that the elements in the synthesized new perspective match and are free of artifacts. Temporal consistency requires that the edited video maintains the original motion information and is sufficiently continuous in the temporal dimension. Cross-perspective consistency requires that the scenes from different perspectives are sufficiently consistent in space, and there are no significant differences in the same spatial point seen from different perspectives. In short, the text-driven zero-sample 6-DOF video editing method needs to be implemented in combination with a variety of technical means, all of which aim to improve the editing quality, so that video editing can play a better role in new fields. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and provide a text-driven zero-sample 6-DOF video editing method and system.

[0006] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a text-driven zero-sample 6-DOF video editing method, which comprises the following steps:

[0008] S1: Obtain a pre-trained diffusion model including a text encoder, an image encoder-decoder and a UNet network, and use the diffusion model to process the original images of all perspectives in the multi-perspective source video frame by frame. First, a frame of the original image is passed to the encoder in the image encoder-decoder for encoding to obtain the original image latent code;

[0009] S2: After encoding the acquired text prompt through a text encoder, a text representation is obtained, the original image hidden code and the text representation are input into the UNet network to invert the diffusion process, the original image hidden code is denoised through multiple time steps to obtain a noisy hidden code, and the cross-attention map and self-attention map generated at each time step of the denoising process are saved;

[0010] S3: The text representation and the noisy latent code obtained at each time step during the denoising process are reconstructed and denoised to obtain the updated text-free control vector at each time step;

[0011] S4: modify the calculation method of the cross-attention layer and the self-attention layer in the UNet network, use the modified UNet network as a UNet editor, replace the text prompt with the edited text prompt, encode the edited text prompt through a text encoder to obtain a target text representation, input the noisy hidden code, the target text representation and the updated text-free control vector into the UNet editor, and denoise the noisy hidden code through multiple time steps to obtain an edited image hidden code;

[0012] S5: inputting the edited image latent code outputted at the last time step of the edit denoising process into the decoder in the image encoder-decoder for decoding to obtain a target image;

[0013] S6: Calculate the pixel-level mask based on the cross-attention map generated by the denoising process, and use the pixel-level mask as a weight to weight the target image and the original image to obtain the edited image and form the edited video;

[0014] S7: Input the edited video into the trained 4D representation network for retraining, input the rendering view and timestamp sequence into the retrained 4D representation network, and output the reconstructed 6-DOF video.

[0015] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0016] As a preferred embodiment of the above-mentioned first aspect, in step S3, in the reconstruction and denoising of the tth time step, the initial text-free control vector of the tth time step is obtained and used as the first UNet control condition, and the UNet network processes the noise hidden code generated at the tth time step to obtain the first text-free noise predicted at the tth time step; the text representation is used as the second UNet control condition, and the UNet network processes the noise hidden code generated at the tth time step to obtain the first text-controlled noise predicted at the tth time step, and the first text-controlled noise and the first text-free noise are weightedly summed to obtain the first denoising noise of the tth time step.

[0017] As a preferred embodiment of the above-mentioned first aspect, the reconstructed denoised hidden code of the (t-1)th time step is calculated based on the first denoised noise of the tth time step, the mean square error between the reconstructed denoised hidden code and the noise hidden code of the (t-1)th time step is calculated, and the text-free control vector of the tth time step is optimized based on minimizing the mean square error.

[0018] Furthermore, the reconstruction denoising hidden code at the (t-1)th time step is The functional form is:

[0019]

[0020] In the formula, represents the reconstructed denoising hidden code at the t-th time step; is the guidance coefficient at the tth time step; is the guidance coefficient of the (t-1)th time step; represents the first denoised noise at the t-th time step.

[0021] As a preferred embodiment of the above-mentioned first aspect, in step S4, the calculation method of the cross-attention layer in the UNet editor is as follows: in the i-th cross-attention layer at the t-th time step, the cross-attention map generated by the i-th cross-attention layer at the t-th time step in the noise addition process is used to replace the cross-attention map generated by the cross-attention layer in the editing denoising process, and the target text representation is used as the first value representation. The replaced cross-attention map and the first value representation are softmax calculated and output.

[0022] As a preferred embodiment of the above-mentioned first aspect, in step S4, the calculation method of the self-attention layer in the UNet editor is as follows: in the i-th self-attention layer at the t-th time step, the self-attention map generated by the i-th self-attention layer at the t-th time step in the denoising process and the self-attention map generated by the self-attention layer in the editing denoising process are weighted summed to obtain a modified self-attention map, and the edited image latent code generated at the t-th time step is used as the second value representation, and the modified self-attention map and the second value representation are output after softmax calculation.

[0023] As a preferred embodiment of the above-mentioned first aspect, in step S4, in the t-th time step of editing and denoising, the updated text-free control vector of the t-th time step is obtained and used as the third UNet control condition, and the UNet editor processes the noise hidden code generated at the t-th time step to obtain the second text-free noise predicted at the t-th time step; the target text representation is used as the fourth UNet control condition, and the UNet editor processes the noise hidden code generated at the t-th time step to obtain the second text control noise predicted at the t-th time step, and the second text control noise and the second text-free noise are weightedly summed to obtain the second denoising noise at the t-th time step, and finally, the edited image hidden code of the (t-1)-th time step is calculated based on the second denoising noise at the t-th time step.

[0024] Furthermore, the hidden code of the edited image at the (t-1)th time step is The functional form is:

[0025]

[0026] In the formula, Represents the hidden code of the edited image at the tth time step; represents the second denoised noise at the t-th time step.

[0027] As a preferred embodiment of the above-mentioned first aspect, in step S6, the specific process of calculating the pixel-level mask from the cross-attention map generated by the noise adding process is: adding the cross-attention maps generated by each cross-attention layer in each time step of the noise adding process to obtain a fused attention map, dividing the fused attention map with the maximum value in the fused attention map, and using the division result as the pixel-level mask.

[0028] In a second aspect, the present invention provides a text-driven zero-sample 6-DOF video editing system, comprising:

[0029] The encoding module is used to obtain a pre-trained diffusion model including a text encoder, an image encoder-decoder and a UNet network, and use the diffusion model to process the original images of all perspectives in the multi-perspective source video frame by frame. First, a frame of the original image is passed to the encoder in the image encoder-decoder for encoding to obtain the original image latent code;

[0030] An inversion module is used to encode the acquired text prompt through a text encoder to obtain a text representation, input the original image hidden code and the text representation into the UNet network to invert the diffusion process, add noise to the original image hidden code through multiple time steps to obtain a noisy hidden code, and save the cross-attention map and self-attention map generated at each time step of the noisy process;

[0031] The reconstruction and denoising module is used to reconstruct and denoise the text representation and the noise latent code obtained at each time step during the denoising process to obtain the updated text-free control vector at each time step;

[0032] An editing and denoising module is used to modify the calculation method of the cross-attention layer and the self-attention layer in the UNet network, use the modified UNet network as a UNet editor, replace the text prompt with the edited text prompt, encode the edited text prompt through a text encoder to obtain a target text representation, input the noise hidden code, the target text representation and the updated text-free control vector into the UNet editor, and edit and denoise the noise hidden code through multiple time steps to obtain an edited image hidden code;

[0033] A decoding module, used for inputting the edited image latent code outputted at the last time step of the edit denoising process into the decoder in the image encoder-decoder for decoding to obtain a target image;

[0034] An edited image generation module is used to calculate a pixel-level mask through the cross-attention map generated by the denoising process, and weight the target image and the original image using the pixel-level mask as a weight to obtain an edited image and form an edited video;

[0035] The video reconstruction module is used to input the edited video into the trained four-dimensional representation network for re-training, input the rendering perspective and timestamp sequence into the re-trained four-dimensional representation network, and output the reconstructed 6-DOF video.

[0036] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the text-driven zero-sample 6-DOF video editing method as described in any of the schemes of the first aspect is implemented.

[0037] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0038] The memory is used to store computer programs;

[0039] The processor is used to implement the text-driven zero-sample 6-DOF video editing method as described in any of the solutions of the first aspect when executing the computer program.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] For the text-driven zero-sample 6-DOF video editing task, the present invention provides a groundbreaking solution, namely, a text-driven zero-sample 6-DOF video editing method. The method of the present invention utilizes the zero-sample generalization capability of the pre-trained diffusion model to embed a reference attention-guided denoising process in the inversion and denoising process, thereby enhancing the contextual semantic consistency, temporal consistency, and cross-view consistency of 6-DOF video editing. Based on the method of the present invention, the 6-DOF video editing task no longer requires any additional training data, and the editing of 6-DOF videos and dynamic three-dimensional scenes can be realized under the drive of text, expanding its application scenarios in the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the steps of the method of the present invention;

[0043] Figure 2 A schematic diagram of the process of generating an edited image by the method of the present invention;

[0044] Figure 3 A comparison diagram of the editing results of the present invention and other video editing methods;

[0045] Figure 4 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0047] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0048] like Figure 1 As shown, in a preferred implementation of the present invention, the above text-driven zero-sample 6-DOF video editing method includes the following steps S1 to S7. The specific implementation process is described below.

[0049] S1: Obtain a pre-trained diffusion model including a text encoder, an image encoder-decoder and a UNet network, and use the diffusion model to process the original images of all perspectives in the multi-perspective source video frame by frame. First, a frame of the original image is passed to the encoder in the image encoder-decoder for encoding to obtain the original image latent code.

[0050] It should be noted that in step S1 of the present invention, the diffusion model used has a structure including a text encoder, an image encoder-decoder and a U-type network (i.e., UNet). The model is pre-trained on a large amount of text image data and has super image generation capabilities. In the present embodiment, Stable Diffusion 1.4 is specifically used as the above-mentioned diffusion model, and its implementation method belongs to the prior art and will not be repeated. In addition, in addition to the diffusion model listed in this embodiment, it can also be adjusted according to actual needs, and other suitable diffusion models can be selected, as long as it is coordinated by the text encoder, image encoder-decoder and UNet to complete the above-mentioned 6-DOF video editing task. Therefore, the specific structure of the above-mentioned diffusion model is not limited in the present invention.

[0051] In this embodiment, the multi-view source video is further split into multiple frames, so that the 6-DOF video editing task is converted into a multi-frame, multi-view image editing task. Then, the original image in RGB format is transformed using the above-mentioned pre-trained diffusion model. The encoder in the incoming image encoder-decoder Processing to get the original image hidden code .

[0052] S2: prompt the text given by the user After encoding by the text encoder, the text representation is obtained , the original image hidden code and text representation are input into the UNet network to invert the diffusion process, and the original image hidden code is passed through After adding noise to the time step, we get the noise hidden code , and save the cross-attention map and self-attention map generated at each time step of the noise addition process.

[0053] It should be noted that in step S2 of the present invention, the specific process of inverting the diffusion process by the UNet network can be understood as the forward noise addition process of the diffusion model, and its implementation process belongs to the prior art. Generate noisy latent code at time steps The way is as follows:

[0054]

[0055] in, It is The bootstrap coefficient for each time step; It is The bootstrap coefficient for each time step; It is Noisy hidden code of time steps; Represents the noise predicted by the UNet network.

[0056] It should be noted that in step S2 of the present invention, in order to solve the challenges brought by contextual semantic consistency, temporal consistency and cross-view consistency in dynamic three-dimensional space, the present invention is committed to enhancing the control of text over each frame. Different from previous DDIM-based editing methods, these methods only aim to obtain noise hidden codes, while the present invention not only injects text prompts in the inversion process of the diffusion model, but also stores the noise hidden codes generated at each time step of the denoising process and the intermediate control factors: cross-attention map and self-attention map. The noise hidden code can be used as a reference in the editing process to record the motion information of the object, so as to constrain the edited image hidden code generated in the denoising process from being over-edited, while the stored cross-attention map and self-attention map guide the diffusion model to perform editing. The cross-attention map can be regarded as a soft mask to mark the area corresponding to each word in the hidden code.

[0057] In addition, in step S2 of the present invention, the process of generating the self-attention map and the cross-attention map belongs to the process completed by the UNet network itself, so it will not be described in detail in the following text, and only the main process will be described. For the noise addition process of the diffusion model, the text prompt is passed through the text encoder to obtain a text representation, and then the cross-attention map output by the cross-attention layer of the UNet network is multiplied by the text representation as the input of the next cross-attention layer. After being processed by K cross-attention layers, each time step will generate a cross-attention map of K layers. The generation process of the self-attention map is similar to that of the cross-attention map. The difference is that when generating the self-attention map, there is no need to obtain additional text representation. Instead, the internally generated value representation is multiplied with the self-attention map generated by the self-attention layer as the input of the next self-attention layer. After being processed by K self-attention layers, each time step will generate a corresponding K-layer self-attention map. In step S2 of this embodiment, in order to avoid memory overflow, it is necessary to filter and store the self-attention map and the cross-attention map whose last dimension is less than 4096.

[0058] S3: The text representation and the noisy latent code obtained at each time step during the denoising process are reconstructed and denoised to obtain the updated text-free control vector at each time step.

[0059] It should be noted that in step S3 of the present invention, since the text prompt given by the user cannot accurately describe the original image, the hidden code of the original image obtained after denoising is not the same as the actual denoising feature. A learnable text-free control vector is used to assist the UNet network in understanding the original image according to the text prompt given by the user, and these control vectors are randomly initialized before the denoising process begins to have the same representation scale as the text prompt given by the user.

[0060] In the reconstruction denoising In the time step, get the The initial text-free control vector of the time step is used as the first UNet control condition. The noise hidden code generated by the time step is processed to obtain the The first time step prediction without text noise :

[0061]

[0062] Representing text As the second UNet control condition, the UNet network The noise hidden code generated by the time step is processed to obtain the The first text control noise of the prediction of time step :

[0063]

[0064] Next, the first text-controlled noise and the first text-free noise are weighted and summed to obtain the first The first denoised noise at time step :

[0065]

[0066] In the formula, Represents the preset weight hyperparameters.

[0067] Finally, use the The first denoised noise at time step Calculate the Reconstruction denoising hidden coding for time steps , calculate the Reconstructed denoising hidden coding and noise hidden coding for time steps The mean square error between A text-free control vector for each time step.

[0068] Further, Reconstruction denoising hidden coding for time steps The functional form is:

[0069]

[0070] In the formula, Indicates Reconstruction denoising hidden coding of time steps; Indicates The first denoised noise at time steps.

[0071] S4: Modify the calculation method of the cross attention layer and the self-attention layer in the UNet network, use the modified UNet network as a UNet editor, replace the text prompt with the edited text prompt, encode the edited text prompt through a text encoder to obtain a target text representation, input the noise hidden code, the target text representation and the updated text-free control vector into the UNet editor, and replace the noise hidden code with the edited text prompt. After editing and denoising for time steps, the edited image hidden code is obtained .

[0072] It should be noted that in step S4 of the present invention, the text prompt is replaced with the edited text prompt. This process is a routine operation for those skilled in the art. The following example illustrates the above replacement process. Figure 2 For example, the text prompt may be a man pouring coffee in the kitchen. After replacement, the edited text prompt may be a Van Gogh pouring coffee in the kitchen.

[0073] It should be noted that in step S4 of the present invention, in the editing and denoising process, the classifier-free guidance is used, specifically, the stored noise latent variable is used as a reference guidance, which can control the editing ability within a moderate range. The classifier-free guidance ratio here can be set to a value much greater than 1. In this embodiment, it is set to a value between 5-15.

[0074] It should be noted that in step S4 of the present invention, the present invention modifies the calculation method of the cross attention layer and the self attention layer in the UNet network to obtain the UNet editor In the UNet Editor, the cross-attention layer and self-attention layer are calculated as follows:

[0075] 1) In the Time step In the cross attention layer, the noise addition process is used. Time step The cross-attention map generated by the cross-attention layer replaces the cross-attention map generated by the cross-attention layer in the editing denoising process, and takes the target text representation as the first value representation. The replaced cross-attention map and the first value representation are calculated by softmax and output.

[0076] 2) In the Time step In the self-attention layer, the noise addition process is used. The self-attention map generated by the i-th self-attention layer in the time step is weighted summed with the self-attention map generated by the self-attention layer in the editing denoising process to obtain the modified self-attention map. The edited image hidden code generated by the time step is used as the second value representation, and the modified self-attention map and the second value representation are output after softmax calculation.

[0077] In this embodiment, the time step of the editing denoising stage is changed from to 0. time step, Taking the cross attention layer as an example, the original calculation method in the UNet network is: Hidden code of the edited image generated by time steps As the first query representation, the target text representation For the first key representation, calculate the original cross attention map , and the target text representation is used as the first value representation, and the output is obtained through softmax calculation. Cross-attention map generated by the layer cross-attention layer Instead of the original cross attention map , the target text representation is used as the first value representation, the output is obtained through softmax calculation, and it is used as the input of the next cross attention layer. Taking the self-attention layer as an example, the original calculation method in the UNet network is: Hidden code of the edited image generated by time steps As the second query representation and the second key representation, calculate the original self-attention map , and the edited image hidden code is represented as the second value, and the output is obtained through softmax calculation. In the present invention, the original self-attention map is compared with the self-attention map outputted by the corresponding layer of the noise adding process. Perform weighted summation to obtain the modified self-attention map The above edited image hidden code is used as the second value representation, and the output is obtained through softmax calculation. This attention-level guidance can inject the target text representation into the area related to the changed word in the text prompt.

[0078] It should be noted that in step S4 of the present invention, in the denoising process In the time step, get the The text-free control vector after the update of time steps And use it as the third UNet control condition, and the UNet editor will The noise hidden code generated by the time step is processed to obtain the The second time step prediction without text noise :

[0079]

[0080] Represent the target text As the fourth UNet control condition, the UNet editor The noise hidden code generated by the time step is processed to obtain the The second text control noise of the prediction of time steps :

[0081]

[0082] Next, the second text-controlled noise and the second text-free noise are weighted and summed to obtain the first The second denoising noise of time steps :

[0083]

[0084] Finally, use the The second denoising noise of time steps Calculate the first Hidden encoding of edited images at time steps :

[0085]

[0086] In the formula, Indicates Hidden code of edited images at time steps.

[0087] In the above denoising and denoising process, by storing the multi-scale cross-attention map, self-attention map and noise hidden code generated by each denoising step, and improving it based on the UNet network, a UNet editor was designed, which determines the degree of correlation between different regions in the text representation. This lays an important foundation for the subsequent realization of accurate 6-DOF video editing.

[0088] S5: Input the edited image latent code outputted at the last time step of the edit denoising process into the decoder in the image encoder-decoder for decoding to obtain the target image .

[0089] S6: Calculate the pixel-level mask based on the self-attention map generated by the denoising process, and use the pixel-level mask as a weight to weight the target image and the original image to obtain the edited image and form the edited video.

[0090] It should be noted that in step S6 of the present invention, after multiple denoising steps, if the final edited image hidden code is directly used to reconstruct the image through the decoder, the subtle differences in the feature space may be magnified in the pixel space. The cross-attention maps are stored and, in a weighted summation manner, a pixel-level mask is obtained and set on the target image to obtain the edited image.

[0091] In this embodiment, the specific process of calculating the pixel-level mask from the cross-attention map generated by the noise addition process is as follows: adding the cross-attention maps generated by each cross-attention layer at each time step of the noise addition process to obtain a fused attention map, dividing the fused attention map with the maximum value in the fused attention map, and using the division result as the pixel-level mask :

[0092]

[0093] In the formula, Represents the cross-attention map generated by the i-th cross-attention layer at the j-th time step of the noisy process. Indicates taking the maximum value. Thus, the final edited image can be obtained :

[0094]

[0095] like Figure 2 As shown, after the above series of processes, an edited image is finally generated, and the edited video composed of the edited image can accurately achieve the local editing effect to meet the requirements of contextual semantic consistency.

[0096] S7: Input the edited video into the trained 4D representation network for retraining, input the rendering view and timestamp sequence into the retrained 4D representation network, and output the reconstructed 6-DOF video.

[0097] It should be noted that in the present invention, after the original images of all perspectives in the multi-perspective source video are edited, they can be used to train the four-dimensional characterization network. In this embodiment, the four-dimensional characterization network adopts the D-NeRF model (full name: Neural Radiance Fields for Dynamic Scenes), and its implementation method belongs to the prior art and will not be repeated. When the four-dimensional characterization network is retrained, the rendering perspective and timestamp sequence can be directly input into it, and the reconstructed 6-degree-of-freedom video can be directly output. It should be noted that the input timestamp sequence needs to be the same length as the reconstructed 6-degree-of-freedom video.

[0098] In order to better demonstrate the specific implementation and technical effects of the present invention, the text-driven zero-sample 6-DOF video editing method shown in steps S1 to S7 in the above preferred implementation is applied to a specific example.

[0099] Example

[0100] The specific implementation process of the text-driven zero-sample 6-DOF video editing method used in this embodiment is as described above and will not be repeated here.

[0101] This embodiment first obtains a multi-view source video data set collected from the real world, and records the internal and external parameter information of the camera at different positions. During the video editing process, each frame of the multi-view source video is inverted and denoised, and multi-scale cross-attention maps, self-attention maps and noise hidden codes are stored. After all videos of all perspectives are edited, the method of the present invention can train a four-dimensional representation network based on limited data. For quantitative indicators, the present invention is compared with Vid2vid-zero, FateZero and IN4D-to-4D methods, and CLIP Text-Image Direction Similarity, CLIP Text-Image Similarity indicators and user preferences are selected for testing in the coffee martini data set, and the test results are shown in Table 1.

[0102] Table 1. Quantitative evaluation table of test results

[0103]

[0104] This embodiment also compares the method of the present invention with the other three methods, which are: IN4D-4D, a 4D editing method based on dynamic NeRF, FateZreo and Vid2vid-Zero. The training data of the D-NeRF model is modified using the above three comparison methods, and then the D-NeRF model is retrained to obtain the final rendering result. The rendering results of the above three comparison methods and the method of the present invention are compared. Figure 3 As shown. Figure 3 In the figure, the first row represents the rendering at time t=0 from the main perspective, the second row represents the rendering at time t=1 from the main perspective, and so on. Figure 3 The first column in is the rendering result of the D-NeRF model when no image modification is performed. Figure 3 The 2nd to 5th columns in the figure are the comparison results between the method of the present invention and IN4D-4D, FateZero and Vid2vid-Zero after changing the word "man" in the text description to "Spider-Man". Figure 3 Columns 6 to 9 in the figure are the comparison results between the method of the present invention and IN4D-4D, FateZero, and Vid2vid-Zero after adding "Van Gogh style" to the text description. It can be seen that the method of the present invention uses the super strong zero-sample generalization ability of the diffusion model to systematically enhance the contextual semantic consistency, temporal consistency, and cross-view consistency in the diffusion process, and shows high-quality editing effects on the 6-DOF video editing task.

[0105] It should also be noted that the text-driven zero-sample 6-DOF video editing method in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a text-driven zero-sample 6-DOF video editing system corresponding to the text-driven zero-sample 6-DOF video editing method provided in the above embodiment, such as Figure 4 As shown, it includes:

[0106] The encoding module is used to obtain a pre-trained diffusion model including a text encoder, an image encoder-decoder and a UNet network, and use the diffusion model to process the original images of all perspectives in the multi-perspective source video frame by frame. First, a frame of the original image is passed to the encoder in the image encoder-decoder for encoding to obtain the original image latent code;

[0107] An inversion module is used to encode the acquired text prompt through a text encoder to obtain a text representation, input the original image hidden code and the text representation into the UNet network to invert the diffusion process, add noise to the original image hidden code through multiple time steps to obtain a noisy hidden code, and save the cross-attention map and self-attention map generated at each time step of the noisy process;

[0108] The reconstruction and denoising module is used to reconstruct and denoise the text representation and the noise latent code obtained at each time step during the denoising process to obtain the updated text-free control vector at each time step;

[0109] An editing and denoising module is used to modify the calculation method of the cross-attention layer and the self-attention layer in the UNet network, use the modified UNet network as a UNet editor, replace the text prompt with the edited text prompt, encode the edited text prompt through a text encoder to obtain a target text representation, input the noise hidden code, the target text representation and the updated text-free control vector into the UNet editor, and edit and denoise the noise hidden code through multiple time steps to obtain an edited image hidden code;

[0110] A decoding module, used for inputting the edited image latent code outputted at the last time step of the edit denoising process into the decoder in the image encoder-decoder for decoding to obtain a target image;

[0111] An edited image generation module is used to calculate a pixel-level mask through the cross-attention map generated by the denoising process, and weight the target image and the original image using the pixel-level mask as a weight to obtain an edited image and form an edited video;

[0112] The video reconstruction module is used to input the edited video into the trained four-dimensional representation network for re-training, input the rendering perspective and timestamp sequence into the re-trained four-dimensional representation network, and output the reconstructed 6-DOF video.

[0113] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the text-driven zero-sample 6-DOF video editing method provided in the above embodiment, which includes a memory and a processor;

[0114] The memory is used to store computer programs;

[0115] The processor is used to implement the text-driven zero-sample 6-DOF video editing method in the above embodiment when executing the computer program.

[0116] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0117] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the text-driven zero-sample 6-degree-of-freedom video editing method provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the text-driven zero-sample 6-degree-of-freedom video editing method in the above embodiment can be implemented.

[0118] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0119] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0120] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0121] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A text-driven zero-sample 6-DOF video editing method, characterized in that: The following steps are involved: S1: Obtain a pre-trained diffusion model including a text encoder, an image encoder-decoder and a UNet network, and use the diffusion model to process the original images of all perspectives in the multi-perspective source video frame by frame. First, a frame of the original image is passed to the encoder in the image encoder-decoder for encoding to obtain the original image latent code; S2: After encoding the acquired text prompt through a text encoder, a text representation is obtained, the original image hidden code and the text representation are input into the UNet network to invert the diffusion process, the original image hidden code is denoised through multiple time steps to obtain a noisy hidden code, and the cross-attention map and self-attention map generated at each time step of the denoising process are saved; S3: The text representation and the noisy latent code obtained at each time step during the denoising process are reconstructed and denoised to obtain the updated text-free control vector at each time step; S4: modify the calculation method of the cross-attention layer and the self-attention layer in the UNet network, use the modified UNet network as a UNet editor, replace the text prompt with the edited text prompt, encode the edited text prompt through a text encoder to obtain a target text representation, input the noisy hidden code, the target text representation and the updated text-free control vector into the UNet editor, and denoise the noisy hidden code through multiple time steps to obtain an edited image hidden code; S5: inputting the edited image latent code outputted at the last time step of the edit denoising process into the decoder in the image encoder-decoder for decoding to obtain a target image; S6: Calculate the pixel-level mask based on the cross-attention map generated by the denoising process, and use the pixel-level mask as a weight to weight the target image and the original image to obtain the edited image and form the edited video; S7: Input the edited video into the trained 4D representation network for retraining, input the rendering view and timestamp sequence into the retrained 4D representation network, and output the reconstructed 6-DOF video.

2. The text-driven zero-sample 6-DOF video editing method according to claim 1, characterized in that: In step S3, in the reconstruction and denoising of the tth time step, the initial text-free control vector of the tth time step is obtained and used as the first UNet control condition, and the UNet network processes the noise hidden code generated at the tth time step to obtain the first text-free noise predicted at the tth time step; the text representation is used as the second UNet control condition, and the UNet network processes the noise hidden code generated at the tth time step to obtain the first text-controlled noise predicted at the tth time step, and the first text-controlled noise and the first text-free noise are weightedly summed to obtain the first denoising noise of the tth time step.

3. The text-driven zero-sample 6-DOF video editing method according to claim 2, characterized in that: The reconstructed denoised hidden code of the (t-1)th time step is calculated based on the first denoised noise of the tth time step, the mean square error between the reconstructed denoised hidden code and the noise hidden code of the (t-1)th time step is calculated, and the text-free control vector of the tth time step is optimized based on minimizing the mean square error.

4. The text-driven zero-sample 6-DOF video editing method according to claim 1, characterized in that: In step S4, the calculation method of the cross-attention layer in the UNet editor is as follows: in the i-th cross-attention layer at the t-th time step, the cross-attention map generated by the i-th cross-attention layer at the t-th time step in the denoising process is used to replace the cross-attention map generated by the cross-attention layer in the editing denoising process, and the target text representation is used as the first value representation. The replaced cross-attention map and the first value representation are calculated by softmax and output.

5. The text-driven zero-sample 6-DOF video editing method according to claim 1, characterized in that: In step S4, the calculation method of the self-attention layer in the UNet editor is as follows: in the i-th self-attention layer at the t-th time step, the self-attention map generated by the i-th self-attention layer at the t-th time step in the denoising process and the self-attention map generated by the self-attention layer in the editing denoising process are weighted summed to obtain a modified self-attention map, and the edited image latent code generated at the t-th time step is used as the second value representation. The modified self-attention map and the second value representation are calculated by softmax and output.

6. The text-driven zero-sample 6-DOF video editing method according to claim 1, characterized in that: In step S4, in the t-th time step of editing and denoising, the updated text-free control vector of the t-th time step is obtained and used as the third UNet control condition, and the UNet editor processes the noise hidden code generated at the t-th time step to obtain the second text-free noise predicted at the t-th time step; the target text representation is used as the fourth UNet control condition, and the UNet editor processes the noise hidden code generated at the t-th time step to obtain the second text-controlled noise predicted at the t-th time step, and the second text-controlled noise and the second text-free noise are weightedly summed to obtain the second denoised noise at the t-th time step, and finally the edited image hidden code of the (t-1)-th time step is calculated based on the second denoised noise at the t-th time step.

7. The text-driven zero-sample 6-DOF video editing method according to claim 1, characterized in that: In step S6, the specific process of calculating the pixel-level mask from the cross-attention map generated by the noisy process is: adding the cross-attention maps generated by each cross-attention layer in each time step of the noisy process to obtain a fused attention map, dividing the fused attention map with the maximum value in the fused attention map, and using the division result as a pixel-level mask.

8. A text-driven zero-sample 6-DOF video editing system, characterized in that: include: The encoding module is used to obtain a pre-trained diffusion model including a text encoder, an image encoder-decoder and a UNet network, and use the diffusion model to process the original images of all perspectives in the multi-perspective source video frame by frame. First, a frame of the original image is passed to the encoder in the image encoder-decoder for encoding to obtain the original image latent code; An inversion module is used to encode the acquired text prompt through a text encoder to obtain a text representation, input the original image hidden code and the text representation into the UNet network to invert the diffusion process, add noise to the original image hidden code through multiple time steps to obtain a noisy hidden code, and save the cross-attention map and self-attention map generated at each time step of the noisy process; The reconstruction and denoising module is used to reconstruct and denoise the text representation and the noise latent code obtained at each time step during the denoising process to obtain the updated text-free control vector at each time step; An editing and denoising module is used to modify the calculation method of the cross-attention layer and the self-attention layer in the UNet network, use the modified UNet network as a UNet editor, replace the text prompt with the edited text prompt, encode the edited text prompt through a text encoder to obtain a target text representation, input the noise hidden code, the target text representation and the updated text-free control vector into the UNet editor, and edit and denoise the noise hidden code through multiple time steps to obtain an edited image hidden code; A decoding module, used for inputting the edited image latent code outputted at the last time step of the edit denoising process into the decoder in the image encoder-decoder for decoding to obtain a target image; An edited image generation module is used to calculate a pixel-level mask through the cross-attention map generated by the denoising process, and weight the target image and the original image using the pixel-level mask as a weight to obtain an edited image and form an edited video; The video reconstruction module is used to input the edited video into the trained four-dimensional representation network for re-training, input the rendering perspective and timestamp sequence into the re-trained four-dimensional representation network, and output the reconstructed 6-DOF video.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the text-driven zero-sample 6-DOF video editing method as described in any one of claims 1 to 7 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the text-driven zero-sample 6-DOF video editing method as described in any one of claims 1 to 7 when executing the computer program.

Images