Interactive personalized video controllable generation method and device, equipment and medium

Abstract

The invention discloses an interactive personalized video controllable generation method and device, equipment and a medium, and belongs to the technical field of video generation. The method comprises the following steps: collecting a personalized image as a source image; inputting the obtained source image into a two-dimensional image appearance updater for fine tuning to obtain an image after text adaptation; copying multiple frames of images from the images subjected to text adaptation, and converting the copied multiple frames of images into initial video frames of the text video; and on the basis of the initial video frame, taking the image and the text after text adaptation as input of a pre-trained video diffusion model, and generating a personalized video. Two-dimensional and time information in text and video generation based on image conditions is separated through the two-dimensional image appearance updater so as to align the two-dimensional text and the input image, and the capacity of an existing diffusion model from the text and the image to the video is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to an interactive, personalized, controllable generation method, device, equipment and medium. Background Art

[0002] With the rapid development of technology, researchers have begun to pay more and more attention to diffusion models on visual generation tasks. Great progress has been made in text-to-image (T2I) and text-to-video (T2V), but generating high-quality visual content based on text is still a challenging task, which requires not only generating visually realistic and smooth and time-consistent motion based on given text and images. Early developments in video diffusion models focused on a single guide, either through images or text. Like animation and zero-shot focus on text-guided video generation, while stable video diffusion models emphasize image-guided video generation. Personalized Image Animator (PIA) extends these methods by integrating text and image guidance, providing greater flexibility in guiding video generation. However, despite its versatility, PIA has difficulty producing high-quality results when the image and text description are not perfectly aligned. For example, if the text describes "a girl's smile" and the corresponding image fits this description, PIA can produce satisfactory results. However, when the text is changed to "a girl playing ball" and the image remains unchanged, PIA often fails to provide high-quality videos. This inconsistency between images and texts poses additional challenges to the diffusion process as the model must manage both temporal consistency and changes in texture simultaneously. Summary of the invention

[0003] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the object of the present invention is to provide an interactive and controllable generation method, device, equipment and medium for personalized video based on a diffusion model.

[0004] The first technical solution adopted by the present invention is:

[0005] An interactive and controllable method for generating personalized videos, comprising the following steps:

[0006] Collecting personalized images as source images;

[0007] The obtained source image is input into the two-dimensional image appearance updater for fine-tuning to obtain the image after text adaptation;

[0008] Copying multiple frames of images from the text-adapted image, and converting the copied multiple frames of images into initial video frames of T2V;

[0009] Based on the initial video frame, the text-adapted image and text are used as input to the pre-trained video diffusion model to generate personalized videos.

[0010] Furthermore, based on the initial video frame, the text-adapted image and text are used as inputs of a pre-trained video diffusion model to generate a personalized video, including:

[0011] Set the number of frames N to fit the image to the text Perform N replications to generate the initial video frame for the video diffusion model

[0012] For N-dimensional random Gaussian noise ε N Sampling is performed and the noise ε N and Merge using the scheduler of the diffusion model;

[0013] Injecting the merged noisy video frame into the intermediate diffusion step is called the hijacking step;

[0014] Generate videos with different results for different hijacking steps to obtain the hijacking interval with better effect;

[0015] The videos generated in the hijacking interval are filtered to obtain videos with satisfactory appearance and motion as the final personalized videos.

[0016] Furthermore, the step of injecting the merged noisy video frame into the intermediate diffusion step is called the hijacking step, and includes:

[0017] Inject random Gaussian noise into different hijacking steps to obtain noisy video frames;

[0018] Match the noisy video frame with the pre-trained video diffusion model as the input frame of the intermediate diffusion process;

[0019] According to the set hijacking steps, these noisy video frames are injected at the specified diffusion steps to control the video generation process.

[0020] Furthermore, the method of generating videos with different results for different hijacking steps and obtaining a hijacking interval with better effect includes:

[0021] Set a range for different hijacking steps, starting from the initial diffusion step, and gradually adjust and record the results generated each time;

[0022] Generate multiple video samples at different hijacking steps and compare the appearance and motion effects of each video sample;

[0023] Filter out the hijacking interval with better effect according to the preset visual and motion evaluation indicators;

[0024] By optimizing the algorithm to lock the optimal hijacking interval, it is ensured that the generated video achieves the expected personalized effect.

[0025] Furthermore, the video generated according to the hijacking interval is filtered to obtain a video with satisfactory appearance and motion as the final personalized video, including:

[0026] The video samples generated based on the optimal hijacking interval are screened in multiple dimensions according to appearance and motion indicators to exclude videos that do not meet the personalized needs of users;

[0027] The samples that meet the appearance and motion requirements are retained as the final output video to ensure a high degree of consistency between the personalization of the video content and the visual effects.

[0028] Furthermore, the step of inputting the obtained source image into a two-dimensional image appearance updater for fine-tuning to obtain an image after text adaptation includes:

[0029] According to the user's personalized text description, the parameters in the 2D image appearance updater are adjusted to adapt to the text requirements;

[0030] The fine-tuned images are filtered and selected, and the selected images are used as inputs for subsequent processing to achieve appearance adaptation of personalized video content.

[0031] Furthermore, the step of copying multiple frames according to the image adapted from the text and converting the copied multiple frames into initial video frames of the Wensheng video includes:

[0032] Preprocessing multiple frames of images, wherein the preprocessing includes resolution adjustment and image quality enhancement;

[0033] The parameters of the initial video frame are set to provide an input basis for the generation process of the diffusion model.

[0034] The second technical solution adopted by the present invention is:

[0035] An interactive and controllable personalized video generation device, comprising:

[0036] An image collection module, used for collecting personalized images as source images;

[0037] A text adaptation module, used for inputting the obtained source image into the two-dimensional image appearance updater for fine-tuning to obtain an image after text adaptation;

[0038] An image conversion module, used for copying multiple frames of images from the text-adapted image, and converting the copied multiple frames of images into initial video frames of T2V;

[0039] The video generation module is used to generate personalized videos based on the initial video frame, using the text-adapted images and text as inputs to the pre-trained video diffusion model.

[0040] The third technical solution adopted by the present invention is:

[0041] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement an interactive personalized video controllable generation method as described above.

[0042] The fourth technical solution adopted by the present invention is:

[0043] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement an interactive personalized video controllable generation method as described above.

[0044] The fifth technical solution adopted by the present invention is:

[0045] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned interactive personalized video controllable generation method.

[0046] The beneficial effects of the present invention are as follows: the present invention separates the two-dimensional and temporal information in the text and image-conditional video generation through a two-dimensional image appearance updater to align the two-dimensional text and the input image, thereby enhancing the capabilities of the existing text and image to video diffusion model; in addition, the present invention proposes Temporal-SDEdit to seamlessly integrate text adaptation to images into a pre-trained video diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1is a schematic diagram of an interactive, controllable method for generating personalized videos based on a diffusion model in an embodiment of the present invention;

[0049] Figure 2 This is a graph showing the results of the video generation algorithm on the AnimateBench dataset in different text adaptation methods;

[0050] Figure 3 This is the Temporal-SDEdit stage process in an embodiment of the present invention;

[0051] Figure 4 It is a flowchart of the steps of an interactive and controllable method for generating personalized videos in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0054] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0055] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0056] In order to solve the existing technical problems, the present invention proposes a method to generate high-quality and temporally smooth videos by separating two-dimensional and temporal information. In short, the goal of the present invention is to resolve the inconsistency between text and image by incorporating an additional 2D image appearance updater. The appearance updater ensures that the image is adapted to the text before being input into the video diffusion model. In addition, in order to effectively integrate the text-adapted image into the video diffusion process, the present invention also extends the stochastic differential equation guided image generation and editing method (SDEdit) to a temporal version, namely SDEdit on temporal sequence (Temporal-SDEdit). This adaptation allows the video diffusion model to generate videos that are more relevant to the input.

[0057] Example 1

[0058] like Figure 4 As shown, this embodiment provides an interactive and controllable method for generating personalized videos, comprising the following steps:

[0059] S1. Collect personalized images as source images.

[0060] S2. Input the obtained source image into the two-dimensional image appearance updater for fine-tuning to obtain the text-adapted image.

[0061] Specifically, an auxiliary interactive 2D diffusion model is used to synchronize the 2D appearance details from text and image inputs to obtain images with different degrees of text adaptation.

[0062] As an optional implementation, step S2 specifically includes the following steps:

[0063] S21, adjusting parameters in the two-dimensional image appearance updater according to the user's personalized text description to adapt to the text requirements;

[0064] S22, filtering and selecting the fine-tuned image, and using the selected image as an input of a subsequent processing process to achieve appearance adaptation of personalized video content.

[0065] S3, copying multiple frames of images from the text-adapted image, and converting the copied multiple frames of images into initial video frames of T2V (T2V).

[0066] In some embodiments, step S3 also includes a step of preprocessing multiple frames of images, wherein the preprocessing includes resolution adjustment, image quality enhancement, etc. In addition, by setting the parameters of the initial video frame, an input basis is provided for the generation process of the diffusion model.

[0067] S4. Based on the initial video frame, the text-adapted image and text are used as input of the pre-trained video diffusion model to generate personalized video.

[0068] As an optional implementation, step S4 specifically includes the following steps:

[0069] S41, set the number of frames N, and adapt the image to the text Perform N replications to generate the initial video frame for the video diffusion model

[0070] S42, for N-dimensional random Gaussian noise ε N Sampling is performed and the noise ε N and The scheduler using the diffusion model is used for merging.

[0071] S43, injecting the merged noisy video frame into the intermediate diffusion step, which is called the hijacking step.

[0072] In some embodiments, step S43 includes the following steps:

[0073] S431: injecting random Gaussian noise into different hijacking steps to obtain noisy video frames;

[0074] S432: Matching the noisy video frame with the pre-trained video diffusion model as an input frame of the intermediate diffusion process;

[0075] S433: According to the set hijacking step number, these noisy video frames are injected in the specified diffusion step to control the video generation process.

[0076] S44. Generate videos with different results for different hijacking steps to obtain a hijacking interval with better effect.

[0077] In some embodiments, step S44 includes the following steps:

[0078] S441: setting ranges for different hijacking steps, starting from the initial diffusion step, and gradually adjusting and recording the results generated each time;

[0079] S442: Generate multiple video samples at different hijacking steps, and compare the appearance and motion effect of each video sample;

[0080] S443: Screening out a hijacking interval with better effect according to preset visual and motion evaluation indicators;

[0081] S444: The optimal hijacking interval is locked by optimizing the algorithm to ensure that the generated video achieves the expected personalized effect.

[0082] S45, filtering the video generated in the hijacking interval to obtain a video with satisfactory appearance and motion as the final personalized video.

[0083] In some embodiments, step S45 includes the following steps:

[0084] S451: performing multi-dimensional screening on the video samples generated based on the optimal hijacking interval according to appearance and motion indicators, and excluding videos that do not meet the personalized needs of the user;

[0085] S452: retain samples that meet the appearance and motion requirements as the final output video to ensure high consistency between video content personalization and visual effects.

[0086] In this embodiment, Temporal-SDEdit is proposed to seamlessly integrate the refined image into the pre-trained video diffusion model, better maintain the consistency of T2V video generation, and use a denoising process with a smaller time step than the original one; finally, a video with satisfactory appearance and motion is generated.

[0087] The following is a detailed explanation with reference to the accompanying drawings and specific embodiments.

[0088] This embodiment provides a method for controllable generation of 2D interactive personalized videos based on a diffusion model, which theoretically verifies that controllable guidance from different sources brings additional complexity, often increases the burden on the model, and makes it difficult to maintain semantic and temporal consistency. It essentially enhances the capabilities of existing diffusion models of text, images, etc. for videos.

[0089] Among them, in order to solve the misalignment problem between input text and image in video generation, a text and an image are required as input conditions. To reduce the pressure on the model, we separate the 2D and temporal information in text and image-based video generation through a 2D image appearance updater. This process adjusts the image in a given text and image pair to enhance coherence. Subsequently, we introduce Temporal-SDEdit to seamlessly integrate the refined input image into the video diffusion model.

[0090] In this embodiment, the overall process is as follows: Figure 1 As shown, first, the collected personalized images are used as source images, which are formulated by the text-adapted T2I diffusion model and then adapted by the same T2I model. In the first step, the text-adapted images reduce the burden of appearance updates in the entire complex video generation process. Secondly, in the second step, we copy 'F' frames from the text-adapted images and convert them into initial video frames for T2V. And through "Temporal-SDEdit" to better maintain the consistency of T2V video generation, a denoising process with a smaller time step than the original is used. Finally, a video with satisfactory appearance and motion is generated.

[0091] The details of the method proposed in this embodiment will be discussed below from two aspects, mainly including the 2D image appearance update phase process and the Temporal-SDEdit phase process.

[0092] (1) 2D image appearance update process

[0093] The goal of this process is to update the input image to a text-adapted image. This can be achieved by using an additional 2D image appearance updater. There are currently popular 2D image appearance updaters, such as Prompt-to-Prompt and InstructPix2Pix, and InstantID uses only one facial image to handle image personalization in various styles. DragDiffusion is an interactive point-based image update. In this embodiment, the image is mainly updated using a two-dimensional diffusion model to obtain the expected text-adapted image. We choose the commonly used InstructPix2Pix as our 2D image appearance updater, which has rich instruction-level prompts, including objects, facial backgrounds, etc. The process specifically includes the following steps:

[0094] Step 1: User personalized image collection.

[0095] Step 2: Get the image after text adaptation. First, input the collected images into the 2D appearance updater (diffusion model) for fine-tuning to obtain the image after text adaptation, such as Figure 2 shown.

[0096] Step 3: Select the randomly generated multiple times adapted images for filtering. Considering the uncertainty brought by randomness, a one-way mapping relationship between the prompt of the source image and the prompt after text adaptation is established, and finally the most satisfactory image is obtained.

[0097] Step 4: Video generation: The text-adapted image and text are used as input at the same time, and the video diffusion model is used for fine-tuning to generate personalized videos.

[0098] (2) Temporal-SDEdit stage process

[0099] The core idea of ​​SDEdit is to “hijack” the normal noise-to-image generation process by injecting a noisy image into an intermediate diffusion step. Formally, instead of gradually denoising the random noise ε into the final diffusion step and generating χ0, we use to replace ε, where Is a real image By “hijacking” the diffusion process, the SDEdit method ensures that the unconditional diffusion model is accurate to real images. Generate image conditions. In this embodiment, we extend the SDEdit method to temporal-SDEdit. Figure 3 , the process specifically includes the following steps:

[0100] Step 1: Given N frames, we repeat the text adaptation of the image Perform N times to generate

[0101] Step 2: We add N-dimensional random Gaussian noise ε N Sampling is performed and the noise ε N and Merged with diffusion scheduler.

[0102] Step 3: Inject the merged noise into the intermediate diffusion step, which is called the hijacking step.

[0103] Step 4: Different step hijacking will generate different video results, and finally you can get the hijacking range with better effect.

[0104] Step 5: Filter the videos generated in the hijacking interval to obtain videos with satisfactory appearance and motion.

[0105] In summary, the present invention has at least the following advantages and beneficial effects compared to the prior art:

[0106] (1) Fast Robust Perception Capability: The system framework of the present invention can have a good understanding of text and image alignment for the diffusion model.

[0107] (2) Rapid deployment and fine-tuning: This method only requires image collection and fine-tuning, and is highly reproducible, laying the foundation for subsequent real-time testing.

[0108] (3) Flexibility and scalability: This framework can be used in any scenario. It only needs to be fully deployed in a fine-tuned manner to complete customization to meet the needs of different scenarios.

[0109] Example 2

[0110] This embodiment provides an interactive and controllable device for generating personalized videos, including:

[0111] An image collection module, used for collecting personalized images as source images;

[0112] A text adaptation module, used for inputting the obtained source image into the two-dimensional image appearance updater for fine-tuning to obtain an image after text adaptation;

[0113] An image conversion module, used for copying multiple frames of images from the text-adapted image, and converting the copied multiple frames of images into initial video frames of T2V;

[0114] The video generation module is used to generate personalized videos based on the initial video frame, using the text-adapted images and text as inputs to the pre-trained video diffusion model.

[0115] Since the device is an interactive personalized video controllable generation device of an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0116] Example 3

[0117] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 4 An interactive and controllable method for generating personalized videos is shown.

[0118] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0119] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0120] Since the electronic device is an electronic device corresponding to an interactive personalized video controllable generation method in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0121] Example 4

[0122] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 4 An interactive and controllable method for generating personalized videos is shown.

[0123] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0124] Since the storage medium is a storage medium corresponding to an interactive personalized video controllable generation method in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0125] Example 5

[0126] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of an interactive personalized video controllable generation method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0127] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0128] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0129] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. An interactive and controllable method for generating personalized videos, characterized in that: The following steps are involved: Collecting personalized images as source images; The obtained source image is input into the two-dimensional image appearance updater for fine-tuning to obtain the image after text adaptation; Copying multiple frames of images from the text-adapted image, and converting the copied multiple frames of images into initial video frames of the text-generated video; Based on the initial video frame, the text-adapted image and text are used as input to the pre-trained video diffusion model to generate personalized videos.

2. The interactive personalized video controllable generation method according to claim 1, characterized in that: The method of generating a personalized video based on the initial video frame by using the text-adapted image and text as input of a pre-trained video diffusion model includes: Set the number of frames N to fit the image to the text Perform N replications to generate the initial video frame for the video diffusion model For N-dimensional random Gaussian noise ε N Sampling is performed and the noise ε N and Merge using the scheduler of the diffusion model; Injecting the merged noisy video frame into the intermediate diffusion step is called the hijacking step; Generate videos with different results for different hijacking steps to obtain the hijacking interval with better effect; The videos generated in the hijacking interval are filtered to obtain videos with satisfactory appearance and motion as the final personalized videos.

3. The interactive personalized video controllable generation method according to claim 2, characterized in that: The step of injecting the merged noisy video frame into the intermediate diffusion step is called the hijacking step, which includes: Inject random Gaussian noise into different hijacking steps to obtain noisy video frames; Match the noisy video frame with the pre-trained video diffusion model as the input frame of the intermediate diffusion process; According to the set hijacking steps, these noisy video frames are injected at the specified diffusion steps to control the video generation process.

4. The interactive personalized video controllable generation method according to claim 2, characterized in that: The method of generating videos with different results for different hijacking steps and obtaining hijacking intervals with better effects includes: Set a range for different hijacking steps, starting from the initial diffusion step, and gradually adjust and record the results generated each time; Generate multiple video samples at different hijacking steps and compare the appearance and motion effects of each video sample; Filter out the hijacking interval with better effect according to the preset visual and motion evaluation indicators; By optimizing the algorithm to lock the optimal hijacking interval, it is ensured that the generated video achieves the expected personalized effect.

5. The interactive personalized video controllable generation method according to claim 2, characterized in that: The video generated according to the hijacking interval is filtered to obtain a video with satisfactory appearance and motion as the final personalized video, including: The video samples generated based on the optimal hijacking interval are screened in multiple dimensions according to appearance and motion indicators to exclude videos that do not meet the personalized needs of users; The samples that meet the appearance and motion requirements are retained as the final output video to ensure a high degree of consistency between the personalization of the video content and the visual effects.

6. The interactive and controllable generation method of personalized video according to claim 1, characterized in that: The step of inputting the obtained source image into a two-dimensional image appearance updater for fine-tuning to obtain an image after text adaptation comprises: According to the user's personalized text description, the parameters in the two-dimensional image appearance updater are adjusted to adapt to the text requirements; the fine-tuned image is filtered and selected, and the selected image is used as input for subsequent processing to achieve appearance adaptation of personalized video content.

7. The interactive personalized video controllable generation method according to claim 1, characterized in that: The method of copying multiple frames according to the image adapted from the text and converting the copied multiple frames of the image into initial video frames of the Wensheng video includes: Preprocessing multiple frames of images, wherein the preprocessing includes resolution adjustment and image quality enhancement; The parameters of the initial video frame are set to provide an input basis for the generation process of the diffusion model.

8. An interactive and controllable personalized video generation device, characterized in that: include: An image collection module, used for collecting personalized images as source images; A text adaptation module, used for inputting the obtained source image into the two-dimensional image appearance updater for fine-tuning to obtain an image after text adaptation; An image conversion module, used for copying multiple frames of images from the text-adapted image, and converting the copied multiple frames of images into initial video frames of the text video; The video generation module is used to generate personalized videos based on the initial video frame, using the text-adapted images and text as inputs to the pre-trained video diffusion model.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images