Method and device for generating Cantonese episode video from text, electronic equipment and medium

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, in particular to a method and device for generating a Cantonese episode video through a text, electronic equipment and a medium. According to the invention, the inversion latent variable is calculated from the input video; inputting the reverse latent variable into the key frame generation module subjected to fine tuning of the time attention layer to obtain a key frame sequence; the key frame sequence comprises motion information; inputting the key frame sequence into the time interpolation module to obtain an extended frame sequence; and inputting the extended frame sequence into the spatial super-resolution module to obtain a target Guangdong episode video. Thus, video generation of customized motion is realized through the key frame generation module in the text generation video diffusion model after time fine tuning, the motion of the target video can be accurately reproduced, and diversified visual changes can be generated.

Description

Technical Field

[0001] The present application relates to the field of computer vision and artificial intelligence technology, and in particular to a method, device, electronic device and medium for generating Cantonese opera videos from text. Background Art

[0002] Text-to-Video (T2V) diffusion models have been a hot topic in the field of artificial intelligence in recent years, aiming to generate corresponding video content from text descriptions. Current text-to-video diffusion models have made significant progress in generating high-quality videos. However, these models still face many challenges in generating videos with customized motion, especially in accurately reproducing the motion of the target video and generating diverse visual changes.

[0003] In the related art, simple extensions of static image customization methods usually lead to complex entanglement of appearance and motion data, which cannot achieve high-quality motion customization. In addition, current models lack the ability to reproduce specific motions or generate diverse variants of the same motion with different visual attributes and backgrounds.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] In view of this, the embodiments of the present application at least provide a method, device, electronic device and medium for generating Cantonese opera videos from text, which can realize customized motion video generation, accurately reproduce the motion of the target video, and generate diverse visual changes.

[0006] This application mainly includes the following aspects:

[0007] In a first aspect, an embodiment of the present application provides a method for generating a Cantonese opera video from text, which is applied to a trained target text generation video diffusion model, wherein the target text generation video diffusion model includes a key frame generation module, a time interpolation module, and a spatial super-resolution module. The method for generating a Cantonese opera video from text includes:

[0008] Compute the inverse latent variable from the input video;

[0009] Inputting the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information;

[0010] Inputting the key frame sequence into the time interpolation module to obtain an extended frame sequence;

[0011] The extended frame sequence is input into the spatial super-resolution module to obtain a target Cantonese opera video.

[0012] In a second aspect, an embodiment of the present application further provides a device for generating a Cantonese opera video from a text, which is applied to a trained target text generation video diffusion model, wherein the target text generation video diffusion model includes a key frame generation module, a time interpolation module, and a spatial super-resolution module, and the device for generating a Cantonese opera video from a text includes a calculation module, a first determination module, a second determination module, and a third determination module; wherein:

[0013] The calculation module is used to calculate the inversion latent variable from the input video;

[0014] The first determination module is used to input the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information;

[0015] The second determination module is used to input the key frame sequence into the time interpolation module to obtain an extended frame sequence;

[0016] The third determination module is used to input the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video.

[0017] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for generating Cantonese opera video from text described in the first aspect above.

[0018] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating Cantonese opera video from text described in the first aspect are executed.

[0019] The embodiment of the present application provides a method, device, electronic device and medium for generating Cantonese opera video from text, which calculates the inversion latent variable from the input video; inputs the inversion latent variable into the key frame generation module after fine-tuning the time attention layer to obtain a key frame sequence; the key frame sequence includes motion information; inputs the key frame sequence into the time interpolation module to obtain an extended frame sequence; inputs the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video. In this way, customized motion video generation is achieved through the key frame generation module in the time-fine-tuned text-generated video diffusion model, which can accurately reproduce the motion of the target video and generate a variety of visual changes.

[0020] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flow chart showing a method for generating a Cantonese opera video from text provided in an embodiment of the present application is shown;

[0023] Figure 2 One of the functional module diagrams of a device for generating Cantonese opera videos from text provided in an embodiment of the present application is shown;

[0024] Figure 3 A second functional module diagram of a method for generating a Cantonese opera video from text provided in an embodiment of the present application is shown;

[0025] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0027] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0028] In order to enable those skilled in the art to use the contents of this application, the following implementation method is given in combination with the specific application scenario of "generating Cantonese opera video from text". For those skilled in the art, the general principles defined here can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0029] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario where text needs to be generated into a Cantonese opera video. The embodiments of the present application are not limited to specific application scenarios. Any scheme using the method and device for generating a Cantonese opera video from text provided by the embodiments of the present application is within the protection scope of the present application.

[0030] It is worth noting that before the present application, the simple extension of the static image customization method in the related art usually leads to a complex entanglement of appearance and motion data, which cannot achieve high-quality motion customization. In addition, the current models lack the ability to reproduce specific motions or generate diverse variants of the same motion with different visual attributes and backgrounds.

[0031] In view of the above problems, the embodiment of the present application calculates the inversion latent variable from the input video; inputs the inversion latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information; inputs the key frame sequence into the temporal interpolation module to obtain an extended frame sequence; inputs the extended frame sequence into the spatial super-resolution module to obtain the target Cantonese opera video. In this way, customized motion video generation is achieved through the key frame generation module in the temporally fine-tuned text generation video diffusion model, which can accurately reproduce the motion of the target video and generate a variety of visual changes.

[0032] To facilitate the understanding of the present application, the technical solution provided by the present application is described in detail below in conjunction with specific embodiments.

[0033] Figure 1 This is a flow chart of a method for generating a Cantonese opera video from text provided in an embodiment of the present application. Figure 1As shown, the method for generating a Cantonese opera video from text provided in the embodiment of the present application includes the following steps:

[0034] S101: Calculate the inversion latent variable from the input video.

[0035] S102: Input the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information.

[0036] S103: Input the key frame sequence into the time interpolation module to obtain an extended frame sequence.

[0037] S104: Input the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video.

[0038] In the specific implementation, the present application proposes a video motion customization (VMC) framework, which realizes customized video generation by adjusting the temporal attention layer of the key frame generation module in the text generation video diffusion model. Specifically, the trained target text generation video diffusion model is obtained by training the initial text generation video diffusion model, that is, the key frame generation module in the initial text generation video diffusion model is fine-tuned in time to obtain the target text generation video diffusion model. In the inference stage, the inverse latent variables are first calculated from the input video through DDIM inversion, and then these inverse latent variables are input into the time-fine-tuned key frame generation module to generate a short and low-resolution key frame sequence, which contains the main motion information; then, the key frame sequence is temporally expanded using the temporal interpolation module to generate more frames to improve the smoothness of the video. Specifically, the temporal interpolation module generates new frames between key frames through an interpolation method to smooth the video playback, and then the extended frames (extended frame sequence) are spatially enhanced through the spatial super-resolution module to generate a high-resolution video output. Among them, the spatial super-resolution module uses deep learning technology to amplify and refine low-resolution frames to improve the visual quality of the video.

[0039] Exemplarily, after training is completed, the customized keyframe generation module is used for object motion-driven video generation with a new appearance context (eg, “two actors performing traditional Cantonese opera”).

[0040] In the embodiment of the present application, the inversion latent variable is calculated from the input video; the inversion latent variable is input into the key frame generation module after the fine-tuning of the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information; the key frame sequence is input into the temporal interpolation module to obtain an extended frame sequence; the extended frame sequence is input into the spatial super-resolution module to obtain the target Cantonese opera video. In this way, the customized motion video generation is realized by the key frame generation module in the temporally fine-tuned text generation video diffusion model, which can accurately reproduce the motion of the target video and generate a variety of visual changes.

[0041] In a possible implementation, the target text generation video diffusion model is generated according to the following steps: based on the motion distillation target of the residual vector, the temporal attention layer in the key frame generation module is adjusted, and the spatial super-resolution module and the temporal interpolation module are kept in a frozen state to generate a trained target text generation video diffusion model.

[0042] In the specific implementation, this application proposes a unique method of adjusting only the temporal attention layer in the key frame generation module. This method achieves efficient customization of motion videos by optimizing the temporal attention layer while keeping the spatial attention layer and other modules (such as the spatial super-resolution module and the temporal interpolation module) unchanged. This adjustment strategy ensures the efficiency and speed of model training, reduces the consumption of computing resources, and enables the training process to be completed under limited hardware conditions.

[0043] It should be noted that, by adopting the above framework, this application can realize efficient motion customized video generation. By optimizing the temporal attention layer, the target text generation video diffusion model can accurately capture and reproduce the motion trajectory in the input video. Compared with related technologies, this application has significant advantages in motion reproduction and visual diversity. The generated video maintains a consistent motion trajectory under different backgrounds and visual attributes, while ensuring high resolution and high quality.

[0044] In addition, the innovative adjustment strategy of this application reduces the computational complexity of model training and significantly reduces the demand for hardware resources. Only the strategy of adjusting the temporal attention layer enables the training process to be completed under limited computing resources. In this way, through efficient training methods, the model can be trained in a shorter time, meeting the demand for rapid generation of high-quality videos.

[0045] Here, through the technical solution of the present application, video generation technology has achieved an important breakthrough, significantly improving the effect of customized video generation, and providing users with an efficient, accurate and personalized motion video generation solution.

[0046] In a possible implementation, the motion distillation target based on the residual vector adjusts the temporal attention layer in the key frame generation module, including the following steps:

[0047] Step a, obtaining an input video sequence containing N frames;

[0048] Step b, initializing the initial text generation video diffusion model, wherein the initialization includes loading pre-trained weights and configuring model parameters;

[0049] Step c, for a first video frame corresponding to any target moment in the input video sequence, sampling the first video frame through the initialized initial text generation video diffusion model to obtain a noise frame latent vector, and determining the difference between the noise frame latent vector of the first video frame and the noise frame latent vector of the second video frame as the frame residual vector of the target moment; the second video frame is a video frame that is a preset frame interval number away from the first video frame;

[0050] Step d, determining a true motion vector based on the frame residual vector;

[0051] Step e, estimating a denoised motion vector based on the denoised video vector and the frame residual vector;

[0052] Step f, adjusting the temporal attention layer in the key frame generation module according to the loss value between the true motion vector and the denoised motion vector.

[0053] In the specific implementation, this application introduces a motion distillation target based on residual vectors, by adjusting the frame residual vectors between consecutive noise potential frames as motion references. The specific technical steps are as follows: (1) Input video acquisition. Specifically, an input video sequence containing N frames is obtained from the user, and each frame is represented as v n, and preprocess the input video sequence to ensure the continuity and consistency of the video frame sequence for subsequent processing; (2) Initialization of the initial text generation video diffusion model. Specifically, the pre-trained initial text generation video diffusion model T2V is used as the basic model. The initialization of the basic model includes loading the pre-trained weights and configuring the model parameters to adapt to the current task. In addition, the temporal attention layer of the basic model is initialized to ensure that it has basic video generation capabilities and can be effectively adjusted in subsequent training; (3) The frame residual vector is calculated. Specifically, the noise frame latent vector is obtained by sampling the first video frame through the initialized initial text generation video diffusion model, and the difference between the noise frame latent vector of the first video frame and the noise frame latent vector of the second video frame is determined as the frame residual vector (also called noise motion vector) at the target moment. The frame residual vector is used to represent the motion change between consecutive frames in the video; wherein, by calculating the frame residual vector between consecutive frames, the motion trajectory in the video is extracted. This process is achieved by calculating each pair of consecutive frames one by one to ensure the accurate extraction of motion information; (4) The temporal attention layer is adjusted. Specifically, the noise motion vector and the denoised video vector are obtained to obtain the real motion vector and the denoised motion vector. The temporal attention layer in the key frame generation module is adjusted according to the loss value between the real motion vector and the denoised motion vector.

[0054] Here, during the model training process, only the temporal attention layer in the keyframe generation module is adjusted, and other modules (such as the spatial super-resolution module and the temporal interpolation module) are kept frozen, which can reduce the computational complexity and improve the training efficiency. Through the back-propagation algorithm, the parameters of the temporal attention layer in the keyframe generation module are continuously optimized according to the motion distillation target, so that it can better capture and reproduce the motion trajectory in the input video.

[0055] Here, the proposed video motion customization framework,the proposed video motion customization (VMC) framework extracts the motion trajectory,i.e., the noisy motion vector,from the residual between the noisy latent variable frames.,Specifically, the temporal attention layer of the keyframe generation module is fine-tuned,only by aligning the true value and the predicted motion vector.

[0056] It should be noted that this application introduces a motion distillation target based on residual vectors, which extracts and reproduces the motion trajectory in the target video by calculating the frame residual vectors between consecutive noise potential frames. The residual cosine similarity can be used as a loss function to minimize the cosine similarity between the true motion vector and the predicted motion vector (denoised motion vector) to ensure that the model can accurately capture and reproduce the motion information in the target video. This method improves the accuracy of motion information extraction and reproduction by accurately modeling the motion changes of consecutive frames.

[0057] In a possible implementation manner, the formula of the true motion vector is: in, is the frame residual vector at the target time t, is the noise frame latent vector of the first video frame, is the noise frame potential vector of the second video frame, c is the preset frame interval number, n is the nth video frame, is the true motion vector, α i is the noise parameter preset at the i-th time step in the forward process of the video diffusion model, is the normal distribution function.

[0058] Here, set Indicates from The sampled The potential value of the nth noise frame is defined as the frame residual vector (also called noise motion vector) at time t: Wherein c represents a fixed frame interval, that is, a preset frame interval number.

[0059] In addition, the initial text generation video diffusion model is used to obtain the noise motion vectors between the noise latent frames from the input video sequence. The noise motion vector (frame residual vector at time t) can be expressed as:

[0060]

[0061] in, is a normal distribution with mean zero and variance 2I (I represents the identity matrix), which can be obtained by the following diffusion kernel

[0062]

[0063] In a possible implementation, the formula for the denoised motion vector is: in, is the residual vector of the target frame at time t, is the noise frame latent vector of the first video frame, is the noise frame potential vector of the second video frame, c is the preset frame interval number, n is the nth video frame, is the denoised motion vector, i.e., an estimate calculated from the model's predicted noise vector, The noise parameters pre-set for the forward process of the video diffusion model, is the noise residual vector of the forward noise addition at time t of the nth frame predicted by the model, is the noise vector of the forward noise addition at time t of the n+cth frame predicted by the model, The noise vector at time t of the forward noise addition of the nth frame predicted by the model.

[0064] In a possible implementation, the formula for the denoised video vector is:

[0065] in, is the denoised video vector, is the video vector of the entire input N frames, The noise parameter pre-set for the forward process of the video diffusion model, ∈ θ is the noise vector predicted by the model, and N is the number of video frames in the input video sequence.

[0066] In a specific implementation, according to the denoised video vector Calculate denoised motion vectors in

[0067] Among them, the Tweedie formula is applied to calculate the denoised video vector The formula is as follows: in is the empirical Bayes optimal posterior expectation The Tweedie formula is a formula used in Bayesian statistics to study the mean problem of the normal distribution. It has important applications in interval estimation, hypothesis testing, and constructing shrinkage estimators.

[0068] Here, the definition of the motion distillation objective includes: designing a new loss function to optimize the temporal attention layer so that it can accurately capture the motion trajectory in the target video. Specifically, the temporal attention layer of the model is adjusted by aligning the true motion vector and the predicted motion vector. The specific formula is:

[0069] Among them, l align represents the alignment loss function, and They represent the true motion vector and the predicted motion vector (i.e., the denoised motion vector), respectively.

[0070] Exemplarily, the cosine similarity loss function is selected to measure the alignment degree between the true motion vector and the denoised motion vector. The optimization framework is finally defined as follows: In other words, the proposed optimization framework aims to maximize and Therefore, this optimization method can be seamlessly applied to video diffusion models trained using ε-matching.

[0071] In a possible implementation, in the process of training the initial text to generate video diffusion model to obtain the target text to generate video diffusion model, the method of generating Cantonese opera video from text also includes: obtaining multiple sample texts; for any sample text among the multiple sample texts, simplifying the prompt words of the sample text to obtain a target text; and using each of the target texts as a training text for training the initial text to generate video diffusion model.

[0072] In a specific implementation, simplified prompt words are used during the training process to remove background and appearance information and reduce the interference of irrelevant information on motion learning. For example, complex descriptive prompt words are simplified to prompt words that only contain motion information. For another example, the descriptive prompt word "drama actors perform classic Cantonese opera on stage" is simplified to "actors perform Cantonese opera" to focus on the extraction of motion information. Simplifying prompt words is crucial for motion distillation. This method of simplifying prompt words helps the model focus on learning motion information and avoids the impact of appearance and background changes on motion trajectories, thereby improving the quality of customized video generation. Through the processing of prompt words with unchanged appearance, the present application can generate personalized and diversified video content. This method not only improves the quality of the generated video, but also enhances the personalization and diversity of the generated video to meet the customization needs of different users.

[0073] Based on the same application concept, the embodiments of the present application also provide a device for generating Cantonese opera video from text corresponding to the method for generating Cantonese opera video from text provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the method for generating Cantonese opera video from text in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0074] like Figure 2 , Figure 3 As shown, Figure 2 One of the functional module diagrams of a device 200 for generating a Cantonese opera video from text provided in an embodiment of the present application, Figure 3 This is a second functional module diagram of a device 200 for generating a Cantonese opera video from text provided in an embodiment of the present application. Figure 2 As shown, the device 200 for generating Cantonese opera video from text is applied to the trained target text generation video diffusion model, the target text generation video diffusion model includes a key frame generation module, a time interpolation module and a spatial super-resolution module, and the device 200 for generating Cantonese opera video from text includes a calculation module 210, a first determination module 220, a second determination module 230, and a third determination module 240; wherein:

[0075] The calculation module 210 is used to calculate the inversion latent variable from the input video;

[0076] The first determination module 220 is used to input the inverted latent variable into the key frame generation module after the fine-tuning of the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information;

[0077] The second determination module 230 is used to input the key frame sequence into the time interpolation module to obtain an extended frame sequence;

[0078] The third determination module 240 is used to input the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video.

[0079] In a possible implementation, Figure 3 As shown, the device 200 for generating Cantonese opera videos from text further includes a training module 250; the training module 250 is used to generate the target text-generated video diffusion model according to the following steps:

[0080] Based on the motion distillation target of the residual vector, the temporal attention layer in the key frame generation module is adjusted, and the spatial super-resolution module and the temporal interpolation module are kept in a frozen state to generate a trained target text generation video diffusion model.

[0081] In a possible implementation, Figure 3 As shown, the training module 250 is specifically used to adjust the temporal attention layer in the key frame generation module according to the following steps:

[0082] Get an input video sequence containing N frames;

[0083] Initializing the initial text generation video diffusion model, wherein the initialization includes loading pre-trained weights and configuring model parameters;

[0084] For a first video frame corresponding to any target moment in the input video sequence, the first video frame is sampled by the initialized initial text generation video diffusion model to obtain a noise frame latent vector, and the difference between the noise frame latent vector of the first video frame and the noise frame latent vector of the second video frame is determined as the frame residual vector of the target moment; the second video frame is a video frame that is a preset frame interval number away from the first video frame;

[0085] Determining a true motion vector based on the frame residual vector;

[0086] estimating a denoised motion vector based on the denoised video vector and the frame residual vector;

[0087] The temporal attention layer in the key frame generation module is adjusted according to the loss value between the true motion vector and the denoised motion vector.

[0088] In the embodiment of the present application, the inversion latent variable is calculated from the input video; the inversion latent variable is input into the key frame generation module after the fine-tuning of the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information; the key frame sequence is input into the temporal interpolation module to obtain an extended frame sequence; the extended frame sequence is input into the spatial super-resolution module to obtain the target Cantonese opera video. In this way, the customized motion video generation is realized by the key frame generation module in the temporally fine-tuned text generation video diffusion model, which can accurately reproduce the motion of the target video and generate a variety of visual changes.

[0089] Based on the same application concept, see Figure 4 As shown, it is a structural diagram of an electronic device 400 provided in an embodiment of the present application, including: a processor 410, a memory 420 and a bus 430, the memory 420 stores machine-readable instructions executable by the processor 410, when the electronic device 400 is running, the processor 410 and the memory 420 communicate through the bus 430, and the machine-readable instructions are executed by the processor 410 when running to perform the steps of the method for generating Cantonese opera video from text as described in any of the above embodiments.

[0090] Specifically, when the machine-readable instructions are executed by the processor 410, the following processing may be performed:

[0091] Compute the inverse latent variable from the input video;

[0092] Inputting the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information;

[0093] Inputting the key frame sequence into the time interpolation module to obtain an extended frame sequence;

[0094] The extended frame sequence is input into the spatial super-resolution module to obtain a target Cantonese opera video.

[0095] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating Cantonese opera video from text provided in the above embodiment are executed.

[0096] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned method of generating Cantonese opera video from text, and realize customized motion video generation through the key frame generation module in the time-fine-tuned text-generated video diffusion model. It can accurately reproduce the motion of the target video and generate a variety of visual changes.

[0097] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0101] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0102] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0103] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a Cantonese opera video from text, characterized in that: The method for generating a Cantonese opera video from a trained target text is applied. The target text-generated video diffusion model includes a key frame generation module, a time interpolation module, and a spatial super-resolution module. The method for generating a Cantonese opera video from a text includes: Compute the inverse latent variable from the input video; Inputting the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information; Inputting the key frame sequence into the time interpolation module to obtain an extended frame sequence; The extended frame sequence is input into the spatial super-resolution module to obtain a target Cantonese opera video.

2. The method for generating Cantonese opera video from text according to claim 1, characterized in that: The target text generation video diffusion model is generated according to the following steps: Based on the motion distillation target of the residual vector, the temporal attention layer in the key frame generation module is adjusted, and the spatial super-resolution module and the temporal interpolation module are kept in a frozen state to generate a trained target text generation video diffusion model.

3. The method for generating Cantonese opera video from text according to claim 2, characterized in that: The motion distillation target based on the residual vector adjusts the temporal attention layer in the key frame generation module, including: Get an input video sequence containing N frames; Initializing the initial text generation video diffusion model, wherein the initialization includes loading pre-trained weights and configuring model parameters; For a first video frame corresponding to any target moment in the input video sequence, the first video frame is sampled by the initialized initial text generation video diffusion model to obtain a noise frame latent vector, and the difference between the noise frame latent vector of the first video frame and the noise frame latent vector of the second video frame is determined as the frame residual vector of the target moment; the second video frame is a video frame that is a preset frame interval number away from the first video frame; Determining a true motion vector based on the frame residual vector; estimating a denoised motion vector based on the denoised video vector and the frame residual vector; The temporal attention layer in the key frame generation module is adjusted according to the loss value between the true motion vector and the denoised motion vector.

4. The method for generating Cantonese opera video from text according to claim 3, characterized in that: The formula of the true motion vector is: in, is the frame residual vector at the target time t, is the noise frame latent vector of the first video frame, is the noise frame potential vector of the second video frame, c is the preset frame interval number, n is the nth video frame, is the true motion vector, α i is the noise parameter preset at the i-th time step in the forward process of the video diffusion model, is the normal distribution function.

5. The method for generating Cantonese opera video from text according to claim 3, characterized in that: The formula for the denoised motion vector is: in, is the residual vector of the target frame at time t, is the noise frame latent vector of the first video frame, is the noise frame potential vector of the second video frame, c is the preset frame interval number, n is the nth video frame, is the denoised motion vector, The noise parameters pre-set for the forward process of the video diffusion model, is the noise residual vector of the forward noise addition at time t of the nth frame predicted by the model, is the noise vector of the forward noise addition at time t of the n+cth frame predicted by the model, The noise vector at time t of the forward noise addition of the nth frame predicted by the model.

6. The method for generating Cantonese opera video from text according to claim 3, characterized in that: The formula of the denoised video vector is: in, is the denoised video vector, is the video vector of the entire input N frames, The noise parameter pre-set for the forward process of the video diffusion model, ∈ θ is the noise vector predicted by the model, and N is the number of video frames in the input video sequence.

7. The method for generating Cantonese opera video from text according to claim 2, characterized in that: In the process of training the initial text-generated video diffusion model to obtain the target text-generated video diffusion model, the method for generating Cantonese opera videos from text also includes: Get multiple sample texts; For any sample text among the multiple sample texts, simplify the prompt words of the sample text to obtain a target text; Each of the target texts is used as a training text for training the initial text to generate a video diffusion model.

8. A device for generating Cantonese opera videos from text, characterized in that: The target text is applied to generate a video diffusion model after training, the target text generating video diffusion model includes a key frame generation module, a time interpolation module and a spatial super-resolution module, and the device for generating Cantonese opera video from text includes a calculation module, a first determination module, a second determination module and a third determination module; wherein: The calculation module is used to calculate the inversion latent variable from the input video; The first determination module is used to input the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information; The second determination module is used to input the key frame sequence into the time interpolation module to obtain an extended frame sequence; The third determination module is used to input the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for generating Cantonese opera video from text as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating Cantonese opera video from text as described in any one of claims 1 to 7 are executed.

Images