Method, device, electronic device and medium for generating Cantonese opera video from text

Abstract

The present application relates to the fields of computer vision and artificial intelligence technology, and in particular to a method, device, electronic device, and medium for generating Cantonese opera videos from text. The present application calculates an inverted latent variable from an input video; inputs the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information; inputs the key frame sequence into the temporal interpolation module to obtain an extended frame sequence; and inputs the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video. In this way, customized motion video generation is achieved through the key frame generation module in the temporally fine-tuned text-generated video diffusion model, which can accurately reproduce the motion of the target video and generate diverse visual changes.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and artificial intelligence technology, and in particular to a method, device, electronic device, and medium for generating Cantonese opera videos from text. Background Art

[0002] Text-to-video (T2V) diffusion models have been a research hotspot in the field of artificial intelligence in recent years. They aim to generate corresponding video content from text descriptions. Current T2V diffusion models have made significant progress in generating high-quality videos. However, these models still face numerous challenges in generating videos with customized motion, particularly in accurately reproducing the target video's motion and generating diverse visual variations.

[0003] In the prior art, simple extensions of static image customization methods often result in a complex entanglement of appearance and motion data, preventing high-quality motion customization. Furthermore, current models lack the ability to reproduce specific motions or generate diverse variants of the same motion with different visual attributes and contexts.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] In view of this, the embodiments of the present application at least provide a method, device, electronic device and medium for generating Cantonese opera videos from text, which can realize customized motion video generation, accurately reproduce the motion of the target video, and generate diverse visual changes.

[0006] This application mainly includes the following aspects:

[0007] In a first aspect, an embodiment of the present application provides a method for generating Cantonese opera videos from text, which is applied to a trained target text-generated video diffusion model. The target text-generated video diffusion model includes a keyframe generation module, a temporal interpolation module, and a spatial super-resolution module. The method for generating Cantonese opera videos from text includes:

[0008] Compute the inverse latent variable from the input video;

[0009] Inputting the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information;

[0010] Inputting the key frame sequence into the time interpolation module to obtain an extended frame sequence;

[0011] The extended frame sequence is input into the spatial super-resolution module to obtain a target Cantonese opera video.

[0012] In a second aspect, an embodiment of the present application further provides a device for generating Cantonese opera videos from text, which is applied to a trained target text-generated video diffusion model. The target text-generated video diffusion model includes a key frame generation module, a temporal interpolation module, and a spatial super-resolution module. The device for generating Cantonese opera videos from text includes a calculation module, a first determination module, a second determination module, and a third determination module; wherein:

[0013] The calculation module is used to calculate the inversion latent variable from the input video;

[0014] The first determination module is configured to input the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information;

[0015] The second determining module is configured to input the key frame sequence into the time interpolation module to obtain an extended frame sequence;

[0016] The third determination module is configured to input the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video.

[0017] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for generating Cantonese opera video from text as described in the first aspect above.

[0018] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating Cantonese opera video from text as described in the first aspect above are executed.

[0019] The embodiments of the present application provide a method, device, electronic device, and medium for generating Cantonese opera videos from text. The method involves calculating an inverted latent variable from an input video; inputting the inverted latent variable into a keyframe generation module that has undergone a fine-tuned temporal attention layer to obtain a keyframe sequence; the keyframe sequence includes motion information; inputting the keyframe sequence into a temporal interpolation module to obtain an extended frame sequence; and inputting the extended frame sequence into a spatial super-resolution module to obtain a target Cantonese opera video. In this way, customized motion video generation is achieved through the keyframe generation module in a temporally fine-tuned text-to-video diffusion model, which accurately reproduces the motion of the target video and generates diverse visual changes.

[0020] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of a method for generating a Cantonese opera video from text provided in an embodiment of the present application is shown;

[0023] Figure 2 One of the functional module diagrams of a device for generating Cantonese opera videos from text provided in an embodiment of the present application is shown;

[0024] Figure 3 A second functional module diagram of a method for generating a Cantonese opera video from text provided in an embodiment of the present application is shown;

[0025] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0027] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0028] In order to enable those skilled in the art to use the contents of this application, the following implementation method is given in combination with the specific application scenario of "generating Cantonese opera video from text". For those skilled in the art, the general principles defined here can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0029] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario where text needs to be generated into Cantonese opera videos. The embodiments of the present application are not limited to specific application scenarios. Any scheme using the method and device for generating Cantonese opera videos from text provided by the embodiments of the present application is within the scope of protection of this application.

[0030] It is worth noting that prior to this application, simple extensions of static image customization methods in related art often resulted in a complex entanglement of appearance and motion data, making it impossible to achieve high-quality motion customization. Furthermore, current models lack the ability to reproduce specific motions or generate diverse variants of the same motion with different visual attributes and contexts.

[0031] To address the above issues, the present embodiment calculates an inverted latent variable from the input video; inputs the inverted latent variable into the keyframe generation module after fine-tuning the temporal attention layer to obtain a keyframe sequence; the keyframe sequence includes motion information; the keyframe sequence is input into the temporal interpolation module to obtain an extended frame sequence; and the extended frame sequence is input into the spatial super-resolution module to obtain the target Cantonese opera video. In this way, customized motion video generation is achieved through the keyframe generation module in the temporally fine-tuned text-generated video diffusion model, which can accurately reproduce the motion of the target video and generate diverse visual changes.

[0032] To facilitate understanding of the present application, the technical solutions provided in the present application are described in detail below in conjunction with specific embodiments.

[0033] Figure 1 This is a flow chart of a method for generating Cantonese opera videos from text provided in an embodiment of the present application. Figure 1As shown, the method for generating a Cantonese opera video from text provided in the embodiment of the present application includes the following steps:

[0034] S101: Calculate the inversion latent variable from the input video.

[0035] S102: Inputting the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information.

[0036] S103: Input the key frame sequence into the time interpolation module to obtain an extended frame sequence.

[0037] S104: Input the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video.

[0038] In the specific implementation, this application proposes a video motion customization (VMC) framework, which realizes customized video generation by adjusting the temporal attention layer of the key frame generation module in the text generation video diffusion model. Specifically, the trained target text generation video diffusion model is obtained by training the initial text generation video diffusion model, that is, the key frame generation module in the initial text generation video diffusion model is fine-tuned in time to obtain the target text generation video diffusion model. In the inference stage, the inverse latent variables are first calculated from the input video through DDIM inversion. Subsequently, these inverse latent variables are input into the time-fine-tuned key frame generation module to generate a short and low-resolution key frame sequence. The key frame sequence contains the main motion information; then, the key frame sequence is temporally expanded using the temporal interpolation module to generate more frames to improve the smoothness of the video. Specifically, the temporal interpolation module generates new frames between key frames through interpolation methods to smooth video playback. The expanded frames (extended frame sequence) are then spatially enhanced by the spatial super-resolution module to generate a high-resolution video output. Among them, the spatial super-resolution module uses deep learning technology to amplify and refine low-resolution frames to improve the visual quality of the video.

[0039] Exemplarily, after training is completed, the customized keyframe generation module is used to perform object motion-driven video generation in a new appearance context (e.g., “two actors performing traditional Cantonese opera”).

[0040] In an embodiment of the present application, an inverted latent variable is calculated from an input video; the inverted latent variable is input into the keyframe generation module after a fine-tuned temporal attention layer to obtain a keyframe sequence; the keyframe sequence includes motion information; the keyframe sequence is input into the temporal interpolation module to obtain an extended frame sequence; the extended frame sequence is input into the spatial super-resolution module to obtain a target Cantonese opera video. In this way, customized motion video generation is achieved through the keyframe generation module in the temporally fine-tuned text-generated video diffusion model, which can accurately reproduce the motion of the target video and generate diverse visual changes.

[0041] In one possible implementation, the target text generation video diffusion model is generated according to the following steps: based on the motion distillation target of the residual vector, the temporal attention layer in the key frame generation module is adjusted, and the spatial super-resolution module and the temporal interpolation module are kept in a frozen state to generate the trained target text generation video diffusion model.

[0042] In practice, this application proposes a unique method for adjusting only the temporal attention layer in the keyframe generation module. This method achieves efficient motion video customization by optimizing the temporal attention layer while keeping the spatial attention layer and other modules (such as the spatial super-resolution module and the temporal interpolation module) unchanged. This adjustment strategy ensures efficient and rapid model training, reduces the consumption of computing resources, and enables the training process to be completed under limited hardware conditions.

[0043] It should be noted that using the above framework, this application can achieve efficient motion-customized video generation. By optimizing the temporal attention layer, the target text generation video diffusion model can accurately capture and reproduce the motion trajectory in the input video. Compared with related technologies, this application has significant advantages in motion reproduction and visual diversity. The generated video maintains a consistent motion trajectory under different backgrounds and visual attributes, while ensuring high resolution and high quality.

[0044] Furthermore, the innovative adjustment strategy employed by this application reduces the computational complexity of model training and significantly reduces hardware resource requirements. Adjusting only the temporal attention layer allows the training process to be completed with limited computing resources. This efficient training method allows the model to be trained in a relatively short time, meeting the need for rapid generation of high-quality videos.

[0045] Here, through the technical solution of this application, video generation technology has achieved an important breakthrough, significantly improving the effect of customized video generation, and providing users with an efficient, accurate and personalized motion video generation solution.

[0046] In one possible implementation, the motion distillation target based on the residual vector adjusts the temporal attention layer in the keyframe generation module, including the following steps:

[0047] Step a, obtaining an input video sequence containing N frames;

[0048] Step b, initializing the initial text generation video diffusion model, wherein the initialization includes loading pre-trained weights and configuring model parameters;

[0049] Step c: for a first video frame corresponding to any target moment in the input video sequence, sampling the first video frame using the initialized initial text generation video diffusion model to obtain a noise frame latent vector, and determining the difference between the noise frame latent vector of the first video frame and the noise frame latent vector of a second video frame as the frame residual vector at the target moment; the second video frame is a video frame that is a preset number of frame intervals away from the first video frame;

[0050] Step d, determining a true motion vector based on the frame residual vector;

[0051] Step e, estimating a denoised motion vector based on the denoised video vector and the frame residual vector;

[0052] Step f, adjusting the temporal attention layer in the key frame generation module according to the loss value between the true motion vector and the denoised motion vector.

[0053] In the specific implementation, this application introduces a motion distillation target based on residual vectors, which is used as a motion reference by adjusting the frame residual vectors between consecutive noise potential frames. The specific technical steps are as follows: (1) Input video acquisition. Specifically, an input video sequence containing N frames is obtained from the user, and each frame is represented as v n, and preprocess the input video sequence to ensure the continuity and consistency of the video frame sequence for subsequent processing; (2) Initialization of the initial text generation video diffusion model. Specifically, the pre-trained initial text generation video diffusion model T2V is used as the basic model. The initialization of the basic model includes loading the pre-trained weights and configuring the model parameters to adapt to the current task. In addition, the temporal attention layer of the basic model is initialized to ensure that it has basic video generation capabilities and can be effectively adjusted in subsequent training; (3) the frame residual vector is calculated. Specifically, the noise frame latent vector is obtained by sampling the first video frame through the initialized initial text generation video diffusion model, and the difference between the noise frame latent vector of the first video frame and the noise frame latent vector of the second video frame is determined as the frame residual vector (also called noise motion vector) at the target moment. The frame residual vector is used to represent the motion change between consecutive frames in the video; wherein, the motion trajectory in the video is extracted by calculating the frame residual vector between consecutive frames. This process is achieved by calculating each pair of consecutive frames one by one to ensure the accurate extraction of motion information; (4) the temporal attention layer is adjusted. Specifically, the noise motion vector and the denoised video vector are obtained to obtain the true motion vector and the denoised motion vector. The temporal attention layer in the key frame generation module is adjusted according to the loss value between the true motion vector and the denoised motion vector.

[0054] During model training, only the temporal attention layer in the keyframe generation module is adjusted, while other modules (such as the spatial super-resolution module and the temporal interpolation module) remain frozen. This reduces computational complexity and improves training efficiency. Using a backpropagation algorithm, the parameters of the temporal attention layer in the keyframe generation module are continuously optimized based on the motion distillation objective, enabling it to better capture and reproduce the motion trajectory of the input video.

[0055] Here, the proposed video motion customization framework, the proposed video motion customization (VMC) framework, extracts motion trajectories from the residuals between noisy latent variables, i.e., noisy motion vectors. Specifically, the temporal attention layer of the keyframe generation module is fine-tuned by simply aligning the ground-truth and predicted motion vectors.

[0056] It should be noted that this application introduces a motion distillation target based on residual vectors, which extracts and reproduces the motion trajectory in the target video by calculating the frame residual vectors between consecutive noise potential frames. Residual cosine similarity can be used as a loss function to minimize the cosine similarity between the true motion vector and the predicted motion vector (denoised motion vector), ensuring that the model can accurately capture and reproduce the motion information in the target video. This method improves the accuracy of motion information extraction and reproduction by accurately modeling the motion changes of consecutive frames.

[0057] In a possible implementation, the formula for the true motion vector is: in, is the frame residual vector at the target time t, is the noise frame latent vector of the first video frame, is the noise frame potential vector of the second video frame, c is the preset frame interval number, n is the nth video frame, is the true motion vector, α i is the noise parameter preset at the i-th time step in the forward process of the video diffusion model, is the normal distribution function.

[0058] Here, set Indicates from The sampled The potential value of the nth noise frame is defined as the frame residual vector (also called noise motion vector) at time t: Wherein c represents a fixed frame interval, that is, a preset number of frame intervals.

[0059] In addition, the initial text generation video diffusion model is used to obtain the noise motion vectors between the noise latent frames from the input video sequence. The noise motion vector (frame residual vector at time t) can be expressed as:

[0060]

[0061] in, It is a normal distribution with mean zero and variance 2I (I represents the identity matrix) and can be obtained by the following diffusion kernel

[0062]

[0063] In a possible implementation, the formula for the denoised motion vector is: in, is the residual vector of the target frame at time t, is the noise frame latent vector of the first video frame, is the noise frame potential vector of the second video frame, c is the preset frame interval number, n is the nth video frame, is the denoised motion vector, i.e., an estimate calculated from the model’s predicted noise vector, The noise parameters pre-set for the forward process of the video diffusion model, is the noise residual vector of the forward noise-added time t of the nth frame predicted by the model, is the noise vector of the forward noise added at time t for the n+cth frame predicted by the model, The noise vector at time t of the forward noise addition of the nth frame predicted by the model.

[0064] In a possible implementation, the formula for the denoised video vector is:

[0065] in, is the denoised video vector, is the video vector of the entire input N frames, The noise parameter pre-set for the forward process of the video diffusion model, ∈ θ is the noise vector predicted by the model, and N is the number of video frames in the input video sequence.

[0066] In a specific implementation, according to the denoised video vector Calculate denoised motion vectors in

[0067] Among them, the Tweedie formula is applied to calculate the denoised video vector The formula is as follows: in is the empirical Bayes optimal posterior expectation The Tweedie formula is a formula used in Bayesian statistics to study the mean problem of the normal distribution. It has important applications in interval estimation, hypothesis testing, and constructing shrinkage estimators.

[0068] Here, the definition of the motion distillation objective includes: designing a new loss function to optimize the temporal attention layer so that it can accurately capture the motion trajectory in the target video. Specifically, the temporal attention layer of the model is adjusted by aligning the true motion vector and the predicted motion vector. The specific formula is:

[0069] Among them, l align represents the alignment loss function, and They represent the true motion vector and the predicted motion vector (i.e., the denoised motion vector), respectively.

[0070] For example, the cosine similarity loss function is selected to measure the alignment degree between the true motion vector and the denoised motion vector. The optimization framework is finally defined as follows: In other words, the proposed optimization framework aims to maximize and Therefore, this optimization method can be seamlessly applied to video diffusion models trained using ε-matching.

[0071] In a possible implementation, in the process of training the initial text-generated video diffusion model to obtain the target text-generated video diffusion model, the method of generating Cantonese opera video from text also includes: obtaining multiple sample texts; for any of the multiple sample texts, simplifying the prompt words of the sample text to obtain a target text; and using each of the target texts as a training text for training the initial text-generated video diffusion model.

[0072] In a specific implementation, simplified prompt words are used during the training process to remove background and appearance information and reduce the interference of irrelevant information on motion learning. For example, complex descriptive prompt words are simplified to prompt words that only contain motion information. For another example, the descriptive prompt word "drama actors perform classic Cantonese opera on stage" is simplified to "actors perform Cantonese opera" to focus on the extraction of motion information. Simplifying prompt words is crucial for motion distillation. This method of simplifying prompt words helps the model focus on learning motion information and avoids the influence of appearance and background changes on motion trajectories, thereby improving the quality of customized video generation. Through the processing of prompt words with unchanged appearance, the present application can generate personalized and diversified video content. This method not only improves the quality of the generated video, but also enhances the personalization and diversity of the generated video to meet the customization needs of different users.

[0073] Based on the same application concept, the embodiments of the present application also provide a device for generating Cantonese opera videos from text corresponding to the method for generating Cantonese opera videos from text provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the method for generating Cantonese opera videos from text in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0074] like Figure 2 、 Figure 3 As shown, Figure 2 This is one of the functional module diagrams of a device 200 for generating Cantonese opera videos from text provided in an embodiment of the present application. Figure 3 This is a second functional module diagram of a device 200 for generating a Cantonese opera video from text provided in an embodiment of the present application. Figure 2 As shown, the apparatus 200 for generating Cantonese opera videos from text is applied to a trained target text-generated video diffusion model, the target text-generated video diffusion model including a key frame generation module, a time interpolation module, and a spatial super-resolution module. The apparatus 200 for generating Cantonese opera videos from text includes a calculation module 210, a first determination module 220, a second determination module 230, and a third determination module 240; wherein:

[0075] The calculation module 210 is used to calculate the inversion latent variable from the input video;

[0076] The first determination module 220 is configured to input the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information;

[0077] The second determining module 230 is configured to input the key frame sequence into the time interpolation module to obtain an extended frame sequence;

[0078] The third determination module 240 is configured to input the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video.

[0079] In one possible implementation, Figure 3 As shown, the apparatus 200 for generating Cantonese opera videos from text further includes a training module 250; the training module 250 is configured to generate the target text-to-video diffusion model according to the following steps:

[0080] Based on the motion distillation target of the residual vector, the temporal attention layer in the key frame generation module is adjusted, and the spatial super-resolution module and the temporal interpolation module are kept in a frozen state to generate a trained target text generation video diffusion model.

[0081] In one possible implementation, Figure 3 As shown, the training module 250 is specifically used to adjust the temporal attention layer in the key frame generation module according to the following steps:

[0082] Get an input video sequence containing N frames;

[0083] Initializing the initial text generation video diffusion model, wherein the initialization includes loading pre-trained weights and configuring model parameters;

[0084] For a first video frame corresponding to any target moment in the input video sequence, sampling the first video frame using the initialized initial text generation video diffusion model to obtain a noise frame latent vector, and determining the difference between the noise frame latent vector of the first video frame and the noise frame latent vector of a second video frame as the frame residual vector at the target moment; the second video frame is a video frame that is a preset number of frame intervals away from the first video frame;

[0085] Determining a true motion vector based on the frame residual vector;

[0086] estimating a denoised motion vector based on the denoised video vector and the frame residual vector;

[0087] The temporal attention layer in the key frame generation module is adjusted according to the loss value between the true motion vector and the denoised motion vector.

[0088] In an embodiment of the present application, an inverted latent variable is calculated from an input video; the inverted latent variable is input into the keyframe generation module after a fine-tuned temporal attention layer to obtain a keyframe sequence; the keyframe sequence includes motion information; the keyframe sequence is input into the temporal interpolation module to obtain an extended frame sequence; the extended frame sequence is input into the spatial super-resolution module to obtain a target Cantonese opera video. In this way, customized motion video generation is achieved through the keyframe generation module in the temporally fine-tuned text-generated video diffusion model, which can accurately reproduce the motion of the target video and generate diverse visual changes.

[0089] Based on the same application concept, see Figure 4 As shown, it is a structural diagram of an electronic device 400 provided in an embodiment of the present application, including: a processor 410, a memory 420 and a bus 430, wherein the memory 420 stores machine-readable instructions executable by the processor 410, and when the electronic device 400 is running, the processor 410 and the memory 420 communicate with each other through the bus 430, and the machine-readable instructions are executed by the processor 410 when running to perform the steps of the method for generating Cantonese opera video from text as described in any of the above embodiments.

[0090] Specifically, when the machine-readable instructions are executed by the processor 410, the following processing may be performed:

[0091] Compute the inverse latent variable from the input video;

[0092] Inputting the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information;

[0093] Inputting the key frame sequence into the time interpolation module to obtain an extended frame sequence;

[0094] The extended frame sequence is input into the spatial super-resolution module to obtain a target Cantonese opera video.

[0095] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the method for generating Cantonese opera video from text provided in the above embodiment are executed.

[0096] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned method of generating Cantonese opera video from text, and realize customized motion video generation through the key frame generation module in the time-fine-tuned text generation video diffusion model, which can accurately reproduce the motion of the target video and generate diverse visual changes.

[0097] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment and will not be repeated here.

[0098] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0099] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0100] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0101] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0102] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0103] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for generating Cantonese opera videos from text, characterized in that: A video diffusion model is applied to a trained target text. The target text generation video diffusion model includes a key frame generation module, a time interpolation module, and a spatial super-resolution module. The method for generating a Cantonese opera video from text includes: Compute the inverse latent variable from the input video; Inputting the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information; Inputting the key frame sequence into the time interpolation module to obtain an extended frame sequence; The extended frame sequence is input into the spatial super-resolution module to obtain a target Cantonese opera video.

2. The method for generating Cantonese opera video from text according to claim 1, characterized in that: The target text generation video diffusion model is generated according to the following steps: Based on the motion distillation target of the residual vector, the temporal attention layer in the key frame generation module is adjusted, and the spatial super-resolution module and the temporal interpolation module are kept in a frozen state to generate a trained target text generation video diffusion model.

3. The method for generating Cantonese opera video from text according to claim 2, characterized in that: The motion distillation target based on the residual vector adjusts the temporal attention layer in the key frame generation module, including: Get an input video sequence containing N frames; Initializing the initial text generation video diffusion model, wherein the initialization includes loading pre-trained weights and configuring model parameters; For a first video frame corresponding to any target moment in the input video sequence, sampling the first video frame using the initialized initial text generation video diffusion model to obtain a noise frame latent vector, and determining the difference between the noise frame latent vector of the first video frame and the noise frame latent vector of a second video frame as the frame residual vector at the target moment; the second video frame is a video frame that is a preset number of frame intervals away from the first video frame; Determining a true motion vector based on the frame residual vector; estimating a denoised motion vector based on the denoised video vector and the frame residual vector; The temporal attention layer in the key frame generation module is adjusted according to the loss value between the true motion vector and the denoised motion vector.

4. The method for generating Cantonese opera video from text according to claim 3, characterized in that: The formula of the true motion vector is: in, is the frame residual vector at the target time t, is the noise frame latent vector of the first video frame, is the noise frame potential vector of the second video frame, c is the preset frame interval number, n is the nth video frame, is the true motion vector, α i is the noise parameter preset at the i-th time step in the forward process of the video diffusion model, is the normal distribution function.

5. The method for generating Cantonese opera video from text according to claim 3, characterized in that: The formula for the denoised motion vector is: in, is the residual vector of the target frame at time t, is the noise frame latent vector of the first video frame, is the noise frame potential vector of the second video frame, c is the preset frame interval number, n is the nth video frame, is the denoised motion vector, The noise parameters pre-set for the forward process of the video diffusion model, is the noise residual vector of the forward noise-added time t of the nth frame predicted by the model, is the noise vector of the forward noise added at time t for the n+cth frame predicted by the model, The noise vector at time t of the forward noise addition of the nth frame predicted by the model.

6. The method for generating Cantonese opera video from text according to claim 3, characterized in that: The formula for the denoised video vector is: in, is the denoised video vector, is the video vector of the entire input N frames, The noise parameter pre-set for the forward process of the video diffusion model, ∈ θ is the noise vector predicted by the model, and N is the number of video frames in the input video sequence.

7. The method for generating Cantonese opera video from text according to claim 2, characterized in that: In the process of training the initial text-generated video diffusion model to obtain the target text-generated video diffusion model, the method for generating Cantonese opera videos from text further includes: Get multiple sample texts; For any of the sample texts among the multiple sample texts, simplify the prompt words of the sample text to obtain a target text; Each of the target texts is used as a training text for training the initial text to generate a video diffusion model.

8. A device for generating Cantonese opera videos from text, characterized in that: A video diffusion model is applied to a trained target text, the target text generation video diffusion model includes a key frame generation module, a time interpolation module and a spatial super-resolution module, and the device for generating Cantonese opera video from text includes a calculation module, a first determination module, a second determination module and a third determination module; wherein: The calculation module is used to calculate the inversion latent variable from the input video; The first determination module is configured to input the inverted latent variable into the key frame generation module after fine-tuning the temporal attention layer to obtain a key frame sequence; the key frame sequence includes motion information; The second determining module is configured to input the key frame sequence into the time interpolation module to obtain an extended frame sequence; The third determination module is configured to input the extended frame sequence into the spatial super-resolution module to obtain a target Cantonese opera video.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus. When the processor is running, the machine-readable instructions execute the steps of the method for generating Cantonese opera video from text as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for generating Cantonese opera videos from text as claimed in any one of claims 1 to 7.

Images