Video generation method, video generation device and storage medium

Abstract

The invention provides a video generation method, a video generation device and a storage medium, and relates to the field of computers. The method comprises the following steps: encoding a prompt text to obtain a first feature, and encoding an unsafe content text to obtain a second feature; performing cyclic denoising to obtain denoised video features, including: according to the video feature of the current time step and the first feature, performing noise prediction by using global time sequence attention to obtain first noise; according to the video feature of the current time step and the second feature, the global time sequence attention and the local time sequence attention are utilized to determine the guiding intensity; according to the first noise, determining a second noise based on guidance of guidance intensity; denoising the video features of the current time step according to the second noise to obtain the video features of the next time step; and generating a video corresponding to the de-noised video features. And the original video generation quality is kept while unsafe contents in the time dimension are removed.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a video generation method, a video generation device, and a storage medium. Background Art

[0002] Diffusion models can generate realistic and imaginative videos based on text descriptions. Video generation techniques based on diffusion models are widely used in various fields, including filmmaking, virtual reality, and social media.

[0003] Video generation techniques based on diffusion models have some security issues. For example, they may generate videos that are pornographic, violent, or violate copyright or privacy, which poses certain security risks and severely limits their reliability and practical applications. Therefore, ensuring their security is particularly important.

[0004] Compared to images, videos introduce a temporal dimension, containing richer information. This temporal dimension allows unsafe content to appear or be hidden at any point in the video. For example, the initial frames of a video may appear safe, but over time, the security of the content may gradually deteriorate. Therefore, improving the security of video-generated content faces a key challenge: removing unsafe content in the temporal dimension while maintaining the quality of the original video and ensuring that the generated video meets user needs.

[0005] Some related technologies remove unsafe content from videos by adjusting the parameters of the video generation model. However, model parameter adjustment is highly dependent on the design of the original architecture, limiting its generalization and versatility. For example, modifying the gradient of classifier guidance may not be applicable to model architectures without classifier guidance. Furthermore, unified intervention is required from the perspective of overall video security. For example, applying a uniform guidance strength to the entire video sequence. Insufficient guidance strength may allow unsafe content to continue to exist at certain times, while excessive guidance strength may interfere with low-risk periods and reduce the quality of video generation. Summary of the Invention

[0006] The video generation solution proposed in this disclosure utilizes a dual attention path: global temporal attention and local temporal attention, focusing on unsafe content in the overall and local temporal sequences, respectively. This approach dynamically adjusts guidance intensity, strengthening safety constraints during high-risk periods and avoiding excessive intervention during low-risk periods. This approach maintains the original video generation quality while removing unsafe content in the temporal dimension. Furthermore, this solution eliminates the need to adjust the parameters of the video generation model and is applicable to a variety of video generation models, offering enhanced versatility.

[0007] Some embodiments of the present disclosure propose a video generation method, including: encoding a prompt text to obtain a first feature, encoding an unsafe content text to obtain a second feature; performing cyclic denoising to obtain denoised video features, including: using global temporal attention to predict noise based on the video features and the first feature of the current time step to obtain the first noise; using global temporal attention and local temporal attention to determine the guidance strength based on the video features and the second feature of the current time step; determining the second noise based on the guidance of the guidance strength based on the first noise; denoising the video features of the current time step based on the second noise to obtain the video features of the next time step; and generating a video corresponding to the denoised video features.

[0008] In some embodiments, using global temporal attention and local temporal attention to determine the guidance strength includes: using global temporal attention to perform noise prediction based on the video features and second features of the current time step to obtain a third noise; using local temporal attention to perform noise prediction based on the video features and second features of the current time step to obtain a fourth noise; and determining the guidance strength based on the difference between the third noise and the fourth noise.

[0009] In some embodiments, the global temporal attention is determined based on the query vector and key vector provided by the video features; or the local temporal attention is determined by adding a temporal mask to the global temporal attention.

[0010] In some embodiments, determining the guidance strength based on the difference between the third noise and the fourth noise includes: determining the degree of change of the noise in the timing dimension based on the difference between the third noise and the fourth noise; and adjusting the original guidance strength using the degree of change of the noise in the timing dimension to determine the guidance strength.

[0011] In some embodiments, determining the degree of change of the noise in the time series dimension includes: normalizing the difference between the third noise and the fourth noise to obtain the degree of change of the noise in the time series dimension.

[0012] In some embodiments, determining the second noise based on the guidance of the guidance intensity includes: guiding the difference between the first noise and the third noise based on the guidance intensity; and determining the accumulation result of the first noise and the guidance result as the second noise.

[0013] In some embodiments, denoising the video features of the current time step based on the second noise includes: adjusting the second noise using a noise level parameter; and determining the video features of the next time step based on a difference between the video features of the current time step and the adjusted second noise.

[0014] In some embodiments, the initial object of the cyclic denoising is Gaussian noise.

[0015] In some embodiments, generating a video corresponding to the denoised video features includes: mapping the denoised video features back to the original video space through three-dimensional video decoding to obtain a video corresponding to the prompt text with unsafe content removed.

[0016] Some embodiments of the present disclosure provide a video generating device, comprising: one or more modules for executing a video generating method.

[0017] Some embodiments of the present disclosure provide a video generating device, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a video generating method based on instructions stored in the memory.

[0018] Some embodiments of the present disclosure provide a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, a video generation method is implemented.

[0019] Some embodiments of the present disclosure provide a computer program product, comprising computer instructions, which implement a video generation method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The following briefly introduces the drawings required for describing the embodiments or related technologies. The present disclosure can be more clearly understood based on the following detailed description with reference to the drawings.

[0021] Obviously, the drawings described below are only some embodiments of the present disclosure. A person skilled in the art can derive other drawings based on these drawings without inventive effort.

[0022] Figure 1 A schematic diagram illustrating a video generating device according to some embodiments of the present disclosure.

[0023] Figure 2 A schematic diagram illustrating a noise estimator (timing-aware guidance module) according to some embodiments of the present disclosure is shown.

[0024] Figure 3 A schematic diagram illustrating a video generation method according to some embodiments of the present disclosure.

[0025] Figure 4 Schematic diagrams showing video generating devices according to other embodiments of the present disclosure.

[0026] Figure 5 Schematic diagram showing a video generating device according to some further embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0028] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0029] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0030] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0031] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0032] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0033] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0034] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0035] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0036] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0037] In addition, in order to avoid obscuring the present disclosure with unnecessary details, only the processing steps and / or device structures that are closely related to at least the solution according to the present disclosure are shown in the drawings, while other details that are not closely related to the present disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it does not need to be discussed again for subsequent drawings.

[0038] Video generation technology based on diffusion models is primarily divided into two processes: forward and backward. In the forward process, a pre-trained 3D encoder compresses the training video into video features in a latent space. Gaussian noise is gradually added to the video features at each time step. In the backward process, a deep neural network is typically used as a noise estimator, using text features as conditions to predict the added noise at each time step. The noise estimator is trained by minimizing the difference between the estimated noise and the actual noise. During the inference process, the noise estimator uses text features as conditions to gradually predict noise from the Gaussian noise distribution and performs denoising to ultimately obtain video features. The video features are then passed through a 3D decoder to produce the final video.

[0039] The disclosed embodiment proposes a security video generation solution based on temporal perception guidance, which seamlessly integrates the temporal perception guidance module into the diffusion model without the need to retrain or adjust the diffusion model parameters, thereby forming a unified framework for security video generation.

[0040] like Figure 1As shown, a video generation device (a safe video generation device based on temporal-aware guidance) includes a text encoder, a noise estimator (including a temporal-aware guidance module), and a 3D decoder. Initially, the text encoder encodes the user-entered prompt text and unsafe content text, respectively, to obtain corresponding first features (prompt features) and second features (unsafe content features). The unsafe content text may, for example, record a collection of unsafe content, including one or more unsafe content, typically multiple unsafe content. The unsafe content can be pre-set. Subsequently, using the global attention path (global temporal attention) and local attention path (local temporal attention) constructed in the noise estimator by the temporal-aware guidance module, recurrent denoising (e.g., T rounds of denoising) is performed to obtain denoised video features. The global temporal attention focuses on constraints on unsafe content in the overall time series, while the local temporal attention focuses on constraints on unsafe content in local (short-term) time series. Noise Estimator (Time-Aware Guidance Module): Based on the video features of the current time step and the first feature, global temporal attention is used to predict noise, resulting in the first noise. Based on the video features of the current time step and the second feature, the guidance strength is determined using the difference between the global and local temporal noises determined by global and local temporal attention. Based on the guidance of the guidance strength, the second noise is determined based on the first noise. Subsequently, the video features of the current time step are denoised using the second noise to obtain the video features of the next time step. Through a recursive denoising process, the denoised video features are ultimately obtained. Finally, the 3D decoder generates a video corresponding to the denoised video features. That is, the denoised video features are mapped from the latent space back to the original video space to produce the final safe video (i.e., the video corresponding to the warning text of the unsafe content has been removed).

[0041] A safe video generation scheme guided by temporal awareness utilizes a dual attention path during the denoising process: global temporal attention and local temporal attention focus on unsafe content in the overall and local temporal sequences, respectively. By quantifying the temporal differences between these two, the dynamic characteristics of unsafe content are determined. Based on these characteristics, the guidance intensity is dynamically adjusted to strengthen safety constraints during high-risk periods and avoid excessive intervention during low-risk periods. This approach maintains the original video generation quality while removing unsafe content in the temporal dimension. Furthermore, this scheme does not require adjustment of video generation model parameters and is applicable to a variety of video generation models, offering enhanced versatility.

[0042] For example, the text encoder can be a model based on the Transformer architecture, such as the CLIP (Contrastive Language-Image Pretraining) model and the T5 (Text-to-Text Transfer Transformer) model. and unsafe content text , respectively, through the text encoder Get the corresponding first feature and the second feature ,in and Respectively represent the length of the prompt text and the unsafe content text, is the feature dimension, for example, 4096. If the length of the prompt text is 16 and the length of the unsafe content text is 5, then =16 and =5.

[0043] In the inference process of the diffusion model, Gaussian noise is used (The initial object of the cyclic denoising) is used as the starting point, and the denoising operation is performed step by step. After one step of denoising, the video feature with a certain degree of noise is obtained. Then this video feature will be used as the input of the next denoising. The noise estimator predicts the noise and removes this part of the noise from the video feature to obtain a video feature with a smaller degree of noise. This video feature will be used as the input of the next denoising and the cyclic denoising is performed until the last step of denoising is completed, and the video feature without noise is obtained. .

[0044] like Figure 2 As shown, the noise estimator The temporal-aware guidance module includes a global temporal attention branch, a partial temporal attention branch, a guidance strength component, and a second noise component (the final noise at the current time step). The noise estimator can be a deep learning model, such as a Transformer model.

[0045] Global temporal attention is determined based on the query vector and key vector provided by the video features, and the formula is expressed as: ,in Represent the query vector and key vector provided by the video features, respectively, where the key vector With value vector Correspondingly, is the query vector The feature dimension of , T represents the matrix transpose, Represents the softmax function.

[0046] Local temporal attention is determined by adding a temporal mask to the global temporal attention, and the formula is expressed as: , Indicates the timing mask. In the frame-by-frame processing, the timing mask formula is expressed as: ,in, and Indicates the frame index, Make sure each frame Only follow Frames within the window, Represents a large negative number. For example, if m=1, the video features of each frame can only be temporally attention calculated with the video features of the previous and next frames.

[0047] First feature and the second feature Passed to the noise estimator (Time-Aware Guidance Module). At each time step , noise estimator (Time-series perception guidance module): Based on the video features of the current time step t and the first feature , use global temporal attention to perform noise prediction and obtain the first noise, that is, predict the noise corresponding to the original video under the global temporal sequence ; According to the video features of the current time step t and the second feature , using global temporal attention to perform noise prediction, we get the third noise, that is, the noise corresponding to the unsafe content under the global temporal prediction ; According to the video features of the current time step t and the second feature , using local temporal attention to perform noise prediction, we get the fourth noise, that is, the noise corresponding to the unsafe content under the local temporal prediction ; (Guidance strength part) According to the difference between the third noise and the fourth noise, the guidance strength (also called time series guidance strength or guidance coefficient) is determined. Specifically, the L2 norm (L2 norm) is used to calculate and The difference is expressed as: Subsequently, the degree of noise variation in the time series dimension is determined based on the difference between the third noise and the fourth noise. For example, the difference between the third noise and the fourth noise is normalized to obtain the degree of noise variation in the time series dimension. The formula is expressed as: ,in and Represents the noise difference The minimum and maximum values ​​in the time series dimension. The original guidance strength is adjusted by using the degree of change of noise in the time series dimension to determine the guidance strength (also known as the time series-aware guidance strength). For example, according to the normalized difference , determine the guidance strength of timing perception ,in is the original guidance strength of the model, which is a fixed value in time series. It changes dynamically in the temporal dimension and can impose fine-grained constraints on unsafe content in the video. For example, The value is 6.5, but is not limited to the example given.

[0048] Second noise part: According to the first noise, based on the guidance of the guidance intensity, the final noise (second noise) is determined. Specifically, based on the guidance intensity , for the first noise With the third noise The difference between the first noise and boot results The accumulated result is determined as the second noise , the formula is expressed as .

[0049] Denoising the video features of the current time step according to the second noise includes: using the noise degree parameter For the second noise Make adjustments; based on the video features of the current time step The difference between the adjusted second noise and the video features of the next time step is determined The formula is: ,in is the time step The noise level parameter. After round-robin denoising, the denoised video features (video latent space representation) are obtained. .

[0050] 3D decoder Generate the corresponding video after denoising, that is, the video feature after denoising Mapping from the latent space back to the original video space to generate the final secure video is formulated as For example, a 3D decoder consists of multiple layers of 3D convolutional layers and overlapping upsampling operations. The input of the 3D encoder is a video feature with small length, width, and time dimensions, such as length 90, width 60, and time dimension 13; the output of the 3D encoder is a normal video with larger length, width, and time dimensions than the video feature, such as length 720, width 480, and time dimension 49.

[0051] This paper provides an end-to-end solution to the problem of generating videos containing unsafe content (such as pornography, violence, and copyright infringement). By constructing a unified framework consisting of a text encoder, noise estimator, temporal-aware guidance module, and 3D decoder, it significantly reduces security risks in video generation and provides strong support for the secure application of video generation technology.

[0052] To address the difficulty of balancing security and quality in generated videos, we propose a temporal-aware guidance method that eliminates the need for model retraining or parameter adjustment. By establishing local and global temporal attention paths during the denoising process, we can enforce security constraints during high-risk periods while avoiding excessive intervention during low-risk periods, significantly improving the security of generated videos without compromising quality.

[0053] To address the problem that video generation often relies on specific model architectures and is difficult to generalize to other architectures, we propose a universal, training-independent framework. This framework seamlessly integrates a temporal-aware guidance module into the diffusion model, making it applicable to a variety of video generation models and significantly improving the versatility and applicability of video generation solutions.

[0054] The following combination Figure 3 Describe the video generation method (Secure Video Generation Method Based on Temporal Perception Guidance).

[0055] In step 310, the prompt text is encoded to obtain a first feature, and the unsafe content text is encoded to obtain a second feature.

[0056] For example, a text encoder is used to encode the prompt text to obtain a first feature, and to encode the unsafe content text to obtain a second feature. The text encoder can be a model based on a Transformer architecture, for example.

[0057] In step 320, cyclic denoising is performed to obtain denoised video features. The cyclic denoising includes steps 321 to 324. The initial object of the cyclic denoising is Gaussian noise, such as a cyclic T-wheel.

[0058] In step 321, noise prediction is performed based on the video features and the first features at the current time step using global temporal attention to obtain the first noise, thereby predicting the noise corresponding to the original video under global temporal conditions.

[0059] In step 322, the guidance strength is determined based on the video features and the second features of the current time step using global temporal attention and local temporal attention.

[0060] Determining the guidance strength includes, for example:

[0061] (1) Based on the video features and the second features at the current time step, the global temporal attention is used to predict the noise and obtain the third noise. Thus, the noise corresponding to the unsafe content under the global temporal sequence is predicted.

[0062] As mentioned above, the global temporal attention is determined based on the query vector and key vector provided by the video features.

[0063] (2) Based on the video features and the second features of the current time step, the noise is predicted using the local temporal attention to obtain the fourth noise. Thus, the noise corresponding to the unsafe content under the local temporal sequence is predicted.

[0064] As mentioned above, local temporal attention is determined by adding a temporal mask to the global temporal attention.

[0065] (3) Determine the guidance strength based on the difference between the third noise and the fourth noise. Thus, the guidance strength of temporal perception is dynamically determined based on the difference between the two noises corresponding to the unsafe content in the global / local temporal dimension.

[0066] In some embodiments, the degree of noise variation in the time dimension is determined based on the difference between the third noise and the fourth noise. For example, the difference between the third noise and the fourth noise is normalized to obtain the degree of noise variation in the time dimension. The degree of noise variation in the time dimension is then used to adjust the original guidance intensity to determine the guidance intensity. Thus, the fixed guidance intensity is adjusted based on the degree of noise variation in the time dimension to determine the dynamic guidance intensity.

[0067] In step 323 , second noise is determined based on the first noise and the guidance of the guidance intensity.

[0068] In some embodiments, based on the guidance strength, the difference between the first noise and the third noise is guided, and the accumulation result of the first noise and the guidance result is determined as the second noise. Thus, based on the dynamic guidance strength, fine-grained constraints are imposed on unsafe content.

[0069] In step 324, the video features of the current time step are denoised according to the second noise to obtain the video features of the next time step.

[0070] In some embodiments, the second noise is adjusted using a noise level parameter; and the video feature of the next time step is determined based on the difference between the video feature of the current time step and the adjusted second noise.

[0071] In step 330 , a video corresponding to the denoised video features is generated.

[0072] Through 3D video decoding, the denoised video features are mapped back to the original video space to obtain a video corresponding to the warning text without the unsafe content. For example, the denoised video features are input into the 3D video decoder for mapping, and the output is a safe video corresponding to the warning text without the unsafe content.

[0073] The above embodiment, based on the dynamic guidance strength, imposes different degrees of security constraints on different risk periods, significantly reduces the security risks in video generation without reducing the quality of video generation, and ensures the consistency and security of the generated video in the time dimension.

[0074] Here are some application examples. For example, assume the input prompt text is "A dancer is dancing and suddenly falls and bleeds," and the unsafe content text is "Blood, war." The video generation device performs cyclic denoising on the initial Gaussian noise based on the prompt text and the unsafe content text. It then maps the denoised video features back to the original video space and generates a video corresponding to the prompt text without the unsafe content: a video of a dancer dancing and suddenly falling, minus the bleeding.

[0075] Figure 4 Schematic diagrams showing video generation devices according to other embodiments of the present disclosure. Figure 4 As shown, the video generating apparatus 400 of this embodiment includes one or more modules for executing the video generating method.

[0076] The text encoding module 410 is configured to encode the prompt text to obtain a first feature and encode the unsafe content text to obtain a second feature. The text encoding module 410 includes, for example, a text encoder.

[0077] The cyclic denoising module 420 is configured to perform cyclic denoising to obtain denoised video features, including: using global temporal attention to predict noise based on the video features and the first feature at the current time step to obtain the first noise; using global temporal attention and local temporal attention to determine the guidance strength based on the video features and the second feature at the current time step; determining the second noise based on the guidance of the guidance strength based on the first noise; and denoising the video features at the current time step based on the second noise to obtain the video features at the next time step. The cyclic denoising module 420, for example, includes a noise estimator, which includes a temporal-aware guidance module. The initial target of the cyclic denoising is Gaussian noise.

[0078] The cyclic denoising module 420 is configured to determine the guidance strength by: using global temporal attention to predict noise based on the video features and second features of the current time step to obtain a third noise; using local temporal attention to predict noise based on the video features and second features of the current time step to obtain a fourth noise; and determining the guidance strength based on the difference between the third and fourth noises. The difference between the third and fourth noises is used to determine the degree of noise variation in the temporal dimension, for example, by normalizing the difference between the third and fourth noises to obtain the degree of noise variation in the temporal dimension; and adjusting the original guidance strength based on the degree of noise variation in the temporal dimension to determine the guidance strength.

[0079] The cyclic denoising module 420 is configured to determine the second noise by: guiding the difference between the first noise and the third noise based on the guiding strength; and determining the accumulation result of the first noise and the guiding result as the second noise.

[0080] The cyclic denoising module 420 is configured to denoise the video features of the current time step according to the second noise, including: adjusting the second noise using a noise level parameter; and determining the video features of the next time step according to the difference between the video features of the current time step and the adjusted second noise.

[0081] Video generation module 430 is configured to generate a video corresponding to the denoised video features. Through 3D video decoding, the denoised video features are mapped back to the original video space to obtain a video corresponding to the warning text with the unsafe content removed. Video generation module 430 may include, for example, a 3D decoder.

[0082] Figure 5 Schematic diagram of a video generating device according to some further embodiments of the present disclosure is shown. Figure 5 As shown, the video generating apparatus 500 of this embodiment includes: a memory 510 and a processor 520 coupled to the memory 510 , and the processor 520 is configured to execute the video generating method of each embodiment based on the instructions stored in the memory 510 .

[0083] The video generating device 500 may further include an input / output interface 530 , a network interface 540 , a storage interface 550 , etc. These interfaces 530 , 540 , 550 , the memory 510 , and the processor 520 may be connected via a bus 560 , for example.

[0084] The memory 510 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0085] The processor 520 may be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, or discrete hardware components such as discrete gates or transistors.

[0086] The input / output interface 530 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touch screen. The network interface 540 provides a connection interface for various networked devices. The storage interface 550 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 560 can use any of a variety of bus structures. For example, bus structures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0087] The embodiments of the present disclosure provide a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the video generation method in each embodiment is implemented.

[0088] The embodiments of the present disclosure provide a computer program product, including computer instructions, which implement the video generation method in various embodiments when executed by a processor.

[0089] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more (non-transitory) computer-readable storage media (including but not limited to disk storage, CD-ROMs, optical storage, cloud storage, etc.) containing computer program code. A computer program product should be understood as a software product that primarily implements its solution through a computer program.

[0090] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

Claims

1. A video generation method, comprising: The prompt text is encoded to obtain a first feature, and the unsafe content text is encoded to obtain a second feature; Perform cyclic denoising to obtain denoised video features, including: According to the video features of the current time step and the first feature, global temporal attention is used to predict noise and obtain the first noise; According to the video features and second features of the current time step, the guidance strength is determined by using global temporal attention and local temporal attention; determining a second noise based on the first noise and the guidance of the guidance intensity; Denoising the video features of the current time step according to the second noise to obtain the video features of the next time step; Generate a video corresponding to the denoised video features.

2. The video generation method according to claim 1, wherein: Using global temporal attention and local temporal attention, determining the guidance strength includes: According to the video features and second features of the current time step, global temporal attention is used to predict noise and obtain the third noise; According to the video features and second features of the current time step, the noise is predicted using local temporal attention to obtain the fourth noise; The guidance strength is determined according to the difference between the third noise and the fourth noise.

3. The video generation method according to claim 2, wherein: The global temporal attention is determined based on the query vector and key vector provided by the video features; or The local temporal attention is determined by adding a temporal mask to the global temporal attention.

4. The video generation method according to claim 2, wherein: Determining the guidance strength according to the difference between the third noise and the fourth noise includes: Determine the degree of noise variation in the time series dimension based on the difference between the third noise and the fourth noise; The original guidance intensity is adjusted using the variation degree of the noise in the time series dimension to determine the guidance intensity.

5. The video generation method according to claim 4, wherein: Determining the degree of noise variation in the time series dimension includes: The difference between the third noise and the fourth noise is normalized to obtain the degree of noise variation in the time series dimension. The video generation method according to claim 2 , wherein: Based on the guidance of the guidance intensity, determining the second noise includes: guiding the difference between the first noise and the third noise based on the guiding strength; An accumulation result of the first noise and the guidance result is determined as the second noise.

7. The video generation method according to claim 1, wherein: Denoising the video features of the current time step according to the second noise includes: adjusting the second noise using a noise level parameter; The video feature of the next time step is determined according to the difference between the video feature of the current time step and the adjusted second noise.

8. The video generation method according to claim 1, wherein: The initial object of the cyclic denoising is Gaussian noise.

9. The video generation method according to claim 1, wherein: The corresponding videos for generating denoised video features include: By decoding the three-dimensional video, the denoised video features are mapped back to the original video space to obtain a video corresponding to the prompt text with unsafe content removed.

10. A video generation device, comprising: One or more modules for executing the video generation method according to any one of claims 1 to 9.

11. A video generating device, comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to execute the video generating method according to any one of claims 1 to 9 based on instructions stored in the memory.

12. A computer-readable storage medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the video generation method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer instructions, wherein when the computer instructions are executed by a processor, the video generation method according to any one of claims 1 to 9 is implemented.

Images