Video generation method and apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a video generation method and apparatus, a device, and a medium. The method comprises: acquiring a first prompt text and a second prompt text, the first prompt text being used for describing a target object to be generated, and the second prompt text being used for describing an object action to be performed by the target object; on the basis of the first prompt text, using a preset first network model to generate a target image comprising the target object; and on the basis of the second prompt text and the target image, using a preset second network model to generate a target video comprising the target object, wherein the target video presents a dynamic picture of the target object performing the target action.

Description

Video generation methods, apparatus, equipment and media

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410841774.0, filed on June 26, 2024, entitled “Video Generation Method, Apparatus, Device and Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of multimedia processing technology, and in particular to a video generation method, apparatus, device, and medium. Background Technology

[0004] With the development of multimedia technology, more and more scenarios require engaging or entertaining videos to enhance the user's visual experience. For example, in live streaming, a virtual gift chosen by a user and displayed as an animated video on all viewers' screens can be generated. In social scenarios, users can also send each other entertaining animated videos. Summary of the Invention

[0005] This disclosure provides a video generation method, apparatus, device, and medium.

[0006] This disclosure provides a video generation method, the method comprising: acquiring a first prompt text and a second prompt text; the first prompt text describing a target object to be generated, and the second prompt text describing an object action to be performed by the target object; generating a target image containing the target object based on the first prompt text using a preset first network model; and generating a target video containing the target object based on the second prompt text and the target image using a preset second network model; wherein the target video presents a dynamic scene of the target object performing the object action.

[0007] Optionally, the second network model includes multiple generator networks, and the multiple generator networks are arranged in parallel; the second prompt text is used as input to each of the generator networks, and the target image is used as input to a specific network among the multiple generator networks.

[0008] Optionally, the second network model further includes a text encoder; the step of generating a target video containing the target object based on the second prompt text and the target image using a preset second network model includes: encoding the second prompt text using the text encoder to obtain prompt text features, and acquiring multiple noise maps; the number of noise maps is consistent with the number of non-specified networks in the plurality of generator networks; inputting the prompt text features into each of the generator networks, inputting the target image into the specified network, and inputting the multiple noise maps into the non-specified networks in the plurality of generator networks respectively; generating their respective output images through the plurality of generator networks based on their respective input information; wherein, the output images contain the target object, and the action form of the target object presented in different output images is different; and obtaining a target video containing the target object based on the parallel arrangement order of the plurality of generator networks and the respective output images of the plurality of generator networks.

[0009] Optionally, generating output images based on input information by the multiple generator networks includes: generating output images by the multiple generator networks using a temporal attention mechanism based on their respective input information, target association information, and parallel arrangement order; wherein the target association information includes target features output by the same specified network layer corresponding to each of the multiple generator networks.

[0010] Optionally, the target object in the output image corresponding to the specified network has a first action form, which is the action form presented by the target object in the target image; the target object in the output image corresponding to the non-specified network has a second action form, and the closer the sorting distance between the non-specified network and the specified network is, the closer the second action form corresponding to the non-specified network is to the first action form.

[0011] Optionally, the multiple generation networks have the same structure and share parameters.

[0012] Optionally, the method further includes: performing special effects processing on the target video to obtain a special effects video.

[0013] This disclosure also provides a video generation apparatus, comprising: an acquisition module for acquiring a first prompt text and a second prompt text; the first prompt text describing a target object to be generated, and the second prompt text describing an object action to be performed by the target object; an image generation module for generating a target image containing the target object based on the first prompt text and using a preset first network model; and a video generation module for generating a target video containing the target object based on the second prompt text and the target image and using a preset second network model; wherein the target video presents a dynamic scene of the target object performing the object action.

[0014] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the video generation method provided in this disclosure.

[0015] This disclosure also provides a computer-readable storage medium storing a computer program for performing the video generation method provided in this disclosure.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 is a flowchart illustrating a video generation method provided in an embodiment of this disclosure;

[0020] Figure 2 is a schematic diagram of the structure of a target network model provided in an embodiment of this disclosure;

[0021] Figure 3 is a schematic diagram of the structure of a generative network provided in an embodiment of this disclosure;

[0022] Figure 4 is a schematic diagram illustrating the principle of a temporal attention mechanism provided in an embodiment of this disclosure;

[0023] Figure 5 is a schematic diagram of a video generation device provided in an embodiment of this disclosure;

[0024] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] As mentioned above, with the development of multimedia technology, more and more scenarios require engaging or entertaining videos to enhance the user's visual experience. However, most existing technologies require professionals to create animated videos, which is costly and inefficient. Using artificial intelligence to generate animated videos offers less controllability.

[0026] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0027] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0028] Figure 1 is a flowchart illustrating a video generation method according to an embodiment of this disclosure. This method can be executed by a video generation device, which can be implemented using software and / or hardware and is generally integrated into an electronic device. As shown in Figure 1, the method mainly includes the following steps S102 to S106:

[0029] Step S102: Obtain the first prompt text and the second prompt text; the first prompt text is used to describe the target object to be generated, and the second prompt text is used to describe the object action to be performed on the target object.

[0030] The first prompt text, i.e., the description information of the target object, may include not only information about the target object itself, such as type and style, but also any information associated with the target object, such as the environment in which the target object is located. This embodiment of the disclosure does not limit the target object; for example, the target object can be a person, animal, plant, vehicle, robot, etc. Furthermore, this embodiment of the disclosure does not limit the object's actions; for example, the object's actions can be the actions of a flower from bud to full bloom, from walking to running, from opening its eyes to closing them, or continuous rotation, etc.

[0031] Step S104: Based on the first prompt text, a target image containing the target object is generated using a preset first network model. This embodiment of the disclosure does not limit the structure of the first network model. Exemplarily, the first network model is a generative model, which can be trained based on a generative model such as a text-to-image model. This first network model can generate an image of a target object with a specific style based on the first prompt text.

[0032] Step S106: Based on the second prompt text and the target image, a target video containing the target object is generated using a preset second network model; wherein, the target video presents a dynamic scene of the target object performing an object action. For example, if the target object is a rose, and the second prompt text indicates that the flower is blooming, then the target video presents a dynamic scene of the rose blooming.

[0033] The embodiments disclosed herein do not limit the structure of the second network model. The second network model can be further improved based on the first network model. For example, the first network model contains a generator network, while the second network model contains multiple generator networks in parallel. In order to achieve better video generation effect, specific functional modules such as a temporal attention module can be added to the generator network. With the help of this functional module, the second network model generates multi-frame images with the motion gradient effect of the target object based on the temporal attention mechanism, thereby forming the target video.

[0034] The technical solutions provided in this disclosure eliminate the need for professional animation video production, significantly reducing the cost and efficiency of animation video creation. Furthermore, the two-stage approach—first generating a target image based on a first prompt text, then generating a video based on a second prompt text and the target image—offers greater controllability and effectively ensures the generated animation video meets expectations, thus better satisfying user needs. In other words, one or more embodiments of this disclosure can significantly reduce the cost and efficiency of animation video creation, while also providing strong controllability in the video generation process, effectively ensuring the generated animation video meets expectations, thereby better satisfying user needs.

[0035] To facilitate understanding, this disclosure first describes how the first network model is obtained. For example, a basic generative model can be pre-trained to provide basic text-to-image generation capabilities. Then, sample data containing the target object is used to fine-tune the model, enabling it to generate images containing the target object based on its descriptive text. In practical applications, specific styles of the target object can be defined using text prompts, allowing the model to generate images with those styles. Specifically, aesthetically pleasing sample images (containing the target object) can be collected first, followed by text annotation. The annotated sample images are then used to train a basic diffusion model. In practical applications, the first network model in this disclosure can be trained by combining the basic diffusion model with a LoRA (Low-Rank Adaptation) model. For example, if the target object is a flower, the first network model can generate flower images. Specifically, parameters of the base diffusion model can be tuned based on LoRA technology. LoRA is a lightweight model fine-tuning technique that allows for customized fine-tuning of a large base model using a small amount of sample data with a specific style, thereby generating a generative model with specific object generation capabilities. During the training phase, the sample images can be noise-added, converting them into noise of the same dimension. Specifically, random Gaussian numbers between 0 and 1 are added to the image to obtain a noise map. This noise map and labeled text are then input into the model to be trained, enabling the model to reconstruct the original image.

[0036] During the inference phase, noise maps can be generated based on Gaussian random numbers ranging from 0 to 1. Different object description prompts and noise maps are input into the trained first network model to generate different object images. In practical applications, other prompts and the LoRA model can also be combined to enrich the object image effects output by the first network model; no restrictions are imposed here.

[0037] Based on the target image corresponding to the first prompt text generated by the first network model, the target video can be further generated by the second network model for the target image and the second prompt text. For example, the second network model is a further improvement on the first network model. The training method of the second network model can refer to the first network model, which will not be repeated here. The following mainly explains the structure and working principle of the second network model.

[0038] In some implementations, the second network model includes multiple generator networks arranged in parallel. Second prompt text serves as input to each generator network, and the target image serves as input to a designated network among the generator networks. This disclosure does not restrict the position of the designated network among the generator networks; for example, the first or last generator network can be used as the designated network, or a generator network from the middle can be selected as the designated network. Furthermore, in practical applications, only one first prompt text can be set, or two or more first prompt texts can be set to describe target objects in different states (such as flower buds, half-open flowers, etc.). The number of generated images is consistent with the number of first prompt texts. In this case, two or more generator networks can be selected as the designated network, and this is not limited here.

[0039] In some specific implementation examples, multiple generative networks have the same structure and share parameters, which facilitates training and information exchange, and allows for better control over the matching degree of the output image. The inputs of each generative network also include noise maps; the noise maps of different generative networks are different, and the noise maps corresponding to each generative network can be randomly generated using relevant techniques, which will not be elaborated here. Furthermore, the second network model also includes a text encoder. Step S106, which is the step of generating a target video containing the target object based on the second prompt text and the target image using the preset second network model, can be performed as follows: Steps A to C:

[0040] Step A: Encode the second prompt text using a text encoder to obtain prompt text features and acquire multiple noise maps; the number of noise maps is consistent with the number of non-specified networks in the multiple generator networks, which are generator networks other than the specified networks.

[0041] Step B involves inputting the cue text features into each generator network, inputting the target image into a designated network, and inputting multiple noise maps into non-designated networks within the generator networks. The generator networks then generate their respective output images based on their input information. Each output image contains the target object, and the target object's movement is presented differently in different output images. In other words, the input information for the non-designated networks is the cue text features and noise maps, while the input information for the designated networks is the cue text features and the target image.

[0042] In some specific implementation examples, multiple generator networks generate their respective output images based on their corresponding input information, target association information, and parallel arrangement order, using a temporal attention mechanism. The target association information includes target features output by the same specified network layer corresponding to each of the multiple generator networks. In practical applications, the same specified network layer corresponding to each of the multiple generator networks can refer to a specific network layer within the same Unet unit of the multiple generator networks. Features at each position in the feature information output by the specified network layer, or features at a specified position, can all serve as target features.

[0043] For ease of understanding, a schematic diagram of a generative network structure can be referenced in Figure 3. The diagram illustrates that the generative network includes an encoder, a denoising network, and a decoder. The input to the encoder is a noise map, and the output of the decoder is an image. It should be noted that Figure 3 illustrates a non-specified generative network; for a specified generative network, the target image can be used instead of the noise map. Figure 3 also shows that the output of the text encoder (i.e., the cue text features) can be injected into each Unet unit, specifically into the cross-attention layer within the Unet unit. Figure 3 uses the Unet network as an example for denoising, comprising multiple Unet units (Unet Blocks). The difference from a conventional Unet network lies in the insertion of a temporal attention module. This module primarily processes information based on a temporal attention mechanism. Specifically, it utilizes the temporal attention mechanism to achieve information interaction between N generative networks. This allows each generative network to determine its own generated features based on the feature information obtained from other generative networks, thereby ensuring that the output images of the N generative networks can achieve a gradual animation effect of the target object's movement, such as the dynamic effect of a flower from bud to full bloom. As shown in Figure 3, in N parallel generative networks, temporal attention modules at the same position can perform temporal attention processing based on their respective target association information and the order of the target networks. Taking target association information 1 as an example, it includes the feature information output by the same specified network layer of the first module (the first Unet unit and / or the first temporal attention module) of the N generative networks. The positions of different temporal attention modules are different, and the corresponding target association information is different. In practical applications, temporal attention modules can be implemented using relevant technologies, such as the Animate diff module, which can be integrated into models such as generative networks and learn motion priors from video datasets, thereby extending the text-generated image model into a video generation model.

[0044] Specifically, the temporal attention mechanism allows the generative network in the target network model to adaptively focus on temporal information when processing N sequence data, capturing dependencies and contextual information in the sequence data, and assigning different weights to features with the same position in the N sequence data. Referring to the schematic diagram of a temporal attention mechanism shown in Figure 4, for each generative network, the temporal attention mechanism can obtain the target features output by the same specified network layer of each of the N generative networks. For example, the features with the same position in the output features of the specified network layer of the i-th Unet unit of each of the N generative networks can be assigned corresponding weights and weighted fusion processing can be performed. In Figure 4, the output features of the specified network layer of the i-th Unet unit are used as an example. Figure 4 uses gray grids to indicate the features with the same position in the output features of the specified network layer of the i-th Unet unit of different generative networks, which are the aforementioned target features. In practical applications, each feature in the 3*3 features shown in Figure 4 can be used as the target feature in turn, and combined with the corresponding features of other generative networks for weighted fusion processing, so as to obtain the feature corresponding to the corresponding position of the generative network based on the weighted fusion features, thereby ensuring the rationality of the output image of the generative network. For example, the output image of generator network N is a flower in full bloom, and the output images of generator networks 1 to N-1 are intermediate states of the flower from the bud to the full bloom state, thus presenting the growth and blooming effect of the flower from the bud to the full bloom.

[0045] Step C: Based on the parallel arrangement of multiple generator networks and the output images of each generator network, a target video containing the target object is obtained.

[0046] Each generator network has a corresponding sequence number. Based on the sequence number of each generator network, the parallel arrangement order of multiple generator networks can be determined. Then, based on the output images of each of the multiple generator networks, the target video can be obtained by sorting them sequentially according to their sequence numbers.

[0047] In some implementations, the target object in the output image corresponding to a specified network has a first action form, which is the action form presented by the target object in the target image; the target object in the output image corresponding to a non-specified network has a second action form. As the sorting distance between the non-specified network and the specified network becomes closer, the second action form corresponding to the non-specified network becomes closer to the first action form. For example, if the target object is a flower, the specified network is the last generator network N, and the second prompt text is used to indicate the flower's movement from bud to full bloom, and the first action form is the flower's state in full bloom, then the second action form is the flower's intermediate state from bud to full bloom. For example, the flower in the output image of generator network 1 is in a bud state, the flower in the output image of generator network 10 is in a half-open state, and the flower in the output image of generator network N-1 is in a nearly full bloom state, thus presenting an animated video of the flower from bud to full bloom.

[0048] It should be noted that the video generation method provided in this embodiment has two significant differences from existing video generation models: (1) It generates video in a two-stage manner, that is, it first generates a target image based on the first prompt text, and then generates a video based on the second prompt text and the target image. Compared with existing text-generated video models, this method has stronger controllability and can effectively ensure that the target object and corresponding actions in the generated animation video meet the requirements. (2) In multiple generation networks, the input of a specified network is the target image, rather than a noise image, which helps to achieve a better freeze-frame effect and further enhances the controllability of the generation of the animation video of the target object. Specifically, in existing technologies, even if multiple generative models exist, the input to each network is a noisy image. However, the embodiments of this disclosure can replace the noisy input image of a specified network with the target image. This not only utilizes a temporal attention mechanism to pass the feature information of the target image to other generative networks, ensuring that these networks can generate animation content with strong continuity with the target image based on its feature information and the order of the networks, but also allows for the use of the target image to achieve a freeze-frame effect in a specified frame of the target video. For example, if the specified network is the last generative network, the last frame of the target video will present the visual appearance of the target image. It should be noted that the last frame of the target video is the output image after processing the target image by the specified network, and this output image is essentially identical to the target image.

[0049] In some embodiments, the video generation method provided in this disclosure further includes: applying special effects processing to the target video to obtain a special effects video. This special effects processing includes, but is not limited to, speed adjustment and masking, and may specifically depend on the application scenario and user needs of the target video; therefore, no limitation is placed on the special effects processing method herein.

[0050] This disclosure does not limit the application scenario of the target video, such as live streaming or social media scenarios. Taking a live streaming scenario as an example, the target object can be flowers such as roses and peonies. Prompt text describing the flowers' blooming and rotating actions can be pre-set. Then, multiple animated videos of various flowers can be generated using a first network model and a second network model. These animated videos can serve as customized gift videos. When a viewer requests to send flowers, the video can be randomly downloaded from a large number of pre-made flower animation videos and played on the interfaces of all users in the target live streaming room. Different users may send different flower animation videos, thus presenting users with the feeling of sending a customized gift, enhancing the fun and appeal of the live stream. It is understood that if existing technology is used, professional personnel are required to create animated videos, which is costly and inefficient, making it difficult to create a large number of different animated videos to achieve personalized gift effects. However, the method provided by this disclosure can quickly and conveniently generate a large number of different animated videos, presenting users with personalized gift effects and better meeting user needs.

[0051] Corresponding to the aforementioned video generation method, this disclosure further provides a video generation device. Figure 5 is a schematic diagram of the structure of a video generation device provided in this disclosure. This device can be implemented by software and / or hardware, and is generally integrated into an electronic device. As shown in Figure 5, the video generation device includes:

[0052] The acquisition module 502 is used to acquire a first prompt text and a second prompt text; the first prompt text is used to describe the target object to be generated, and the second prompt text is used to describe the object action to be performed on the target object;

[0053] Image generation module 504 is used to generate a target image containing the target object based on the first prompt text and using a preset first network model.

[0054] The video generation module 506 is used to generate a target video containing the target object based on the second prompt text and the target image using a preset second network model; wherein the target video presents a dynamic picture of the target object performing an object action.

[0055] The apparatus provided in this disclosure eliminates the need for professionals to create animated videos, thereby significantly reducing the cost and efficiency of animation video production. Furthermore, the two-stage approach—first generating a target image based on the first prompt text, and then generating a video based on the second prompt text and the target image—offers greater controllability and effectively ensures that the generated animated video meets expectations, thus better satisfying user needs.

[0056] In some implementations, the second network model includes multiple generator networks arranged in parallel; the second prompt text is used as input to each generator network, and the target image is used as input to a specific network among the multiple generator networks.

[0057] In some embodiments, the second network model further includes a text encoder; the video generation module 506 is specifically used to: encode the second prompt text using the text encoder to obtain prompt text features, and acquire multiple noise maps; the number of noise maps is consistent with the number of non-specified networks in the plurality of generation networks; input the prompt text features into each of the generation networks, input the target image into the specified network, and input the multiple noise maps into the non-specified networks in the plurality of generation networks respectively; generate their respective output images through the plurality of generation networks based on their respective input information; wherein the output images contain the target object, and the action form of the target object presented in different output images is different; based on the parallel arrangement order of the plurality of generation networks and the respective output images of the plurality of generation networks, obtain a target video containing the target object.

[0058] In some implementations, the video generation module 506 is specifically used to: generate respective output images by the multiple generation networks based on their respective input information, target association information, and parallel arrangement order using a temporal attention mechanism; wherein, the target association information includes target features output by the same specified network layer corresponding to each of the multiple generation networks.

[0059] In some implementations, the target object in the output image corresponding to the designated network has a first action form, which is the action form presented by the target object in the target image; the target object in the output image corresponding to the non-designated network has a second action form, and the closer the sorting distance between the non-designated network and the designated network is, the closer the second action form corresponding to the non-designated network is to the first action form.

[0060] In some implementations, the plurality of generating networks have the same structure and share parameters.

[0061] In some embodiments, the apparatus further includes a special effects processing module for performing special effects processing on the target video to obtain a special effects video.

[0062] The video generation apparatus provided in this disclosure can execute the video generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0064] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0065] Referring now to FIG6, a schematic diagram of the structure of an electronic device 600 suitable for implementing embodiments of the present disclosure is shown. The terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG6 is merely an example and should not impose any limitation on the functionality and scope of use of embodiments of the present disclosure.

[0066] As shown in Figure 6, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0067] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG. 6 illustrates electronic device 600 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0068] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0069] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the image processing methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0070] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the video generation method provided in embodiments of this disclosure.

[0071] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0072] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the video generation method of this disclosure.

[0073] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0074] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0075] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0076] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0077] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, comprising: Get the first and second prompt texts; The first prompt text is used to describe the target object to be generated, and the second prompt text is used to describe the object action to be performed on the target object; Based on the first prompt text, a target image containing the target object is generated using a preset first network model; Based on the second prompt text and the target image, a target video containing the target object is generated using a preset second network model; wherein, the target video presents a dynamic scene of the target object performing the object action.

2. The method according to claim 1, wherein the second network model comprises a plurality of generative networks, and the plurality of generative networks are arranged in parallel; the second prompt text is used as input to each of the generative networks, and the target image is used as input to a designated network among the plurality of generative networks.

3. The method according to claim 2, wherein the second network model further includes a text encoder; the step of generating a target video containing the target object using a preset second network model based on the second prompt text and the target image includes: The second prompt text is encoded using the text encoder to obtain prompt text features and multiple noise images are acquired. The number of noise maps is consistent with the number of non-specified networks in the plurality of generated networks; The prompt text features are input into each of the generating networks, the target image is input into the designated network, and the multiple noise images are input into the non-designated networks among the multiple generating networks respectively; the multiple generating networks generate their respective output images based on their respective input information; wherein, the output image contains the target object, and the action form of the target object presented in different output images is different; Based on the parallel arrangement of the multiple generator networks and the output images of each generator network, a target video containing the target object is obtained.

4. The method according to claim 3, wherein generating respective output images through the plurality of generation networks based on their respective input information comprises: The multiple generator networks generate their respective output images using a temporal attention mechanism based on their respective input information, target association information, and parallel arrangement order; wherein, the target association information includes target features output by the same specified network layer corresponding to each of the multiple generator networks.

5. The method according to claim 3, wherein the target object in the output image corresponding to the specified network has a first action form, and the first action form is the action form presented by the target object in the target image; The target object in the output image corresponding to the non-specified network has a second action form. As the sorting distance between the non-specified network and the specified network becomes closer, the second action form corresponding to the non-specified network becomes closer to the first action form.

6. The method according to claim 2, wherein the plurality of generating networks have the same structure and the plurality of generating networks share parameters.

7. The method according to claim 1, wherein the method further comprises: The target video is processed with special effects to obtain a special effects video.

8. A video generation apparatus, comprising: The acquisition module is used to acquire the first prompt text and the second prompt text; The first prompt text is used to describe the target object to be generated, and the second prompt text is used to describe the object action to be performed on the target object; The image generation module is used to generate a target image containing the target object based on the first prompt text and using a preset first network model; The video generation module is used to generate a target video containing the target object based on the second prompt text and the target image using a preset second network model; wherein the target video presents a dynamic scene of the target object performing the object action.

9. An electronic device, wherein the electronic device comprises: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the video generation method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program for performing the video generation method according to any one of claims 1-7.

Images