Dynamic view angle video generation method and device based on modular network

Abstract

The invention provides a dynamic view angle video generation method and device based on a modular network, relates to the technical field of artificial intelligence, and aims to generate a multi-scene dynamic view angle long video matched with a user instruction. The method comprises the steps that a user instruction is analyzed into a plurality of scenes through a large language model, each scene comprises scene description and a scene conversion instruction, and the scene conversion instruction represents a conversion mode between the scenes; selecting a modular scene converter according to the scene conversion instruction, wherein the modular scene converter is used for enabling a generated video to have a conversion mode corresponding to the scene conversion instruction; a control image and the scene description serve as control information, videos of all scenes are sequentially generated through a base video generator and the modular scene converter, and the control image is the last frame image of the video of the previous scene; and splicing the videos of the scenes to obtain a multi-scene dynamic view angle video matched with the user instruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a dynamic perspective video generation method and device based on a modular network. BACKGROUND

[0002] Text-to-Image diffusion models have attracted much attention due to their ability to generate high-fidelity and high-resolution images. Compared with traditional generative adversarial networks (GANs), diffusion models learn data distribution in a latent space and use a variational autoencoder (VAE) to map the latent space to the real space. Furthermore, combining a control encoder can achieve fine guidance of generated content. This technology has been widely used in design and personalized content creation, significantly reducing the threshold for image generation.

[0003] Although the application scenarios (such as film and television production, game development, and personalized short video generation) have great potential, text-to-video technology is still in its early stages. User instructions required by text-to-video models often involve multi-scene dynamic switching and perspective transformation (for example, starting from a long shot of a field under a blue sky, gradually focusing on a house in the distance, then gradually moving the camera to the left to reveal a large field, and the house moves out of the field). For such a complex user instruction, the model needs to consider the actual content of multiple scenes while taking into account the movement of the camera and the consistency of the content between multiple scenes.

[0004] However, related technologies cannot handle complex user instructions, and the generated videos are usually short and mostly contain only a single scene or a single perspective, and cannot finely control the movement of the camera. SUMMARY

[0005] In view of the above problems, the embodiments of the present application provide a dynamic perspective video generation method and device based on a modular network to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, the embodiments of the present application disclose a dynamic perspective video generation method based on a modular network, comprising: parsing a user instruction into multiple scenes by a large language model, each scene including a scene description and a scene transition instruction, the scene transition instruction representing a transition mode between scenes; selecting a modular scene transitioner according to the scene transition instruction, the modular scene transitioner being configured to generate a video with a transition mode corresponding to the scene transition instruction; generating videos of respective scenes in sequence by a base video generator and the modular scene converter, the control image being a last frame image of a video of a previous scene; stitching the videos of respective scenes to obtain a multi-scene dynamic perspective video matching the user instruction.

[0007] Optionally, the user instruction is parsed into multiple scenes by a large language model, including: inputting the user instruction into the large language model, and parsing the user instruction according to parsing control information to obtain multiple scenes; wherein the parsing control information includes scene extraction sequence control information and conversion mode identification control information, and the conversion mode is at least one of zooming in, zooming out, panning left, panning right, tilting up, and tilting down.

[0008] Optionally, the method further includes: inputting the control image into an adaptive control encoder for feature extraction to obtain control features, the control features being used to guide the base video generator and the modular scene converter to generate videos; generating videos of respective scenes in sequence by a base video generator and the modular scene converter, the control image being a last frame image of a video of a previous scene; inputting the randomly sampled Gaussian noise, the control features, and the scene description into the base video generator and the modular scene converter for multiple rounds of denoising iteration to obtain the video of the scene.

[0009] Optionally, the base video generator includes an encoder and a decoder, the encoder includes multiple first network layers and a time dimension self-attention mechanism layer after each first network layer, and the decoder includes multiple second network layers and a time dimension self-attention mechanism layer after each second network layer; each modular scene converter includes multiple scene conversion layers after each first network layer and each second network layer. For each round of denoising iteration, each time dimension self-attention mechanism layer of the base video generator passes a noise vector in the following manner: dimensionally transforming a first noise vector from a previous network layer to obtain a second noise vector, the previous network layer being a first network layer or a second network layer, and a dimension of the first noise vector being greater than a dimension of the second noise vector; inputting the second noise vector into the time dimension self-attention mechanism layer for self-attention processing in the time dimension to obtain a third noise vector; The third noise vector is dimensionally transformed to obtain a fourth noise vector, and the fourth noise vector is transmitted to a next network layer, which is a scene conversion layer, and the fourth noise vector has the same dimension as the first noise vector.

[0010] Optionally, the adaptive control encoder comprises a plurality of first network layers, a time dimension self-attention mechanism layer and a scene conversion layer after each first network layer, and the output of each time dimension self-attention mechanism layer and scene conversion layer is connected with a corresponding second network layer in the base video generator. The randomly sampled Gaussian noise, the control feature, and the scene description are input into the base video generator and the modular scene converter for multiple rounds of denoising iteration to obtain a video of the scene, comprising: In the first round of denoising iteration, the Gaussian noise, the control feature, and the scene description are input into the base video generator and the modular scene converter for denoising iteration to obtain a noise vector generated in the first round of denoising iteration. In the tth round of denoising iteration, the noise vector generated in the (t-1)th round of denoising iteration, the control feature, and the scene description are input into the base video generator and the modular scene converter for denoising iteration to obtain a noise vector generated in the tth round of denoising iteration, wherein t is an integer between 2 and N, and N is the total number of denoising iterations. According to the noise vector generated in the Nth round of denoising iteration, a video of the scene is generated.

[0011] Optionally, the method further comprises: Before the tth round of denoising iteration begins, a random number is obtained, and if the random number is less than a target value, a noise vector of a control image in the tth round of denoising iteration in the video generation process of the previous scene is obtained. The noise vector of the control image is used to replace the noise vector of the first frame in the noise vector generated in the (t-1)th round of denoising iteration to obtain a fused noise vector. In the tth round of denoising iteration, the noise vector generated in the (t-1)th round of denoising iteration, the control feature, and the scene description are input into the base video generator and the modular scene converter for denoising iteration to obtain a noise vector generated in the tth round of denoising iteration, comprising: In the tth round of denoising iteration, the fused noise vector, the control feature, and the scene description are input into the base video generator and the modular scene converter for denoising iteration to obtain a noise vector generated in the tth round of denoising iteration.

[0012] Optionally, the method further comprises: According to the variance and mean of the control image, the variance and mean of each frame image of the video of each scene are adjusted to obtain an adjusted video of each scene; The videos of each scene are spliced to obtain a multi-scene dynamic perspective video matched with the user instruction, including: The videos of each scene are spliced to obtain a multi-scene dynamic perspective video matched with the user instruction.

[0013] Optionally, the adaptive control encoder is trained according to the following steps: The first frame image of the first sample video is taken as a sample control image, and the adaptive control encoder is used to restore the sample video according to the sample control image to obtain a restored video; According to the difference between the restored video and the first sample video, the parameters of the adaptive control encoder are updated to obtain a trained adaptive control encoder.

[0014] Optionally, the modular scene converter is trained according to the following steps: For a target conversion mode, a second sample video with the corresponding conversion mode is simulated through a data enhancement method, and the target conversion mode is one of magnification, reduction, left translation, right translation, upward inclination and downward inclination; The second sample video is used to train the modular scene converter to obtain a trained modular scene converter.

[0015] In a second aspect of the embodiments of the application, a dynamic perspective video generation device based on a modular network is disclosed, and the device includes: The analysis module is configured to parse a user instruction into multiple scenes through a large language model, each scene including a scene description and a scene conversion instruction, and the scene conversion instruction representing a conversion mode between scenes; The selection module is configured to select a modular scene converter according to the scene conversion instruction, and the modular scene converter is configured to make the generated video have a conversion mode corresponding to the scene conversion instruction; The generation module is configured to take a control image and the scene description as control information, and sequentially generate videos of each scene through a base video generator and the modular scene converter, and the control image is a last frame image of a video of a previous scene; The splicing module is configured to splice the videos of each scene to obtain a multi-scene dynamic perspective video matched with the user instruction.

[0016] In a third aspect, the electronic device is disclosed, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the dynamic view video generation method based on the modular network according to the first aspect of the embodiments of the present application when executing the computer program.

[0017] The embodiments of the present application have the following advantages: In the embodiments of the present application, the user instruction is parsed into multiple scenes by a large language model, each scene including a scene description and a scene transition instruction, which realizes accurate understanding of the user instruction and accurate identification of the scene transition mode, so as to decompose the generation of the long video with multiple dynamic scenes into the generation of multiple scene videos. When generating the video of each scene, the control image and the scene description are used as control information to generate the video of each scene by the base video generator and the modular scene transitioner. Since the modular scene transitioner is selected according to the scene transition instruction, the generated video has the transition mode corresponding to the scene transition instruction through the modular scene transitioner, thereby realizing accurate control of the scene transition mode. In addition, the last frame image of the video of the previous scene is used as the control image, so that the videos of different scenes are connected at the head and tail, ensuring the coherence between the videos of different scenes. In this way, the long video with multiple dynamic scenes matching the user instruction is generated. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a step flowchart of a dynamic view video generation method based on a modular network provided by the embodiments of the present application; Figure 2 is a whole schematic diagram of a dynamic view video generation model provided by the embodiments of the present application; Figure 3 is a schematic diagram of a video generation module of a dynamic view video generation model provided by the embodiments of the present application; Figure 4 is a schematic diagram of a generated video of a dynamic view video generation method based on a modular network provided by the embodiments of the present application; Figure 5 is a schematic diagram of a generated video of another dynamic view video generation method based on a modular network provided by the embodiments of the present application; Figure 6is a structural schematic diagram of a dynamic view video generation device based on a modular network provided by an embodiment of the present application. Figure 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned purposes, features and advantages of the present application more apparent, clear and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative work are within the scope of protection of the present application.

[0021] In order to better understand the technical solutions of the present application, the related art will be described first.

[0022] The powerful text-to-image diffusion model has attracted widespread attention due to the high realism and resolution of the pictures it generates. Under its leadership, a series of related technologies based on diffusion models have emerged, making the pictures generated by artificial intelligence increasingly realistic, even to the extent that humans themselves cannot distinguish them. These technologies are also benefiting various industries, allowing everyone to turn their imagination into real pictures at a very small cost. For example, web or application designers no longer need to search the web for the pictures they need; ordinary people who do not know how to draw can use artificial intelligence technology to generate personalized and customized avatars or photos.

[0023] Unlike traditional picture generation models based on generative adversarial networks, diffusion models are a kind of latent variable model that first learns the distribution of real data in the latent space and then maps it from the latent space to the real space through a variational autoencoder. On the basis of the diffusion model, the concept of control is introduced, and with the help of the control encoder, the guidance of the diffusion model denoising process can be realized, thereby generating customized picture content.

[0024] Although the technology of text-to-image has been greatly developed, the field of text-to-video is just starting. Text-to-video is based on text-to-image, and its application scenarios are actually more extensive than text-to-image, that is, artificial intelligence can be used to generate videos containing virtual characters and scenes, which has great potential in the fields of movies, games and virtual reality; in addition, based on user data and preferences, artificial intelligence can generate personalized video content to meet the unique needs and interests of users, for example, short video platforms have become an essential application in people's lives, and people strongly need higher quality video generation. Further, if the content of the generated video can be finely controlled, the process of, for example, movie shooting, game production, etc. can be simplified, and the customization of the video can be greatly improved.

[0025] However, the user instructions that text-to-video models need to face are very complex and diverse, which may contain multiple scenes and dynamic view transformations. When facing such instructions, how can the model generate a video that meets the requirements? For example, "start with a long shot of a field under a blue sky, gradually zoom in on a house in the distance. Then move the camera to the left, and a large field appears, and the house moves out of the field", this instruction for video generation contains three scenes, respectively, the field and the blue sky, the house in the distance, and the large field. It also contains transition instructions (conversion instructions) between adjacent scenes, respectively, "zoom in" and "move left". For such a complex instruction, the model needs to consider the actual content of multiple scenes while taking into account the movement of the camera and the consistency of the content between multiple scenes.

[0026] To solve this problem, the following challenges need to be overcome: 1) user instructions are very diverse and complex, and multiple camera movements may be implicit rather than directly given, which requires the model to have very strong semantic understanding ability. 2) The movement of the camera in the video is difficult to control finely and smoothly. 3) In dynamic view videos, the coherence within a single scene and the consistency between multiple scenes are difficult to guarantee at the same time.

[0027] Considering that a video is actually a series of images, the generation of a video is essentially inseparable from the generation of images. In fact, many existing video generation works are indeed based on image generation. However, video generation is not simply generating multiple images, it has many difficulties: first, the generated multiple images cannot be random, but should have very high coherence. And video generation needs to consume more resources than image generation. If we calculate according to the fact that a 1-second video contains 16 images, even if we only generate a 10-second short video, we need to generate a total of 160 images.

[0028] Therefore, existing technologies typically do not directly train a video generation model. Instead, they add video generation modules or network layers to a pre-trained textural image diffusion model and then fine-tune it on a large video dataset. The main function of these video modules is to ensure the coherence of the generated multi-frame images. However, existing technologies mainly focus on short videos (primarily generating 16 or 24-frame clips), which contain only a single scene or a single shot, making it difficult to handle user commands containing multiple scenes. It should be noted that these short videos are different from the short videos people usually enjoy watching; they are often only 1-2 seconds long, more like a moving image, and contain relatively little information, resulting in very limited applications in real life. Furthermore, existing textural video generation work based on diffusion models cannot control the content of the generated video very precisely, relying solely on the "free play" of the diffusion model, let alone control the transition between shots across multiple scenes.

[0029] Therefore, existing technologies still have the following problems: they cannot handle complex user commands, the generated videos are usually short, and most contain only a single scene or a single perspective. Furthermore, they cannot precisely control camera movement, such as "zoom in" or "push left."

[0030] To overcome the limitations of related technologies, this application provides a method for generating dynamic perspective videos based on modular networks. This method parses user commands into scene descriptions and transition methods for each scene, thus decomposing the generation of a long video with multiple dynamic perspectives into the generation of multiple scene videos. For the generation of a video for a specific scene, the last frame of the video from the previous scene is used as the control image, combined with scene transition commands, to generate a smooth video. Finally, the videos from each scene are simply stitched together to obtain a long video with multiple dynamic perspectives that matches the user commands.

[0031] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a dynamic viewpoint video generation method based on a modular network, as provided in an embodiment of this application. Figure 1 As shown, the dynamic viewpoint video generation method based on modular networks provided in this application embodiment may include steps S110 to S140: Step S110: Parse the user command into multiple scenarios using a large language model. Each scenario includes a scenario description and a scenario conversion command, wherein the scenario conversion command represents the conversion method between scenarios.

[0032] The user instruction refers to a description of the video content expected to be generated by the user. The scene description and the transition mode between scenes in the user instruction can be implicit rather than directly given. For these implicit scene descriptions and transition modes, a single video generation model cannot read them, so a large language model is used to analyze the user instruction.

[0033] The transition mode between scenes can be understood as a lens transition mode or a lens motion mode. The transition mode between scenes includes zoom-in, zoom-out, left pan, right pan, upward tilt, and downward tilt.

[0034] Step S120: Select a modular scene converter according to the scene transition instruction, the modular scene converter being used to make the generated video have a transition mode corresponding to the scene transition instruction.

[0035] Each transition mode corresponds to a modular scene converter. For each scene, a corresponding modular scene converter is selected according to the scene transition instruction, which is used for video generation of the scene, so that the generated video has a transition mode corresponding to the scene transition instruction.

[0036] Since each transition mode corresponds to a modular scene converter, these modular scene converters can be freely combined to generate more complex lens motion, such as "zoom-in and left pan", thanks to the modular approach. Therefore, the transition mode between scenes corresponding to the scene transition instruction can include multiple, and the selected modular scene converter includes multiple. In this way, complex scene transition modes (lens motion) are decomposed into basic transition mode combinations to simulate any complex transition mode.

[0037] Step S130: Generate the video of each scene in sequence through the base video generator and the modular scene converter, taking the control image and the scene description as control information, the control image being the last frame image of the video of the previous scene.

[0038] The base video generator is based on a base diffusion model (Stable Diffusion model) and increases a time dimension self-attention mechanism layer (Transformer layer) between network layers to improve the inter-frame consistency of the video through the time dimension self-attention mechanism layer. The modular scene converter is embedded and deployed between each network layer of the base video generator. In this way, the modular scene converter is introduced to cooperate with the base video generator to generate a video, so that the generated video has a transition mode corresponding to the scene transition instruction, and the precise control of the scene transition mode is realized.

[0039] To ensure the continuity between videos of different scenes, when generating a video of a certain scene, the last frame image of the video of the previous scene is used as a control image, and the control image and the scene description are used as control information to guide the base video generator and the modular scene converter to generate the video, thereby obtaining the generated video of the scene.

[0040] It can be understood that for the video of the first scene, there is no video of the previous scene, and at this time there can be no control image, or a reference image provided by the user can be used as the control image.

[0041] Step S140: splicing the videos of the scenes to obtain a multi-scene dynamic perspective video matching the user instruction.

[0042] The multi-scene dynamic perspective video is spliced from the videos of multiple scenes, and each scene has a different conversion mode, so the video contains multiple scenes and different perspectives. Specifically, the videos of the scenes are spliced in the order of the multiple scenes in the user instruction, thereby obtaining a multi-scene dynamic perspective video matching the user instruction.

[0043] The technical solution of the sample application embodiment decomposes the generation of a multi-scene dynamic perspective long video into the generation of multiple scene videos by parsing the user instruction into multiple scenes through a large language model, each scene including a scene description and a scene conversion instruction, thereby accurately understanding the user instruction and accurately identifying the conversion mode between scenes. When generating a video of each scene, the control image and the scene description are used as control information to generate the video of each scene through a base video generator and a modular scene converter. Since the modular scene converter is selected according to the scene conversion instruction, the video generated by the modular scene converter has the conversion mode corresponding to the scene conversion instruction, thereby accurately controlling the scene conversion mode. Furthermore, the last frame image of the video of the previous scene is used as the control image to connect the beginning and the end of the videos of different scenes, thereby ensuring the continuity between the videos of different scenes. In this way, a multi-scene dynamic perspective long video matching the user instruction is generated.

[0044] In an optional embodiment, the step S110 of "parsing the user instruction into multiple scenes through a large language model" specifically includes: inputting the user instruction into the large language model, parsing the user instruction according to parsing control information to obtain multiple scenes. The parsing control information includes scene extraction order control information and conversion mode identification control information, and the conversion mode is at least one of zooming in, zooming out, panning left, panning right, tilting up, and tilting down.

[0045] In the embodiments of the present application, it is considered that the scene description in the user instruction and the conversion mode between scenes can be implicit rather than directly given, so a large language model is used to analyze the user instruction, and the large language model is controlled to analyze by involving analysis control information, and the analysis is divided into scene descriptions of multiple scenes and conversion modes between scenes.

[0046] For example, the analysis control information can be "extract the scenes in the user instruction in order (i.e., scene extraction order control information), identify the conversion mode between adjacent scenes (i.e., conversion mode identification control information), the scene description should contain rich information, and the conversion mode should be selected from [zoom in, zoom out, left translation, right translation, upward tilt, downward tilt]".

[0047] After the large language model analyzes the user instruction according to the analysis control information, the large language model can output multiple scenes in a target format, where the target format is: [scene n:

scene description information

conversion mode

[0048] By using the technical solution of the embodiments of the present application, each scene in the user instruction is accurately identified by the large language model, which avoids the problem that the user instruction is directly used as the description information of each scene, resulting in the mixing of scenes (i.e., the object that should appear in scene B appears in scene A), thereby ensuring the accuracy of each scene video.

[0049] In an optional embodiment, the method further includes the step of inputting the control image into an adaptive control encoder to extract control features, and the control features are used to guide the base video generator and the modular scene converter to generate a video.

[0050] The control features include control features corresponding to multiple frames of images, and each network layer of the adaptive control encoder has a time dimension self-attention mechanism layer, and the time dimension self-attention mechanism layer is used to control the inter-frame consistency.

[0051] Further, the step S130 specifically includes: inputting the randomly sampled Gaussian noise, the control features, and the scene description into the base video generator and the modular scene converter for multiple rounds of denoising iteration to obtain a video of the scene.

[0052] In the embodiments of the present application, the process of generating a video of each scene is a denoising iteration process, and the randomly sampled Gaussian noise is used as an initial input, and the control features extracted by the adaptive control encoder and the scene description are used as control information, so as to guide the base video generator and the modular scene converter to perform multiple rounds of denoising iteration.

[0053] The technical scheme is implemented in the application. When generating a video of each scene, the last frame image of the video of the previous scene is taken as a control image, the control features of the control image are taken as control information, the generation of the video of the current scene is guided, and the videos of adjacent scenes have continuity. Moreover, the modular scene converter is introduced to cooperate with the base video generator to perform denoising iteration, so that the generated video has a conversion mode corresponding to the scene conversion instruction, and accurate control of the scene conversion mode is realized.

[0054] In an optional embodiment, the base video generator comprises an encoder and a decoder, the encoder comprises a plurality of first network layers and a time dimension self-attention mechanism layer located after each first network layer, and the decoder comprises a plurality of second network layers and a time dimension self-attention mechanism layer located after each second network layer; each modular scene converter comprises a plurality of scene conversion layers located after each first network layer and each second network layer.

[0055] In the embodiments of the application, the plurality of scene conversion layers of the modular scene converter are deployed after the first network layers and the second network layers of the base video generator (i.e., the modular scene converter is embedded and deployed between the network layers of the base video generator), so that the modular scene converter is introduced to cooperate with the base video generator to perform denoising iteration, so that the generated video has a conversion mode corresponding to the scene conversion instruction, and accurate control of the scene conversion mode is realized.

[0056] In some embodiments, the modular scene converter is trained according to the following steps: for a target conversion mode, a second sample video having a corresponding conversion mode is simulated by a data enhancement method, the target conversion mode being one of zooming in, zooming out, moving left, moving right, tilting up and tilting down; and the modular scene converter is trained using the second sample video to obtain a trained modular scene converter.

[0057] In the embodiments of the application, after the training of the base video generator is completed, each scene conversion layer of the modular scene converter is deployed after each first network layer and each second network layer of the base video generator, and by fine-tuning each scene conversion layer of the modular scene converter on a video having a corresponding conversion mode, the generated video also has a corresponding conversion mode.

[0058] Specifically, for each conversion mode (e.g., zoom in, move left, etc.), a set of corresponding parameters is fine-tuned and then loaded onto the parameters of the time dimension self-attention mechanism layer:

[0059] wherein, represents the parameters of the time dimension self-attention mechanism layer, denotes the newly added scene transition layer, which can be decomposed into the product of two low-rank matrices and .

[0060] During training, for a target transition mode (e.g. zoom-in), the data augmentation method first simulates a second sample video with the corresponding transition mode, i.e. the second sample video should have the target transition mode of shot movement. Then, the zoom-in corresponding modular scene transitioner is fine-tuned on the second sample video. For each transition mode, there is a modular scene transitioner corresponding to it, and each modular scene transitioner is independent of each other. When one of them is loaded into the base video generator, the generated video will have the corresponding transition mode.

[0061] Further, for each round of denoising iteration, each temporal dimension self-attention mechanism layer of the base video generator passes the noise vector in the following manner: Step A1: dimensionally transforming the first noise vector from the previous network layer to obtain the second noise vector, the previous network layer being the first network layer or the second network layer, the dimension of the first noise vector being greater than the dimension of the second noise vector.

[0062] Specifically, a temporal dimension self-attention mechanism layer is deployed after each first network layer and second network layer of the base video generator, so the previous network layer of the temporal dimension self-attention mechanism layer is the second network layer.

[0063] wherein the first noise vector is a 5-dimensional vector, and the first noise vector can be represented as wherein denotes the number of batches of samples, denotes the number of channels, and the corresponding RGB image is 3, denotes the number of frames of the video, and denote the height and width of the picture. The first noise vector is processed into a 3-dimensional vector, i.e. the second noise vector .

[0064] Step A2: inputting the second noise vector into the temporal dimension self-attention mechanism layer to perform self-attention processing in the temporal dimension to obtain a third noise vector.

[0065] Exemplarily, the third noise vector can be represented as:

[0066] wherein, denotes the self-attention processing, denotes the query vector, denotes a key vector, denotes a value vector, denotes a weight parameter of the query vector, denotes a weight parameter of the key vector, denotes a weight parameter of the value vector.

[0067] Step A3: performing dimension transformation on the third noise vector to obtain a fourth noise vector, and passing the fourth noise vector to a next network layer, the next network layer being a scene conversion layer, the fourth noise vector having the same dimension as the first noise vector.

[0068] In the embodiments of the present application, the third vector of 3 dimensions is processed into a fourth noise vector of 5 dimensions after self-attention, and the fourth noise vector is continuously passed down. That is, each noise vector is transmitted in the form of 5 dimensions in network transmission, but in the time dimension self-attention mechanism layer, self-attention is performed in the form of 3 dimensions.

[0069] In this way, through this transmission mode, each time dimension self-attention mechanism layer in the base video generator will transmit information in the time dimension to adjust the content of each frame, thereby improving the inter-frame consistency.

[0070] In an optional embodiment, the adaptive control encoder comprises a plurality of first network layers, a time dimension self-attention mechanism layer and a scene conversion layer located after each first network layer, and the output of each time dimension self-attention mechanism layer and scene conversion layer is connected to a corresponding second network layer in the base video generator.

[0071] In the embodiments of the present application, the adaptive control encoder only receives the control image to generate the control feature based on the control image to guide the base video generator and the modular scene converter. The structure and initial parameters of the adaptive control encoder are the same as those of the encoder of the base video generator, that is, the adaptive control encoder is obtained by copying the structure and parameters of the encoder of the base video generator, and a time dimension self-attention mechanism layer is also added in the adaptive control encoder to control the inter-frame consistency.

[0072] In some embodiments, the adaptive control encoder is obtained by training according to the following steps: taking the first frame image of the first sample video as a sample control image, restoring the sample video according to the sample control image by the adaptive control encoder to obtain a restored video; updating the parameters of the adaptive control encoder according to the difference between the restored video and the first sample video to obtain a trained adaptive control encoder.

[0073] That is, the first frame of the existing sample video is taken as a control image, and the adaptive control encoder is allowed to restore the whole video according to the control image, so that the adaptive control encoder has the ability to generate video only according to the control image.

[0074] In the embodiments of the present application, the output of each time-dimension self-attention mechanism layer and the scene conversion layer is connected with the corresponding second network layer in the base video generator, so that when the control feature output to each second network layer of the base video generator is used to guide the base video generator and the modular scene converter to perform denoising iteration, the control feature is output to the base video generator.

[0075] Further, the randomly sampled Gaussian noise, the control feature, and the scene description are input into the base video generator and the modular scene converter to perform multiple rounds of denoising iteration to obtain the video of the scene, including: Step B1: In the first round of denoising iteration, the Gaussian noise, the control feature, and the scene description are input into the base video generator and the modular scene converter to perform denoising iteration to obtain a noise vector generated in the first round of denoising iteration. Step B2: In the tth round of denoising iteration, the noise vector generated in the (t-1)th round of denoising iteration, the control feature, and the scene description are input into the base video generator and the modular scene converter to perform denoising iteration to obtain a noise vector generated in the tth round of denoising iteration, where t is an integer between 2 and N, and N is the total number of rounds of denoising iteration. Step B3: The video of the scene is generated according to the noise vector generated in the Nth round of denoising iteration.

[0076] In the embodiments of the present application, the randomly sampled Gaussian noise is taken as the initial input, and the control feature and the scene description are taken as the control information to guide the base video generator and the modular scene converter to perform multiple rounds of denoising iteration, where each round of denoising iteration is performed on the basis of the previous round of denoising iteration, for example, the 5th round of denoising iteration is performed on the basis of the noise vector generated in the 4th round of denoising iteration, and thus the video of the scene is generated through multiple rounds of denoising iteration.

[0077] In an alternative embodiment, the method further includes: Step C1: Before the tth round of denoising iteration starts, a random number is obtained, and if the random number is less than a target value, the noise vector of the control image in the tth round of denoising iteration in the video generation process of the previous scene is obtained.

[0078] Step C2: The noise vector of the control image is used to replace the noise vector of the first frame in the noise vector generated in the (t-1)th round of denoising iteration to obtain a fused noise vector.

[0079] In the embodiments of the present application, in order to ensure that the color tones of multiple scenes remain consistent, so that the final spliced multi-scene dynamic view video has good realism, a random fusion mechanism of noise vectors is proposed, so that the first frame image of the current scene is as close to the control image as possible. The random fusion mechanism of noise vectors is realized through the above steps C1 and step C2.

[0080] Specifically, in the tthround of denoising iteration, whether noise vector fusion needs to be performed is determined by a random number and a target value three-dimensional size relationship; wherein the random number is a number between 0 and 1, and the target value can be 0.5. If the random number is less than the target value, noise vector fusion needs to be performed, that is, the noise vector of the first frame in the noise vector generated by the (t-1) thround of denoising iteration is replaced by the noise vector of the control image, otherwise, noise vector fusion is not performed.

[0081] For example, the noise vector fusion can be expressed as:

[0082] Wherein, represents the noise vector of the first frame of the tthround of denoising iteration, represents the noise vector of the control image of the tthround of denoising iteration, a random number between 0 and 1 will be generated, represents the target value. That is, if the random number is less than the target value, noise vector fusion needs to be performed, and at this time, is equal to the noise vector of the control image of the tthround of denoising iteration, otherwise, vector fusion is not performed, that is, the noise vector of the first frame is unchanged.

[0083] By using the technical scheme of the embodiments of the present application, the input noise vector in the denoising iteration is fused through the random fusion mechanism, so that the first frame image of the current scene is as close to the control image as possible, thereby ensuring that the color tones of multiple scenes remain consistent.

[0084] Further, the above step B2 can specifically include: in the tthround of denoising iteration, inputting the fused noise vector, the control feature, and the scene description into the base video generator and the modular scene converter for denoising iteration, to obtain the noise vector generated by the tthround of denoising iteration.

[0085] In the embodiments of the present application, if noise vector fusion is performed in the tthround of denoising iteration, denoising iteration is performed based on the fused noise vector, so that the first frame image of the video of the current scene is as close to the control image as possible.

[0086] ​In an alternative embodiment, the method further comprises adjusting the variance and mean of each frame of video of each scene according to the variance and mean of the control image, to obtain adjusted video of each scene.

[0087] In the embodiments of the present application, in order to ensure that the color tones of multiple scenes are consistent, so that the multi-scene dynamic perspective video obtained by splicing finally has good reality, the variance and mean of the video are adjusted to make the mean and variance of the generated video consistent with the control image.

[0088] For example, the adjusted color of each frame of image can be represented as:

[0089] wherein, represents the adjusted color channel of the current frame of image, represents the color channel of the current frame of image before adjustment, represents the mean of the current frame of image, represents the variance of the current frame of image, represents the variance of the control image, represents the mean of the control image.

[0090] Further, the step S140 of "splicing the video of each scene to obtain a multi-scene dynamic perspective video matched with the user instruction" specifically comprises: splicing the adjusted video of each scene to obtain a multi-scene dynamic perspective video matched with the user instruction.

[0091] By using the technical solution of the embodiments of the present application, the variance and mean of the video of each scene are adjusted according to the variance and mean of the control image, so that the variance and mean of the generated video are consistent with the control image, thereby ensuring that the multi-scene dynamic perspective video obtained finally has good reality.

[0092] The dynamic perspective video generation method based on modular network provided by the embodiments of the present application can be implemented by one dynamic perspective video generation model, such as Figure 2 and Figure 3As shown, the dynamic perspective video generation model includes a large language model and a video generation module, and the video generation module includes a modular scene converter, an adaptive control encoder, and a base video generator. The base video generator includes an encoder and a decoder, the encoder includes a plurality of first network layers and a time dimension self-attention mechanism layer after each first network layer, and the decoder includes a plurality of second network layers and a time dimension self-attention mechanism layer after each second network layer. Each modular scene converter includes a plurality of scene conversion layers after each first network layer and each second network layer. The structure and parameters of the adaptive control encoder are the same as those of the encoder of the base video generator, and the output of each time dimension self-attention mechanism layer and scene conversion layer in the adaptive control encoder is connected to the corresponding second network layer in the base video generator.

[0093] Specifically, the user instruction is parsed into multiple scenes by the large language model, each scene including a scene description and a scene conversion instruction, and the scene conversion instruction represents a conversion mode between scenes, wherein the conversion mode is at least one of zooming in, zooming out, left translation, right translation, upward tilting, and downward tilting, so as to decompose the long video generation of multiple scenes of dynamic perspective into video generation of multiple scenes.

[0094] In generating the video of each scene, a modular scene converter is selected according to the scene conversion instruction, the last frame image of the video of the previous scene is taken as a control image, and the control image is input into the adaptive control encoder for feature extraction to obtain control features. Then, the randomly sampled Gaussian noise, the control features, and the scene description are input into the base video generator and the modular scene converter for multiple rounds of denoising iteration to obtain the video of the scene. Finally, the videos of the scenes are spliced to obtain a multiple-scene dynamic perspective video matching the user instruction.

[0095] Compared with related prior art, the dynamic perspective video generation method based on a modular network of the embodiments has the following advantages: 1) The video generated by the existing method mostly contains only a single scene or a single perspective, while the video generated by the method of the embodiments contains multiple scenes or multiple perspectives, and can maintain consistency between multiple scenes and coherence within a single scene. 2) The existing method cannot generate dynamic perspective videos, while the method of the embodiments can finely control the movement mode of the lens.

[0096] Further, in order to better illustrate the dynamic perspective video generation method based on a modular network of the embodiments, the video generated by the method is compared with the videos generated by other baseline models, wherein the baseline models include four models of AnimateDiff, FreeNoise, SparseCtrl, and StreamingT2V.

[0097] For the user instruction "start from a close-up shot of flowers on the grass, slowly move the lens to the right to aim at the distant mountain peak, and gradually zoom in; then, continue to move the lens to the right, and the mountain peak and the lake surface reflect each other", the videos generated by each baseline model and the method of the embodiment of the present application are as shown in FIG. 6 (only part of the key frames are shown here). Figure 4 For the user instruction "start from a scene of a beach, gradually zoom in, and the sea waves are splashing against the reefs; then, slowly move the lens to the right, and a large sea surface is revealed", the videos generated by each baseline model and the method of the embodiment of the present application are as shown in FIG. 7 (only part of the key frames are shown here). Figure 5 Figure 4 Figure 5 As can be seen, the dynamic view video generation method based on the modular network in the embodiment of the present application generates a more coherent video, the dynamic view is stronger, and is most consistent with the user instruction.

[0098] The embodiment of the present application also provides a dynamic view video generation device based on a modular network, referring to Figure 6 as shown in FIG. 8, Figure 6 is a structural schematic diagram of a dynamic view video generation device based on a modular network provided by the embodiment of the present application, and the device comprises: The analysis module 610 is configured to parse the user instruction into multiple scenes by a large language model, each scene comprising a scene description and a scene transition instruction, and the scene transition instruction representing a transition mode between scenes. The selection module 620 is configured to select a modular scene transitioner according to the scene transition instruction, and the modular scene transitioner is configured to make the generated video have the transition mode corresponding to the scene transition instruction. The generation module 630 is configured to generate the video of each scene in sequence by a base video generator and the modular scene transitioner, taking the control image and the scene description as control information, and the control image is the last frame image of the video of the previous scene. The splicing module 640 is configured to splice the videos of each scene to obtain a multi-scene dynamic view video matched with the user instruction.

[0099] It can be understood that the dynamic view video generation device based on a modular network in the embodiment of the present application can implement the dynamic view video generation method based on a modular network in the above-mentioned embodiment. The dynamic view video generation device based on a modular network has the same advantages as the dynamic view video generation method based on a modular network in the above-mentioned embodiment, and thus will not be described here.

[0100] The embodiment of the present application also provides an electronic device, referring to Figure 7 as shown in FIG. 9,​​Figure 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 7 the electronic device 700 includes a memory 710 and a processor 720, the memory 710 and the processor 720 are in communication connection through a bus, the memory 710 stores a computer program, the computer program can run on the processor 720, and then the steps of the dynamic view video generation method based on the modular network are realized.

[0101] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between each embodiment can be referred to each other.

[0102] Embodiments of the present application are described with reference to flowcharts and / or block diagrams of the method and device according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal equipment realize the functions specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 the device that realizes the functions specified in one block or multiple blocks.

[0103] These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing terminal equipment to work in a specific way, so that the instructions stored in the computer readable memory generate a product including instruction devices, which realize the functions specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 the device that realizes the functions specified in one block or multiple blocks.

[0104] These computer program instructions can also be loaded into a computer or other programmable data processing terminal equipment, so that a series of operation steps are performed on the computer or other programmable terminal equipment to generate a computer implemented process, so that the instructions executed on the computer or other programmable terminal equipment provide steps for realizing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 the device that realizes the functions specified in one block or multiple blocks.

[0105] While the preferred embodiments of the application have been described above, it will be recognized and understood that various modifications and changes can be made to the embodiments of the present application by those skilled in the art having the benefit of this detailed description without departing from the scope of the present application. It is therefore intended that the appended claims be construed to include all such embodiments and modifications as fall within the scope of the present application.

[0106] Finally, it should be noted that, in the specification, relational terms such as first and second, and the like, can be used solely to distinguish one entity or action from another entity or action without necessarily implying any actual relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0107] The above describes in detail a dynamic view video generation method and device based on a modular network provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application scopes can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for dynamic view video generation based on a modular network, characterized in that, The method comprises: parsing a user instruction into multiple scenes by a large language model, each scene comprising a scene description and a scene transition instruction, the scene transition instruction representing a transition manner between scenes; selecting a modular scene transitioner according to the scene transition instruction, the modular scene transitioner being used to make the generated video have a transition manner corresponding to the scene transition instruction; generating a video of each scene in turn by a base video generator and the modular scene transitioner, taking a control image and the scene description as control information, the control image being the last frame image of the video of the previous scene; splicing the videos of each scene to obtain a multi-scene dynamic perspective video matching the user instruction.

2. The method of claim 1, wherein, The user instruction is parsed into multiple scenes by a large language model, comprising: inputting the user instruction into the large language model, and parsing the user instruction according to parsing control information to obtain multiple scenes; wherein the parsing control information comprises scene extraction sequence control information and transition manner identification control information, and the transition manner is at least one of zooming in, zooming out, panning left, panning right, tilting up and tilting down.

3. The method of claim 1, wherein, The method further comprises: inputting the control image into an adaptive control encoder to extract control features, the control features being used to guide the base video generator and the modular scene transitioner to generate a video; generating a video of each scene in turn by a base video generator and the modular scene transitioner, taking a control image and the scene description as control information, comprising: inputting randomly sampled Gaussian noise, the control features and the scene description into the base video generator and the modular scene transitioner for multiple rounds of denoising iteration to obtain a video of a scene.

4. The method of claim 3, wherein, The base video generator comprises an encoder and a decoder, the encoder comprises multiple first network layers and a time dimension self-attention mechanism layer after each first network layer, and the decoder comprises multiple second network layers and a time dimension self-attention mechanism layer after each second network layer; each modular scene transitioner comprises multiple scene transition layers after each first network layer and each second network layer; For each round of denoising iteration, each time dimension self-attention mechanism layer of the base video generator passes a noise vector in the following manner: dimensionally transforming a first noise vector from a previous network layer to obtain a second noise vector, the previous network layer being a first network layer or a second network layer, the dimension of the first noise vector being greater than the dimension of the second noise vector; inputting the second noise vector into the time dimension self-attention mechanism layer to perform self-attention processing in the time dimension to obtain a third noise vector; dimensionally transforming the third noise vector to obtain a fourth noise vector, and passing the fourth noise vector to a next network layer, the next network layer being a scene transition layer, the dimension of the fourth noise vector being the same as the dimension of the first noise vector.

5. The method of claim 4, wherein, The adaptive control encoder comprises a plurality of first network layers, a time dimension self-attention mechanism layer after each first network layer, and a scene conversion layer, an output of each time dimension self-attention mechanism layer and scene conversion layer being connected with a corresponding second network layer in the base video generator; The random sampled Gaussian noise, the control feature, and the scene description are input into the base video generator and the modular scene converter for multiple rounds of denoising iteration to obtain a video of a scene, comprising: In the first round of denoising iteration, the Gaussian noise, the control feature, and the scene description are input into the base video generator and the modular scene converter for denoising iteration to obtain a noise vector generated in the first round of denoising iteration; In the tth round of denoising iteration, a noise vector generated in the (t-1)th round of denoising iteration, the control feature, and the scene description are input into the base video generator and the modular scene converter for denoising iteration to obtain a noise vector generated in the tth round of denoising iteration, wherein t is an integer between 2 and N, and N is the total number of rounds of denoising iteration; According to the noise vector generated in the Nth round of denoising iteration, a video of a scene is generated.

6. The method of claim 5, wherein, The method further comprises: Before the tth round of denoising iteration starts, a random number is obtained, and in the case that the random number is less than a target value, a noise vector of a control image in the tth round of denoising iteration in the video generation process of a previous scene is obtained; The noise vector of the control image is used to replace a noise vector of a first frame in the noise vector generated in the (t-1)th round of denoising iteration to obtain a fused noise vector; In the tth round of denoising iteration, the noise vector generated in the (t-1)th round of denoising iteration, the control feature, and the scene description are input into the base video generator and the modular scene converter for denoising iteration to obtain a noise vector generated in the tth round of denoising iteration, comprising: In the tth round of denoising iteration, the fused noise vector, the control feature, and the scene description are input into the base video generator and the modular scene converter for denoising iteration to obtain a noise vector generated in the tth round of denoising iteration.

7. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: According to the variance and mean of the control image, the variance and mean of each frame of image of the video of each scene are adjusted to obtain an adjusted video of each scene; The videos of each scene are spliced to obtain a multi-scene dynamic perspective video matching the user instruction, comprising: The adjusted videos of each scene are spliced to obtain a multi-scene dynamic perspective video matching the user instruction.

8. The method of claim 3, wherein, The adaptive control encoder is trained according to the following steps: A first frame of image of a first sample video is taken as a sample control image, and the adaptive control encoder is used to restore the sample video to obtain a restored video; According to the difference between the restored video and the first sample video, the parameters of the adaptive control encoder are updated to obtain a trained adaptive control encoder.

9. The method of claim 1, wherein, The modular scene converter is trained according to the following steps: For a target conversion mode, a second sample video having the corresponding conversion mode is simulated through a data enhancement method, the target conversion mode being one of zooming in, zooming out, shifting left, shifting right, tilting up and tilting down; The modular scene converter is trained by using the second sample video, to obtain a trained modular scene converter. 10.A dynamic view video generation apparatus based on a modular network, characterized by The device comprises: A parsing module configured to parse a user instruction into multiple scenes by a large language model, each scene comprising a scene description and a scene conversion instruction, the scene conversion instruction representing a conversion mode between scenes; A selection module configured to select a modular scene converter according to the scene conversion instruction, the modular scene converter being configured to make a generated video have a conversion mode corresponding to the scene conversion instruction; A generation module configured to generate a video of each scene in sequence by a base video generator and the modular scene converter, taking a control image and the scene description as control information, the control image being a last frame image of a video of a previous scene; A splicing module configured to splice the videos of the scenes to obtain a multi-scene dynamic perspective video matching the user instruction.

Images