Video generation method, electronic device, storage medium, and product

Abstract

The present disclosure relates to a video generation method, an electronic device, a storage medium and a product, and relates to the technical field of computer. The video generation method comprises: obtaining a plot text and an image corresponding to the plot text; determining one or more visualized objects and description information of the one or more visualized objects from the plot text; determining a plurality of image elements from the image; determining a dynamic strategy of the image according to a corresponding relationship between the plurality of image elements and the one or more visualized objects and the description information of the one or more visualized objects; and processing the image by using the dynamic strategy to generate a video corresponding to the plot text.

Description

Video generation methods, electronic devices, storage media and products Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video generation method, electronic device, storage medium, and product. Background Technology

[0002] With the development of multimedia and mobile internet technologies, works presented in text form can be showcased in a more three-dimensional way through multimedia. For example, when describing or explaining literary works such as stories, novels, and essays, corresponding videos can be used to enhance the description or explanation effect. Summary of the Invention

[0003] According to some embodiments of this disclosure, a video generation method is provided, including: acquiring plot text and images corresponding to the plot text; determining one or more visual objects and descriptive information for the one or more visual objects from the plot text; determining multiple image elements from the images; determining an image dynamization strategy based on the correspondence between the multiple image elements and one or more visual objects, and the descriptive information for the one or more visual objects; and processing the images using the dynamization strategy to generate a video corresponding to the plot text.

[0004] According to other embodiments of the present disclosure, an electronic device is provided, including a memory; and a processor coupled to the memory, the processor being configured to perform a video generation method of any embodiment of the present disclosure based on instructions stored in the memory.

[0005] According to some embodiments of the present disclosure, an electronic device is provided, including: a memory; and a processor coupled to the memory, the processor being configured to execute a video generation method of any embodiment of the present disclosure based on instructions stored in the memory.

[0006] According to some embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, performs the video generation method of any embodiment described in the present disclosure.

[0007] Other features, aspects, and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0008] Embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the drawings described below are merely illustrative of some embodiments of this disclosure and are not intended to limit the scope of this disclosure.

[0009] Figure 1 shows a schematic flowchart of a video generation method according to some embodiments of the present disclosure.

[0010] Figure 2 shows a flowchart illustrating a method for determining an image dynamization strategy according to some embodiments of the present disclosure.

[0011] Figure 3 illustrates a flowchart of a method for determining an image dynamization strategy according to other embodiments of the present disclosure.

[0012] Figure 4 shows a flowchart illustrating a method for determining an image dynamization strategy according to some embodiments of the present disclosure.

[0013] Figure 5 shows a schematic diagram of the structure of a video generation apparatus according to some embodiments of the present disclosure.

[0014] Figure 6 shows a schematic diagram of the structure of a video generation apparatus according to other embodiments of the present disclosure.

[0015] Figure 7 shows a block diagram of an electronic device according to some embodiments of the present disclosure.

[0016] Figure 8 shows a block diagram of an electronic device according to other embodiments of the present disclosure. Detailed Implementation

[0017] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0018] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0019] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". The term "based on" means "at least partially based on".

[0020] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0021] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0022] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0023] It should be understood that this disclosure does not limit how the image to be applied / processed is obtained. In some embodiments of this disclosure, it can be obtained from a storage device, such as internal memory or external storage device. In other embodiments of this disclosure, a camera component can be invoked to capture an image. It should be noted that the acquired image can be a captured image or a frame from a captured video, and is not particularly limited to these.

[0024] In the context of this disclosure, "image" can refer to any of a variety of images, such as color images, grayscale images, etc. It should be noted that the type of image is not specifically limited in the context of this specification. Furthermore, an image can be any suitable image, such as a raw image obtained by a camera device, or an image from which specific processing has been performed, such as preliminary filtering, dealiasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that preprocessing operations may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0025] Currently, videos corresponding to literary works are typically produced through manual filming and acting. Creating videos through manual filming and acting requires significant human and material resources, and the production cycle is generally quite long, making this method of video generation quite challenging. However, in an era of booming multimedia content, users' demand for combining written works with video is increasing, leading to a surge in demand for video production.

[0026] The development of artificial intelligence technology has provided convenience for solving the above problems. Users can use generative models to generate videos corresponding to narrative text, such as video generation models using adversarial networks and diffusion models. After the user inputs the narrative text into the video generation model, the model can generate images corresponding to the narrative text and stitch together images corresponding to narrative texts with contextual relationships. For example, after each image plays for a certain duration, the model switches to the next image to obtain the video corresponding to this narrative text with contextual relationships.

[0027] However, videos created based on image stitching do not have a very good effect. This is because images are static, and the various visual objects within them are also static. Videos created by stitching images together simply display these images one by one; they do not actually provide a dynamic display based on the corresponding narrative text. For example, if the narrative text includes the information "The leaves are gently swaying in the wind," the images can only show the state of the leaves at a certain moment, but cannot show the process of the leaves swaying.

[0028] Based on this, this disclosure provides a video generation method, electronic device, storage medium, and product. After acquiring the plot text and its corresponding image, the image elements in the image are dynamically generated based on the understanding of the plot text, thus obtaining a video that matches the plot text. The video generation method of this disclosure is not only highly efficient but also achieves good matching results with the plot text.

[0029] Figure 1 shows a schematic flowchart of a video generation method according to some embodiments of the present disclosure. As shown in Figure 1, the video generation method of this embodiment includes steps S102 to S110. The method of this embodiment can be executed by an electronic device, such as a computer, terminal device, etc.

[0030] In step S102, the plot text and the corresponding image are obtained.

[0031] A plot text refers to a text containing a plot. Plot texts can be taken from various literary works, such as novels, plays, and essays. The length of a plot text can be determined as needed; for example, a plot text can be taken from one or more paragraphs of a literary work.

[0032] The images corresponding to the plot text can be automatically generated (e.g., generated using a generative model) or manually drawn, etc. In the case of automatically generated images, the images corresponding to the plot text can be generated by an electronic device performing the video method of this disclosure, or by other devices. This disclosure does not limit the method of generating the images corresponding to the plot text.

[0033] In step S104, one or more visual objects and descriptive information for one or more visual objects are determined from the plot text.

[0034] The narrative text includes a wide variety of information, such as character dialogue, inner thoughts, environmental descriptions, and plot development. Visual objects are information within the narrative text that can be displayed visually, such as characters, objects, environments, or parts of those environments. Identifying one or more visual objects from the narrative text can be a process of filtering information—selecting information that is likely to guide the animation of the images. Alternatively, it can filter out information from the narrative text that cannot be displayed visually.

[0035] A visualization object is used to determine which part of an image should be animated, while the descriptive information of the visualization object is used to determine how the image should be animated. The descriptive information of a visualization object often includes its characteristics and state. Based on the descriptive information, the animation method of the visualization object can be determined, and the image can be animated accordingly.

[0036] The process of identifying one or more visual objects from the narrative text can be achieved using various machine learning models, such as natural language processing models and deep learning models. It can also be implemented directly through an intelligent agent. This intelligent agent can be referred to as a robot, digital human, or a virtual proxy for a machine learning model. The intelligent agent can be implemented based on one or more machine learning models, such as a Large Language Model (LLM) or Foundation Model. It can also be implemented through keyword matching. It's understood that when using a machine learning model to identify one or more visual objects from the narrative text, that machine learning model is trained. When identifying one or more visual objects from the narrative text through an intelligent agent, prompts can be given to the agent to provide the required information for identifying one or more visual objects from the narrative text, such as quantity and type.

[0037] Descriptive information for one or more visual objects can also be obtained through various machine learning models or agents. This descriptive information includes information directly obtained from the narrative text, as well as information generated based on the narrative text. For example, the narrative text might include "The mother is checking on her daughter's injuries," where "mother" is a visual object. Although the narrative text doesn't describe the mother's emotions, based on an understanding of the narrative text, the descriptive information for "mother" could include emotions like "worry" or "anxiety."

[0038] The determination of one or more visual objects from the narrative text, along with descriptive information about those objects, is achieved based on an understanding of the narrative text. In other words, the electronic device implementing the video generation method of this disclosure possesses the capability to understand the narrative text.

[0039] In step S106, multiple image elements are determined from the image.

[0040] Image elements are the basic components that make up an image. Animating an image involves dynamically manipulating the image elements within it.

[0041] At least some image elements in the image correspond to one or more visual objects. After determining the dynamic mode of one or more visual objects, the image elements corresponding to the one or more visual objects are dynamically processed so that the image is dynamic in accordance with the dynamic mode of one or more visual objects.

[0042] In step S108, a dynamic strategy for the image is determined based on the correspondence between multiple image elements and one or more visual objects, as well as the descriptive information of one or more visual objects.

[0043] By identifying multiple image elements corresponding to one or more visual objects from an image, and determining the dynamic mode of one or more visual objects based on the description information of one or more visual objects, it is possible to determine the dynamic mode of multiple image elements corresponding to one or more visual objects, and thus determine the dynamic strategy of the image.

[0044] Alternatively, the dynamics of the image can be determined based on the descriptive information of one or more visual objects. The dynamics of the image include the dynamics of one or more visual objects, as well as the overall dynamics of the image, such as camera movement. In this way, the image's dynamism strategy can be determined based on its dynamics and the multiple image elements corresponding to one or more visual objects.

[0045] An image dynamization strategy is determined based on the method of dynamically processing multiple image elements in the image. For example, based on the description information of one or more visual objects, if it is determined that the first visual object is dynamized according to a first dynamic method, and the second visual object is dynamized according to a second dynamic method, then the image dynamization strategy includes processing one or more image elements corresponding to the first visual object according to the first image processing method, and processing one or more image elements corresponding to the second visual object according to the second image processing method, wherein the first image processing method corresponds to the first dynamic method, and the second image processing method corresponds to the second dynamic method.

[0046] In step S110, the image is processed using a dynamic strategy to generate a video corresponding to the plot text.

[0047] After determining the image dynamization strategy, the images are processed according to the strategy to obtain a video corresponding to the plot text. This video is generated based on an understanding of the plot text and by dynamizing the images corresponding to the text; therefore, the video has a high degree of matching with the plot text.

[0048] In the above embodiments, after acquiring the plot text and its corresponding images, the images are dynamically processed based on the understanding of the plot text to generate a video corresponding to the plot text. This video generation method is convenient, efficient, and highly applicable. By identifying the more important visual objects in the plot text that enable dynamic image processing, as well as the descriptive information of these visual objects, the images can be dynamically processed more accurately and effectively. Therefore, the embodiments of this disclosure have high video generation efficiency, thereby providing users with high-quality multimedia content and more efficiently assisting users in understanding the plot content.

[0049] Identifying visual objects and their descriptions from the narrative text is based on an understanding of the text. This understanding involves first generating information across multiple specified dimensions, and then using that information to generate visual objects and their descriptions.

[0050] That is, determining the visual object and its descriptive information from the plot text includes: generating information in multiple specified dimensions based on the plot text, wherein the multiple specified dimensions include at least one of scene, environment, character, action, and emotion; and determining the visual object and its descriptive information based on the attributes and relationships of the information in the multiple specified dimensions.

[0051] The specified dimensions are configured based on the goal of comprehensively reflecting the content of the narrative text. These dimensions include different attributes; for example, characters and scenes are objective attributes, while actions, emotions, and environment are descriptive attributes. Objective attributes correspond to the visualized objects, while descriptive attributes correspond to the descriptive information about those objects. Based on the relationships between different attributes, the visualized objects and their descriptive information are determined. For example, when determining a character as a visualized object from the objective attributes, information associated with that character is extracted from the descriptive attributes to serve as the character's descriptive information. Information associated with a character could include their actions and emotions. Information associated with a scene could include the environment and atmosphere.

[0052] In some embodiments, one or more visual objects include at least one of a person and a scene, the descriptive information of the person includes at least one of an emotion dimension and an action dimension, and the descriptive information of the scene includes at least one of an environment dimension and an atmosphere dimension.

[0053] While narrative text often includes many visual objects, to consider the video generation effect, only a portion of these visual objects are animated during image processing. This is because animate selections of important visual objects highlight key elements in the video, thus better showcasing the content of the narrative text.

[0054] Furthermore, since humans tend to focus on and remember key information when interpreting narrative text, paying less attention to peripheral details, the video presentation of the narrative text should also showcase the main information to minimize interference with the user's understanding.

[0055] Figure 2 shows a flowchart illustrating a method for determining an image dynamization strategy according to some embodiments of the present disclosure. As shown in Figure 2, the method of this embodiment includes steps S202 to S208.

[0056] In step S202, based on the description information of one or more visual objects, the target visual object and the dynamic mode of the target visual object are determined from the one or more visual objects.

[0057] A target visualization object is a visualization object that will be dynamically processed in an image, determined from one or more visualization objects. A target visualization object can consist of only one visualization object or multiple visualization objects. For example, a target visualization object can be determined from one or more visualization objects based on the amount of information in the descriptive information of those objects. An information threshold can be set; if the amount of information in the descriptive information of a visualization object exceeds the threshold, that visualization object is determined as the target visualization object. For example, if the plot text extensively describes visualization object A, i.e., the amount of information in the descriptive information of visualization object A is high, then visualization object A is determined as the target visualization object.

[0058] Alternatively, the target visual object can be determined based on the relationships between one or more visual objects. That is, a visual object with strong connections to other visual objects can be identified as the target visual object. Alternatively, visual objects associated with important visual objects can be identified as target visual objects. For example, characters, items, or scenes related to the main character.

[0059] After identifying the target visual object, the dynamic mode of the target visual object is determined based on the description information of the target visual object. For example, if the plot text includes "the leaves are swaying gently in the wind," and the target visual object is a leaf, the dynamic mode of the leaf is "swaying gently."

[0060] Multiple alternative dynamic modes can be configured based on the type of the target visualization object. This allows for direct selection from these options during practical applications, thereby improving the efficiency of image animation. The dynamic mode for the target visualization object is determined from the corresponding alternative modes based on its type. Note that the alternative dynamic modes are not entirely the same for different types of target visualization objects.

[0061] The types of the target visualization objects include at least one of the following: people and scenes. The alternative dynamic methods corresponding to the people type include at least one of the following: facial expression changes, action changes, and clothing changes. The alternative dynamic methods corresponding to the scene type include at least one of the following: changes in the natural environment and object movement.

[0062] Since narrative text typically unfolds through characters' emotions and actions, character dynamism is a crucial aspect of visual animation. Scenery, on the other hand, describes the background and environment, including various objects such as buildings, vehicles, plants, animals, the sky, stars, raindrops, and lightning.

[0063] A scene can also include characters, such as a crowd. That is, the character's dynamics can be determined from either the alternative dynamics corresponding to the character type or the alternative dynamics corresponding to the scene type. Compared to the alternative dynamics corresponding to the character type, determining the character's dynamics from the alternative dynamics corresponding to the scene type focuses more on the overall picture, such as the handling of crowd dynamics. In other words, the alternative dynamics corresponding to the character type can be used to depict the details of the character's dynamics, such as facial expressions.

[0064] Facial expressions convey changes in a character's emotions, including one or more of the following: smiling, crying, and frowning. Actions convey changes in a character's state, including one or more of the following: running and waving. Clothing changes convey changes in a character's appearance, such as a skirt being blown by the wind or shoes getting wet in the rain. Dynamic actions corresponding to different character types include: running from a distance, smiling, and long hair being blown by the wind. Changes in the natural environment convey changes in weather or conditions, such as rain or wind. Object movement conveys changes in various objects within the scene, such as moving vehicles or running animals.

[0065] By selecting alternative animation methods based on character type and scene type, the target visual objects in the plot text can be displayed dynamically and visually in a reasonable and vivid way. It can be understood that the alternative animation methods for various types of target visual objects can be updated as needed. Alternatively, the type of the target visual object can also be updated; for example, when animals require sufficient animation, alternative animation methods for the animal type can be configured separately.

[0066] In step S204, the target image element corresponding to the target visual object is determined from the multiple image elements according to the correspondence between multiple image elements and one or more visual objects.

[0067] After determining the target visualization object, the target image element corresponding to the target visualization is determined based on the correspondence between multiple image elements and one or more visualization objects. Then, the target image element is dynamically processed according to the dynamic mode of the target visualization object.

[0068] In step S206, a dynamic strategy for the target image elements is determined based on the dynamic nature of the target visualization object.

[0069] After identifying the target image elements, dynamic processing is required based on the dynamic characteristics of the target visualization object. For example, when a person is the target visualization object, the person's dynamic characteristics include facial expressions; therefore, processing needs to be performed on the image elements corresponding to the person's face within the target image elements. Furthermore, the dynamic characteristics of the target visualization object can also be a combination of multiple alternative dynamic characteristics. For instance, if a person's dynamic characteristics include facial expressions and movement changes, then processing needs to be performed on the image elements corresponding to the person's face and movements (e.g., arms) within the target image elements.

[0070] That is, based on the dynamic nature of the target visualization object, the target visualization elements within the target visualization object are determined; based on the dynamic nature of the target visualization object and the target visualization elements, the dynamicization strategy for the target image elements is determined. Target visualization elements can be parts or the entire target visualization object. For example, the target visualization elements for a person can be parts such as the face or arms, or the entire person.

[0071] In step S208, the dynamic strategy of the image is determined according to the dynamic strategy of the target image elements.

[0072] A dynamic strategy for the image can be obtained by integrating the dynamic strategies of the target image elements. For example, the dynamic strategies of the target image elements can be integrated in a specified order, meaning that the dynamics of the target image elements are performed in a specified order. Alternatively, the dynamics of the target image elements can be performed simultaneously.

[0073] In the above embodiments, the image dynamization strategy is determined by identifying the target visual object and its dynamic mode; that is, image dynamization is based on the visual objects in the image. Alternatively, the image dynamization strategy can also be determined based on the overall camera movement of the image.

[0074] That is, in some embodiments, the dynamization strategy includes a camera movement strategy. Determining the dynamization strategy of an image based on the correspondence between multiple image elements and one or more visual objects, and the descriptive information of one or more visual objects, includes: determining the association relationship of one or more visual objects based on the descriptive information of one or more visual objects; determining the camera movement strategy of the image based on the association relationship of one or more visual objects; and determining the dynamization strategy of the image based on the correspondence between multiple image elements and one or more visual objects and the camera movement strategy of the image.

[0075] Camera movement strategies can make generated videos more vivid. For example, in a narrative text such as "an apple fell off the table," applying camera movement strategies, including push-in shots and tracking shots, can more vividly show the process of the apple falling. In other words, camera movement strategies can further vividly demonstrate the dynamic process of visual objects by animatening them within the image.

[0076] In addition to determining the image dynamization strategy based on one or more visual objects and their dynamic modes as described in the above embodiments, determining the image dynamization strategy also includes determining the dynamic duration of the image. Appropriately determining the dynamic duration of the image can improve its matching with the narrative text.

[0077] Figure 3 illustrates a flowchart of a method for determining an image dynamization strategy according to other embodiments of the present disclosure. As shown in Figure 3, the method of this embodiment includes steps S302 to S306.

[0078] In step S302, the duration of the audio corresponding to the image is determined.

[0079] The duration of the audio corresponding to the image can be determined based on the playback or narration duration of the corresponding plot text. The audio corresponding to the image refers to the audio used to describe or narrate the plot text, which can be generated based on the plot text, for example, by extracting the main information from the plot text.

[0080] Audio corresponding to images can be generated using a generative model. The plot text and instructions are input into the generative model; the instructions specify the duration and content of the audio generated by the model. This process yields the audio file output by the generative model.

[0081] In step S304, the dynamic duration of the image is determined based on the duration of the audio.

[0082] The duration of the image's animation can match the duration of the audio, allowing the generated video to complement the playback or narration of the story text. Alternatively, the duration of the image's animation can differ from the audio duration, enabling adaptive pauses and presentations as needed when narrating or describing the story text.

[0083] In step S306, the image dynamization strategy is determined based on the correspondence between multiple image elements and one or more visual objects, the description information of one or more visual objects, and the dynamic duration of the image.

[0084] The duration of an image's dynamic range can affect the number of image elements that can be dynamically processed, or the processing time for each image element. For example, the shorter the duration of an image's dynamic range, the fewer image elements can be dynamically processed, and the shorter the processing time for each image element. Alternatively, by combining the importance of the visualization object corresponding to each image element with the image's dynamic range, the number of image elements that can be dynamically processed can be further filtered, or the processing time for each image element can be further determined.

[0085] Based on the description information of one or more visualized objects, determine one or more dynamic visualization methods, and combine the correspondence between multiple image elements and one or more visualized objects, as well as the dynamic duration of the images, to determine the image dynamization strategy. In other words, the image dynamization strategy includes controlling the duration of dynamic processing of image elements.

[0086] Besides determining the dynamic duration of an image based on the duration of the audio corresponding to the image, the dynamic duration of an image can also be determined based on the plot text. Figure 4 shows a flowchart illustrating a method for determining an image dynamization strategy according to some embodiments of the present disclosure. As shown in Figure 4, the method of this embodiment includes steps S402 to S406.

[0087] In step S402, the style of the plot text is determined based on the description information of one or more visual objects.

[0088] The narrative text can correspond to various styles, such as heartwarming, terrifying, or oppressive. The style of the narrative text can be generated by analyzing the content of the narrative text using a natural language processing model, or it can be specified in advance through tagging.

[0089] In step S404, the dynamic duration of the image is determined according to the style of the plot text.

[0090] Different styles of narrative text correspond to different dynamic rhythms in the images. For example, a warm and soothing narrative text might correspond to a slow dynamic rhythm, while a scary narrative text might correspond to a fast dynamic rhythm. By determining the style of the narrative text, the dynamic rhythm of the images can be determined, and consequently, the duration of the images can be determined based on that dynamic rhythm. For example, a slow dynamic rhythm can be set to a longer duration, allowing the video to match the style of the narrative text.

[0091] In step S406, the image dynamization strategy is determined based on the correspondence between multiple image elements and one or more visual objects, the description information of one or more visual objects, and the dynamic duration of the image.

[0092] The above embodiments describe the steps for generating videos corresponding to plot text based on images. Furthermore, based on multiple plot texts and multiple images, multiple videos can be generated, and thus a long video can be generated, that is, a long video corresponding to multiple plot texts.

[0093] The video generation method described above also includes: obtaining associated videos, wherein the plot text corresponding to the associated videos and the plot text corresponding to the videos have a contextual relationship; and concatenating the video with the associated videos to generate a video corresponding to multiple plot texts with a contextual relationship.

[0094] When splicing videos with related videos, you can either splice them directly or add transition processing, such as using the last frame of the previous video as the first frame of the current video.

[0095] By splicing the video with related videos, a long video corresponding to a long narrative text can be generated. The long narrative text includes multiple narrative texts as described above. Therefore, the video generation method disclosed herein has good applicability, applicable not only to short video generation but also to long video generation.

[0096] The following is a simple example to describe the process of generating a video corresponding to the narrative text.

[0097] The scenario is as follows: "A stretched and prepared to go out to wash up. As soon as he opened the door, he saw a large crowd gathered around the dormitory building across the street. In the crowd, B was shouting at the top of his lungs, 'Comrades! I brought you some local specialties from my hometown. I hope you won't mind them. I hope you'll all take good care of me in the coming year!'"

[0098] Based on the plot text, the visual objects are identified as A and B. The description of A includes: "A stretched and prepared to wash up and opened the door; A may have been a little curious about the lively scene outside the door." The description of B includes: "B shouted loudly in the crowd about the local specialties he brought; A showed enthusiasm, generosity, and excitement in interacting with the people around him."

[0099] Based on the descriptions of the visual objects A and B, the dynamics of A are determined to include "opening a door with an arm and a curious expression on his face," while the dynamics of B are determined to include "waving and an excited expression on his face." Next, the image elements corresponding to A and B in the images corresponding to the plot text are identified, and these image elements are animated according to the dynamics of A and B, taking into account the audio duration corresponding to the plot text.

[0100] By utilizing the video generation method disclosed herein, videos corresponding to the plot text can be generated quickly and with a high degree of matching with the plot text, thereby improving the explanation or description effect of the plot text.

[0101] Figure 5 shows a schematic diagram of a video generation apparatus according to some embodiments of the present disclosure. As shown in Figure 5, the video generation apparatus 50 of this embodiment includes: an acquisition module 510 configured to acquire plot text and images corresponding to the plot text; a first determination module 520 configured to determine one or more visual objects and descriptive information of the one or more visual objects from the plot text; a second determination module 530 configured to determine multiple image elements from the images; a third determination module 540 configured to determine an image dynamization strategy based on the correspondence between the multiple image elements and one or more visual objects and the descriptive information of the one or more visual objects; and a generation module 550 configured to process the images using the dynamization strategy to generate a video corresponding to the plot text.

[0102] In some embodiments, the third determining module 540 is configured to: determine a target visual object and a dynamic mode of the target visual object from one or more visual objects based on description information of one or more visual objects; determine a target image element corresponding to the target visual object from multiple image elements based on the correspondence between multiple image elements and one or more visual objects; determine a dynamic strategy of the target image element based on the dynamic mode of the target visual object; and determine a dynamic strategy of the image based on the dynamic strategy of the target image element.

[0103] In some embodiments, the third determining module 540 is configured to determine target visualization elements in the target visualization object based on the dynamic mode of the target visualization object; and to determine a dynamic strategy for the target image elements based on the dynamic mode of the target visualization object and the target visualization elements.

[0104] In some embodiments, the third determining module 540 is configured to determine the dynamic mode of the target visualization object from the candidate dynamic modes corresponding to the type of the target visualization object, wherein the candidate dynamic modes for different types of target visualization objects are not completely the same.

[0105] In some embodiments, the type of the target visualization object includes at least one of person and scene, and the alternative dynamic methods corresponding to the person type include at least one of facial expression change, action change and clothing change, and the alternative dynamic methods corresponding to the scene type include at least one of natural environment change and object movement.

[0106] In some embodiments, the third determining module 540 is configured to determine a target visualization object from one or more visualization objects based on the amount of information in the descriptive information of one or more visualization objects.

[0107] In some embodiments, the third determining module 540 is configured to determine the association relationship of one or more visual objects based on the description information of one or more visual objects; determine the camera movement strategy of the image based on the association relationship of one or more visual objects; and determine the animation strategy of the image based on the correspondence between multiple image elements and one or more visual objects and the camera movement strategy of the image.

[0108] In some embodiments, the third determining module 540 is configured to determine the duration of the audio corresponding to the image; determine the dynamic duration of the image based on the duration of the audio; and determine the dynamic strategy of the image based on the correspondence between multiple image elements and one or more visual objects, the description information of one or more visual objects, and the dynamic duration of the image.

[0109] In some embodiments, the third determining module 540 is configured to determine the style of the plot text based on the description information of one or more visual objects; determine the dynamic duration of the image based on the style of the plot text; and determine the animation strategy of the image based on the correspondence between multiple image elements and one or more visual objects, the description information of one or more visual objects, and the dynamic duration of the image.

[0110] In some embodiments, the first determining module 520 is configured to generate information of multiple specified dimensions based on the plot text, wherein the multiple specified dimensions include at least one of scene, environment, character, action, and emotion; and determine the visualization object and the descriptive information of the visualization object based on the attributes and relationships of the information of the multiple specified dimensions.

[0111] In some embodiments, one or more visual objects include at least one of a person and a scene, the descriptive information of the person includes at least one of an emotion dimension and an action dimension, and the descriptive information of the scene includes at least one of an environment dimension and an atmosphere dimension.

[0112] In some embodiments, the video generation apparatus 50 is further configured to acquire associated videos of the video, wherein the plot text corresponding to the associated video has a contextual relationship with the plot text corresponding to the video; and to concatenate the video with the associated video to generate a video corresponding to multiple plot texts having a contextual relationship.

[0113] Figure 6 shows a schematic diagram of a video generation apparatus according to some other embodiments of the present disclosure. As shown in Figure 6, the video generation apparatus of this embodiment includes: an understanding module 610, configured to understand input plot text and output an understanding result, the understanding result including one or more visual objects and descriptive information for the one or more visual objects; a video generation control module 620, configured to generate an image dynamization strategy based on the understanding result input by the understanding module 610, and generate instruction information based on the image dynamization strategy; and a video processing module 630, configured to generate a video corresponding to the plot text based on the instruction information input by the video generation module 620 and combined with static images corresponding to the plot. Furthermore, multiple videos corresponding to plot texts can be stitched together to generate a long video.

[0114] Figure 7 shows a block diagram of an electronic device according to some embodiments of the present disclosure.

[0115] Memory 71 is used to store one or more computer-readable instructions. Memory 71 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 71 may, for example, store operating systems, application programs, boot loaders, databases, and other programs, as well as various application programs and various data.

[0116] The processor 72 is configured to execute computer-readable instructions to implement the method described in any of the foregoing embodiments. Specific implementations of each step of the method can be found in the above embodiments; repeated details will not be elaborated upon here.

[0117] Processor 72 can be configured to execute the steps shown in Figures 1 through 4. Processor 72 can be various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be based on x86 or ARM architectures, etc.

[0118] The processor 72 and the memory 71 can communicate with each other directly or indirectly. For example, the processor 72 and the memory 71 can communicate via a network. The network can include a wireless network, a wired network, and / or any combination of wireless and wired networks. The processor 72 and the memory 71 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0119] It should be noted that the components of the electronic device 7 shown in Figure 7 are merely exemplary and not limiting. The electronic device 7 may have other components depending on the specific application requirements. The processor 72 can control other components in the electronic device 7 to perform the desired functions.

[0120] Electronic device 7 can be implemented by software, firmware and / or hardware, and can be integrated into a device with the relevant application installed.

[0121] Figure 8 shows a block diagram of an electronic device according to other embodiments of the present disclosure.

[0122] The electronic device 8 shown in Figure 8 can be a computer system with a dedicated hardware structure, capable of performing corresponding functions when relevant applications are installed.

[0123] Electronic devices include, but are not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PCs), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital televisions and desktop computers.

[0124] As shown in Figure 8, the Central Processing Unit (CPU) 81 performs various processes based on programs stored in the Read-Only Memory (ROM) 82 or programs loaded from the storage section 88 into the Random Access Memory (RAM) 83. The RAM 83 stores data required as needed when the CPU 81 performs various processes. The CPU is merely exemplary and can also be other types of processors, such as the various processors described above. The ROM 82, RAM 83, and storage section 88 can be various forms of computer-readable storage media. It should be noted that although the ROM 82, RAM 83, and storage section 88 are shown separately in Figure 8, one or more of them can be combined or located in the same or different memories or storage modules.

[0125] CPU 81, ROM 82 and RAM 83 are interconnected via bus 84. Input / output interface 85 is also connected to bus 84.

[0126] The following components are connected to the input / output interface 85: input section 86, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 87, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 88, including hard disk, magnetic tape, etc.; and communication section 89, including network interface cards such as LAN cards, modems, etc. The communication section 89 allows communication processing to be performed via a network such as the Internet. It is readily understood that although parts of the electronic device 8 shown in Figure 8 communicate via bus 84, they can also communicate via a network or other means, wherein the network can include wireless networks, wired networks, and / or any combination of wireless and wired networks.

[0127] As needed, drive 810 is also connected to input / output interface 85. Removable media 811, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 810 as needed, so that computer programs read from them can be installed into storage section 88 as needed.

[0128] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as a removable medium 1211.

[0129] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to perform the methods described in any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer instructions can be downloaded and installed from a network via communication section 89, or installed from storage section 88, or installed from ROM 82. When the computer program is executed by CPU 81, the methods of embodiments of this disclosure are performed.

[0130] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0131] A computer-readable medium may be a computer-readable storage medium, a computer-readable signal medium, or any combination thereof.

[0132] Computer-readable storage media include, but are not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the methods described in any of the foregoing embodiments.

[0133] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0134] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0135] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the methods described in any of the foregoing embodiments. For example, the instructions may be embodied in computer program code.

[0136] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0139] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A video generation method, comprising: Obtain the plot text and the image corresponding to the plot text; Identify one or more visual objects from the plot text, and descriptive information for the one or more visual objects; Identify multiple image elements from the image; Based on the correspondence between the plurality of image elements and the one or more visual objects, and the descriptive information of the one or more visual objects, the dynamic strategy of the image is determined; The image is processed using the dynamic strategy to generate a video corresponding to the plot text.

2. The video generation method according to claim 1, wherein, The step of determining the image dynamization strategy based on the correspondence between the image elements and the one or more visual objects, and the descriptive information of the one or more visual objects, includes: Based on the description information of the one or more visualization objects, determine the target visualization object and the dynamic mode of the target visualization object from the one or more visualization objects; Based on the correspondence between the plurality of image elements and the one or more visual objects, determine the target image element corresponding to the target visual object from the plurality of image elements; Based on the dynamic nature of the target visualization object, determine the dynamic strategy for the target image elements; The dynamic strategy of the image is determined based on the dynamic strategy of the target image elements.

3. The video generation method according to claim 2, wherein, The step of determining the dynamic strategy of the target image elements based on the dynamic mode of the target visualization object includes: Based on the dynamic behavior of the target visualization object, determine the target visualization elements within the target visualization object; Based on the dynamic mode of the target visualization object and the target visualization elements, determine the dynamic strategy of the target image elements.

4. The video generation method according to claim 2, wherein, The method of determining a target visualization object from the one or more visualization objects based on the description information of the one or more visualization objects, and the dynamic method of the target visualization object, includes: Based on the type of the target visualization object, the dynamic mode of the target visualization object is determined from the alternative dynamic modes corresponding to the type, wherein the alternative dynamic modes of different types of target visualization objects are not completely the same.

5. The video generation method according to claim 4, wherein, The type of the target visualization object includes at least one of person and scene. The alternative dynamic methods corresponding to the person type include at least one of facial expression change, action change, and clothing change. The alternative dynamic methods corresponding to the scene type include at least one of natural environment change and object movement.

6. The video generation method according to claim 2, wherein, The method of determining a target visualization object from the one or more visualization objects based on the description information of the one or more visualization objects, and the dynamic method of the target visualization object, includes: The target visualization object is determined from the one or more visualization objects based on the amount of information in the description information of the one or more visualization objects.

7. The video generation method according to claim 1, wherein, The dynamic strategy includes a camera movement strategy. Determining the dynamic strategy for the image based on the correspondence between the plurality of image elements and the one or more visual objects, and the descriptive information of the one or more visual objects, includes: The relationships between the one or more visual objects are determined based on the description information of the one or more visual objects; Based on the relationships between the one or more visual objects, determine the camera movement strategy for the image; Based on the correspondence between the plurality of image elements and the one or more visual objects, and the camera movement strategy of the image, the dynamic strategy of the image is determined.

8. The video generation method according to claim 1, wherein, The step of determining the image dynamization strategy based on the correspondence between the plurality of image elements and the one or more visual objects, and the descriptive information of the one or more visual objects, includes: Determine the duration of the audio corresponding to the image; The dynamic duration of the image is determined based on the duration of the audio. The dynamic strategy of the image is determined based on the correspondence between the multiple image elements and the one or more visual objects, the description information of the one or more visual objects, and the dynamic duration of the image.

9. The video generation method according to claim 1, wherein, The dynamic strategy includes a dynamic duration. Determining the dynamic strategy for the image based on the correspondence between the plurality of image elements and the one or more visual objects, and the descriptive information of the one or more visual objects, includes: The style of the plot text is determined based on the description information of the one or more visual objects; The dynamic duration of the image is determined based on the style of the plot text; The dynamic strategy of the image is determined based on the correspondence between the multiple image elements and the one or more visual objects, the description information of the one or more visual objects, and the dynamic duration of the image.

10. The video generation method according to claim 1, wherein, The process of determining the visual object from the plot text and the descriptive information of the visual object includes: Based on the plot text, information of multiple specified dimensions is generated, wherein the multiple specified dimensions include at least one of scene, environment, character, action, and emotion; Based on the attributes and relationships of the information in the multiple specified dimensions, the visualization object and the descriptive information of the visualization object are determined.

11. The video generation method according to claim 10, wherein, The one or more visualization objects include at least one of a person and a scene. The descriptive information of the person includes at least one of an emotion dimension and an action dimension. The descriptive information of the scene includes at least one of an environment dimension and an atmosphere dimension.

12. The video generation method according to claim 1, further comprising: Obtain the associated videos of the video, wherein the plot text corresponding to the associated video and the plot text corresponding to the video have a contextual relationship; The video is then spliced ​​with the associated video to generate a video corresponding to multiple plot texts with contextual relationships.

13. An electronic device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the video generation method as described in any one of claims 1 to 12 based on instructions stored in the memory.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video generation method of any one of claims 1 to 12.

15. A computer program product, when run on a computer, causes the computer to implement the video generation method according to any one of claims 1 to 12.

Images