Video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a video generation method and device, electronic equipment and a storage medium, and belongs to the technical field of multimedia. The method comprises the steps that an input image is recognized, an image description text is obtained, and the image description text is used for describing a scene in the image; the image description text and a scene deduction prompt are input into a large language model to obtain a plot text, the scene deduction prompt is used for prompting the large language model to deduce an event containing a scene shown by the image based on the image description text, and the plot text is used for describing the deduced event; and generating a video based on the image and the plot text, the video including events in the image and the plot text. According to the method, videos containing different scenes can be generated based on different images, the video generation diversity is improved, the scenes presented by the videos are more reasonable and accord with user intentions, and the video generation accuracy and the video quality can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of multimedia, and in particular, to a video generation method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the development of multimedia technology, more and more user objects are accustomed to sharing things through videos. Before publishing a video, a user object usually generates a video through a video editing application.

[0003] Among them, a video template library is usually built in the video editing application, and each video template is bound to one or more sets of fixed and unchangeable text prompts written by the developer. When a user object selects a certain video template (for example, a video of "Cyberpunk style"), the system generates a video based on the image input by the user object and the text prompts corresponding to the video template (for example, "Cyberpunk street, neon lights, rainy night, high contrast").

[0004] Since the text prompts of the video template are fixed and unchangeable, the videos generated for various images are similar, that is, the generated videos are relatively single. SUMMARY

[0005] The present disclosure provides a video generation method, device, electronic device, and storage medium, which can generate videos containing different scenes based on different images, improve the diversity of video generation, and make the scenes presented by the videos more reasonable, meet user intentions, and improve the accuracy and quality of video generation. The technical solutions of the present disclosure are as follows: According to an aspect of an embodiment of the present disclosure, a video generation method is provided, including: identifying an input image to obtain an image description text, the image description text being used to describe a scene in the image; inputting the image description text and a scene deduction prompt into a large language model to obtain a plot text, the scene deduction prompt being used to prompt the large language model to deduce an event containing the scene shown in the image based on the image description text, the plot text being used to describe the deduced event; generating a video based on the image and the plot text, the video containing the event in the image and the plot text.

[0006] According to another aspect of an embodiment of the present disclosure, a video generation device is provided, including: an identifying unit configured to perform identification on an input image to obtain an image description text, the image description text being used to describe a scene in the image; a plot generation unit configured to perform inputting the image description text and a scenario deduction prompt into the large language model to obtain a plot text, the scenario deduction prompt being used to prompt the large language model to deduce an event containing the scenario shown in the image based on the image description text, the plot text being used to describe the deduced event; a video generation unit configured to perform generating a video based on the image and the plot text, the video containing the image and the event in the plot text.

[0007] In some embodiments, the image description text comprises at least one of the following: a subject object in the image; a scene in the image; an atmosphere in the image; a pose of the subject object in the image, the pose comprising at least one of a behavior, an expression, and an appearance of the subject object; a location where the subject object is in the image; a relationship between subject objects in the image, the relationship comprising at least one of a positional relationship and an interaction relationship.

[0008] In some embodiments, the plot generation unit is configured to perform inputting the image description text and a first scenario deduction prompt into the large language model to obtain a first plot text, the first scenario deduction prompt being used to prompt the large language model to deduce an event that will occur after the scenario shown in the image based on the image description text, the first plot text being used to describe the event that will occur after the scenario shown in the image; the video generation unit is configured to perform generating a video with a first frame being the image based on the image and the first plot text.

[0009] In some embodiments, the plot generation unit is configured to perform inputting the image description text and a second scenario deduction prompt into the large language model to obtain a second plot text, the second scenario deduction prompt being used to prompt the large language model to deduce an event that has occurred before the scenario shown in the image based on the image description text, the second plot text being used to describe the event that has occurred before the scenario shown in the image; the video generation unit is configured to perform generating a video with a last frame being the image based on the image and the second plot text.

[0010] In some embodiments, the plot generation unit is configured to perform inputting the image description text and a third scenario deduction prompt into the large language model to obtain a third plot text, the third scenario deduction prompt being used to prompt the large language model to deduce events occurring before and after the scenario shown in the image based on the image description text, the third plot text being used to describe the events occurring before and after the scenario shown in the image. The video generation unit is configured to perform generating an intermediate frame as a video of the image based on the image and the third plot text.

[0011] In some embodiments, the third scenario deduction prompt includes a first sub-prompt and a second sub-prompt, the first sub-prompt being used to prompt the large language model to deduce events occurring after the scenario shown in the image based on the image description text, and the second sub-prompt being used to prompt the large language model to deduce events occurring before the scenario shown in the image based on the image description text. The plot generation unit is configured to perform processing the image description text and the third scenario deduction prompt by the large language model based on weights of the first sub-prompt and the second sub-prompt to obtain the third plot text. The third plot text includes a first sub-text generated based on the first sub-prompt and a second sub-text generated based on the second sub-prompt, the first sub-text being used to describe the events occurring after the scenario shown in the image, and the second sub-text being used to describe the events occurring before and after the scenario shown in the image, a data amount of the sub-text generated based on each sub-prompt being positively correlated with the weight of the sub-prompt.

[0012] In some embodiments, the weight of each sub-prompt is positively correlated with a duration requirement of a user object for the event corresponding to the sub-prompt.

[0013] In some embodiments, the scenario deduction prompt has multiple; The plot generation unit is configured to perform inputting the image description text and the scenario deduction prompt selected by the user object into the large language model to obtain the plot text in a case where any of the multiple scenario deduction prompts is selected by the user object.

[0014] In some embodiments, the scenario deduction prompt has multiple; The plot generation unit is configured to perform determining a target scenario deduction prompt from the multiple scenario deduction prompts based on a behavior feature of a user object, the behavior feature being used to indicate a scenario deduction prompt that is usually used by the user object in a historical event segment; inputting the image description text and the target scenario deduction prompt into the large language model to obtain the plot text.

[0015] In some embodiments, the scenario generation unit is configured to perform inputting the image description text and a scenario deduction prompt into the large language model to obtain a plurality of scenario texts; The video generation unit is configured to perform, in a case where any of the plurality of scenario texts is selected by a user object, generating a video based on the image and the scenario text selected by the user object.

[0016] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, which includes: one or more processors; a memory for storing program code executable by the processor; wherein the processor is configured to execute the program code to implement the video generation method described above.

[0017] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which, when program code in the computer readable storage medium is executed by a processor of an electronic device, enables the electronic device to perform the video generation method described above.

[0018] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, which includes computer programs / instructions that, when executed by a processor, implement the video generation method described above.

[0019] The embodiments of the present disclosure provide a video generation method, in the process of generating a video based on an image, the image is first recognized to obtain an image description text to determine the scenario in the image, and then the image description text and a scenario deduction prompt are input into a large language model, through the scenario deduction prompt, the large language model is instructed to deduce an event that can occur when the scenario shown in the image can occur based on the image description text. Since the event is deduced based on the image description text, it is strongly related to the scenario in the image, so that the video generated based on the scenario text describing the event and the image is also strongly related to the scenario in the image, and the event occurring in the video matches the scenario in the image. Therefore, not only can videos containing different scenarios be generated based on different images, improving the diversity of video generation, but also the scenarios presented in the video are more reasonable, meeting the user's intention, which can improve the accuracy and quality of video generation.

[0020] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are incorporated into and constitute a part of the specification, illustrate preferred embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure and do not limit the present disclosure in any inappropriated way.

[0022] Figure 1 is a schematic diagram of an implementation environment of a video generation method according to an example embodiment.

[0023] Figure 2 is a flowchart of a video generation method according to an example embodiment.

[0024] Figure 3 is a flowchart of another video generation method according to an example embodiment.

[0025] Figure 4 is a block diagram of a video generation apparatus according to an example embodiment.

[0026] Figure 5 is a block diagram of a terminal according to an example embodiment.

[0027] Figure 6 is a block diagram of a server according to an example embodiment. DETAILED DESCRIPTION

[0028] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0029] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the present disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards in relevant countries and regions. For example, the images and scenario deduction prompts involved in the present disclosure are obtained under full authorization.

[0031] Figure 1 is an implementation environment schematic diagram of a video generation method according to an exemplary embodiment. Taking the electronic device provided as a server as an example, referring to Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102.

[0032] In some embodiments, the terminal 101 is at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player, an MP4 player, a laptop computer, and the like. The terminal 101 has an application program supporting video generation installed and running thereon. The application program can be a multimedia application program, a clipping application program, a smart assistant, and the like, which are not limited in the embodiments of the present disclosure. A user object can log in to the application program through the terminal 101 to obtain the services provided by the application program. The terminal 101 can be connected to the server 102 through a wireless network or a wired network, and then can send images used for video generation to the server 102, and the server 102 generates a video based on the images.

[0033] The terminal 101 generally refers to one of a plurality of terminals, and the embodiments are exemplified by the terminal 101. Those skilled in the art can know that the number of the above terminals can be more or less. For example, the above terminals can be several, or the above terminals can be dozens or hundreds, or more, and the number of terminals and the type of devices are not limited in the embodiments of the present disclosure.

[0034] In some embodiments, the server 102 is a stand-alone physical server, and can also be a server cluster or a distributed system composed of a plurality of physical servers, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), big data and artificial intelligence platforms, and the like. The server 102 is used to provide background services for the application program supporting video generation. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing.

[0035] Figure 2 is a flowchart of a video generation method according to an exemplary embodiment, referring to Figure 2 , the video generation method is applied to a server, and includes the following steps: In step 201, the server identifies the input image to obtain an image description text, and the image description text is used to describe the scene in the image.

[0036] In the embodiment of the present disclosure, the server obtains an image input by a user object from a terminal, identifies the image to obtain an image description text. The server can use any image recognition model to perform the identification, which is not limited in the embodiment of the present disclosure. The image description text can include subject objects, scenes, and current behaviors of the subject objects in the image, so as to reflect the scene shown in the image.

[0037] In step 202, the server inputs the image description text and a scene deduction prompt into the large language model to obtain a plot text. The scene deduction prompt is used to prompt the large language model to deduce an event containing the scene shown in the image based on the image description text, and the plot text is used to describe the deduced event.

[0038] In the embodiment of the present disclosure, the server inputs the image description text and the scene deduction prompt into the large language model, and processes the image description text and the scene deduction prompt by the large language model to obtain the plot text. In the process of generating the plot text by the large language model, the scene deduction prompt can instruct the large language model to deduce (or predict) an event containing the scene shown in the image based on the image description text. That is, the large language model is used to predict an event that can exist (or occur) when the scene shown in the image appears. The event can be a preceding cause of the scene shown in the image, or a subsequent event of the scene shown in the image, which is not limited in the embodiment of the present disclosure. In other words, the scene shown in the image can appear at the end stage of the event, or at the start stage of the event, which is not limited in the embodiment of the present disclosure. Then, for the deduced event, the large language model describes it in natural language to output the plot text.

[0039] The large language model described above can be any architecture model, which is not limited in the embodiment of the present disclosure.

[0040] In step 203, the server generates a video based on the image and the plot text, and the video contains the image and the event in the plot text.

[0041] In the embodiment of the present disclosure, the server inputs the image input by the user object and the plot text into the video output model, and processes the image and the plot text by the video generation model. The plot text is used to instruct the video generation model to generate a video containing the visual elements in the image and the event in the plot text based on the image. Then, the server sends the generated video to the terminal, and the terminal shows the video to the user object.

[0042] The embodiment of the present disclosure provides a video generation method. In the process of generating a video based on an image, the image is first identified to obtain an image description text, so as to determine a scene in the image. Then, the image description text and a scene deduction prompt are input into a large language model. Through the scene deduction prompt, the large language model is instructed to deduce an event that may occur when the scene shown in the image appears based on the image description text. Since the event is deduced based on the image description text, the event is strongly related to the scene in the image. Therefore, a video generated based on a plot text describing the event and the image is also strongly related to the scene in the image, and the event occurring in the video matches the scene in the image. As a result, not only can videos containing different scenes be generated based on different images, thereby improving the diversity of video generation, but also the scene presented by the video is more reasonable and meets the user's intention, thereby improving the accuracy of video generation and the quality of the video.

[0043] In some embodiments, the image description text comprises at least one of the following: a subject object in the image; a scene in the image; an atmosphere in the image; a posture of the subject object in the image, the posture comprising at least one of a behavior, an expression and an appearance of the subject object; a position of the subject object in the image; an association relationship between the subject objects in the image, the association relationship comprising at least one of a positional relationship and an interaction relationship.

[0044] The scheme provided by the embodiment of the present disclosure can make the image description text more accurately describe the scene in the image, so that a more reasonable event can be deduced based on the image description text in the subsequent process, thereby generating a video that is more consistent with the scene in the image.

[0045] In some embodiments, the image description text and the scene deduction prompt are input into the large language model to obtain a plot text, comprising: the image description text and a first scene deduction prompt are input into the large language model to obtain a first plot text, the first scene deduction prompt is used to prompt the large language model to deduce an event that will occur after the scene shown in the image based on the image description text, and the first plot text is used to describe the event that will occur after the scene shown in the image; the image and the plot text are used to generate a video, comprising: the image and the first plot text are used to generate a video whose first frame is the image.

[0046] The scheme provided by the embodiment of the present disclosure indicates the large language model to deduce the event that may occur after the scenario shown in the image based on the image description text through the first scenario deduction prompt. Since the event is deduced based on the image description text and is strongly related to the scenario in the image, the video generated based on the plot text describing the event and the image is also strongly related to the scenario in the image, and the above image in the video is the first video frame, and the subsequent video frames are used to present the deduced event that may occur afterwards. Therefore, not only can videos containing different scenarios be generated based on different images, improving the diversity of video generation, but also the scenario presented by the video is more reasonable, meets the user's intention, and can improve the accuracy of video generation and video quality.

[0047] In some embodiments, inputting the image description text and the scenario deduction prompt into the large language model obtains a plot text, including: Inputting the image description text and the second scenario deduction prompt into the large language model obtains a second plot text, and the second scenario deduction prompt is used to prompt the large language model to deduce an event that occurred before the scenario shown in the image based on the image description text. The second plot text is used to describe the event that occurred before the scenario shown in the image. Generating a video based on the image and the plot text includes: Generating a video with the image as the tail frame based on the image and the second plot text.

[0048] The scheme provided by the embodiment of the present disclosure indicates the large language model to deduce the event that may occur before the scenario shown in the image based on the image description text through the second scenario deduction prompt. Since the event is deduced based on the image description text and is strongly related to the scenario in the image, the video generated based on the plot text describing the event and the image is also strongly related to the scenario in the image, and the above image in the video is the last video frame, and the previous video frames are used to present the deduced event that may occur before. Therefore, not only can videos containing different scenarios be generated based on different images, improving the diversity of video generation, but also the scenario presented by the video is more reasonable, meets the user's intention, and can improve the accuracy of video generation and video quality.

[0049] In some embodiments, inputting the image description text and the scenario deduction prompt into the large language model obtains a plot text, including: Inputting the image description text and the third scenario deduction prompt into the large language model obtains a third plot text, and the third scenario deduction prompt is used to prompt the large language model to deduce the event that occurs before and after the scenario shown in the image based on the image description text. The third plot text is used to describe the event that occurs before and after the scenario shown in the image. Generating a video based on the image and the plot text includes: generate a video with the image as an intermediate frame based on the image and the third plot text.

[0050] The scheme provided by the embodiments of the present disclosure can indicate the large language model to deduce the events that can occur before and after the scenario shown in the image based on the image description text through the third scenario deduction prompt. Since the events are deduced based on the image description text and are strongly related to the scenario in the image, the video generated based on the plot text describing the events and the image is also strongly related to the scenario in the image. Moreover, the above-mentioned image in the video is an intermediate video frame, the previous video frame is used to present the deduced event that can have occurred before, and the subsequent video frame is used to present the deduced event that can have occurred after. Therefore, not only can videos containing different scenarios be generated based on different images, improving the diversity of video generation, but also the scenario presented by the video is more reasonable, meeting the user's intention, and the accuracy and quality of video generation can be improved.

[0051] In some embodiments, the third scenario deduction prompt includes a first sub-prompt and a second sub-prompt, the first sub-prompt is used to prompt the large language model to deduce the event that will occur after the scenario shown in the image based on the image description text, and the second sub-prompt is used to prompt the large language model to deduce the event that has occurred before the scenario shown in the image; inputting the image description text and the third scenario deduction prompt into the large language model to obtain a third plot text, comprising: processing the image description text and the third scenario deduction prompt by using the large language model based on the weight of the first sub-prompt and the weight of the second sub-prompt to obtain the third plot text; wherein the third plot text includes a first subtext generated based on the first sub-prompt and a second subtext generated based on the second sub-prompt, the first subtext is used to describe the event that will occur after the scenario shown in the image, and the second subtext is used to describe the event that occurs before and after the scenario shown in the image, and the data amount of the subtext generated based on each sub-prompt is positively correlated with the weight of the sub-prompt.

[0052] The scheme provided by the embodiments of the present disclosure can deduce the events before and after the scenario shown in the image by assigning weights to the sub-prompt in the third scenario deduction prompt for deducing each event, so as to deduce the events before and after the scenario shown in the image to different degrees. Wherein, the higher the weight of the sub-prompt for deducing the event, the more data amount of the text for describing the event obtained, that is, the more or more detailed events deduced, so as to occupy more space (or frame number) in the subsequently generated video, thereby facilitating the control of the degree of event deduction, and further being able to control the position (or timing) of the image in the video, that is, better controllability, which is beneficial to improve the accuracy and quality of video generation.

[0053] In some embodiments, the weight of each sub-prompt is positively correlated with the duration requirement of the user object for the sub-prompt corresponding event.

[0054] The scheme provided by the embodiments of the present disclosure can determine the weight of the sub-prompt for deducing each event based on the duration requirement of the user object for the event. The longer the duration requirement of the user object for the event, the higher the weight of the sub-prompt for deducing the event, thereby facilitating the large language model to perform deeper deduction and obtain more or more detailed events, so as to occupy more duration in the subsequently generated video, meet the video generation requirement of the user object, and improve the accuracy and quality of video generation.

[0055] In some embodiments, there are multiple scenario deduction prompts. The image description text and the scenario deduction prompt are input into the large language model to obtain a plot text, including: In the case where any of the multiple scenario deduction prompts is selected by the user object, the image description text and the scenario deduction prompt selected by the user object are input into the large language model to obtain a plot text.

[0056] The scheme provided by the embodiments of the present disclosure can provide multiple scenario deduction prompts to the user object, so that the user object can select a scenario deduction prompt according to the user's own needs to instruct the large language model to deduce reasonable events based on the image description text, thereby generating a video containing reasonable events, meeting the video generation requirement of the user object, and having stronger controllability and flexibility, which can improve the accuracy and quality of video generation.

[0057] In some embodiments, there are multiple scenario deduction prompts. The image description text and the scenario deduction prompt are input into the large language model to obtain a plot text, including: Based on the behavior characteristics of the user object, a target scenario deduction prompt is determined from the multiple scenario deduction prompts. The behavior characteristics are used to indicate the scenario deduction prompt that the user object is used to in the historical event segment. The image description text and the target scenario deduction prompt are input into the large language model to obtain a plot text.

[0058] The scheme provided by the embodiments of the present disclosure can determine the target scenario deduction prompt according to the behavior characteristics of the user object, so that the target scenario deduction prompt meets the scenario deduction prompt that the user object is used to, thereby being more likely to meet the intention of the user object, facilitating the subsequent generation of a video that meets the user's requirements, and ensuring the accuracy of video generation; and without manual operation of the user object, the efficiency of video generation can be improved.

[0059] In some embodiments, the image description text and the scenario deduction prompt are input into the large language model to obtain a plot text, including: The image description text and the scenario deduction prompt are input into the large language model to obtain a plurality of plot texts. Based on the image and the plot text, a video is generated, including: In a case where any plot text in the plurality of plot texts is selected by a user object, a video is generated based on the image and the plot text selected by the user object.

[0060] The scheme provided by the embodiments of the present disclosure can generate a plurality of plot texts for a user object to select through a large language model, and in a case where any plot text is selected by the user object, a video is generated based on the plot text, that is, the generated video conforms to the intention of the user object, the accuracy of video generation is improved, and redundant videos do not need to be generated, thereby saving resources.

[0061] The above Figure 2 The above Figure 3 is a flowchart of another video generation method according to an example embodiment. Taking an electronic device provided as a server as an example, referring to Figure 3 , the method includes: In step 301, the server identifies an input image to obtain an image description text, and the image description text is used to describe a scenario in the image.

[0062] In the embodiments of the present disclosure, the server identifies an image input by a user object to obtain an image description text used to describe a scenario in the image. The specific content of the image description text in the embodiments of the present disclosure is not limited.

[0063] In some embodiments, the above image description text can include at least one of the following.

[0064] The first item is a subject object in the image. The subject object can be an animal, a person, a virtual character, etc., and the embodiments of the present disclosure do not limit this. There can be one, two or more subject objects in the image, and the embodiments of the present disclosure do not limit this. For example, the subject objects in the image are “a brown cat” and “a girl wearing a red hat”.

[0065] The second item is a scene in the image. The scene refers to the environment in which the subject object is located in the image. The scene can be an indoor scene, an outdoor scene, a natural scenery, a building scene, etc., and the embodiments of the present disclosure do not limit this. For example, the scene in the image is “a wooden window sill” or “a bustling city street at night”.

[0066] Third, the atmosphere in the image. The atmosphere refers to a comprehensive concept, which is not a specific element, but a unified, perceptible emotion, feeling or mood evoked after all visual elements are combined together. For example, the atmosphere in the image is "a gloomy rainy day", "mysterious and suspenseful".

[0067] Fourth, the pose of the subject object in the image. The pose includes at least one of the behavior, expression and appearance of the subject object. The behavior of the subject object can include information such as the action and posture of the subject object. The appearance of the subject object includes information such as the accessories, clothing and hairstyle carried by the subject object.

[0068] Fifth, the position of the subject object in the image.

[0069] Sixth, the association relationship between the subject objects in the image. The association relationship includes at least one of the position relationship and the interaction relationship. The position relationship between the subject objects can be a distance-based relationship (such as closeness or distance), a direction-based relationship (such as left-right relationship, up-down relationship) or a direction-based relationship (such as facing or facing away), which is not limited by the embodiments of the present disclosure. The interaction relationship between the subject objects can be a behavior interaction relationship (such as cooperation, confrontation or dependence) or a visual interaction relationship (such as looking at each other or ignoring each other), which is not limited by the embodiments of the present disclosure. For example, "the cat is looking out the window", "the girl is smiling at the camera".

[0070] The scheme provided by the embodiments of the present disclosure can make the image description text more accurately describe the scene in the image, so that a more reasonable event can be deduced based on the image description text in the subsequent process, so as to generate a video more consistent with the scene in the image.

[0071] The image description text can be a structured text description, such as a set of labels; or the image description text can also be an unstructured text description, such as a text described in natural language, which is not limited by the embodiments of the present disclosure.

[0072] For example, for an image of a cat, the output image description text can be: {subject: "cat"; position: "window sill"; background: "rainy street"; behavior: "staring outwards"; atmosphere: "quiet, slightly melancholy"}.

[0073] In step 302, the server inputs the image description text and the scene deduction prompt into the large language model to obtain a plot text, and the scene deduction prompt is used to prompt the large language model to deduce (or predict) an event containing the scene shown in the image based on the image description text, and the plot text is used to describe the deduced event.

[0074] In the embodiments of the present disclosure, in the process of generating the plot text by the large language model, the scene deduction prompt can instruct the large language model to deduce (or predict) an event containing the scene shown in the image based on the image description text. This is not a specific video generation instruction, but an open-ended and creative “seed”. For example, the scene deduction prompt is “According to the following scene description, please imagine an event that will happen in the next second, full of surprises or drama. Describe it in vivid and story-like language”.

[0075] The scene deduction prompt is a pre-set prompt text. After the image description text is identified, the server inputs the image description text and the scene deduction prompt into the large language model, and processes the image description text and the scene deduction prompt by the large language model to obtain a plot text. The large language model can output one plot text or multiple plot texts at a time, and the number of plot texts is not limited in the embodiments of the present disclosure.

[0076] For example, for an image of a cat, the large language model can generate the following several completely different plot texts: Plot text A (fantasy): The cat staring at the raindrops suddenly discovers that between the puddles on the street, there is a small, shiny elf jumping. Its eyes instantly widen, full of disbelief, and the tail also excitedly starts to swing; Plot text B (philosophy): Suddenly, the reflection on the window glass changes, and in an instant, the cat no longer sees itself, but a majestic lion, ready to make a silent roar at the storm outside the window; Plot text C (warm): “A paper boat that came from nowhere, drifted along the water flow beside the street. When it passed under a street lamp, the cat saw a small person made of light on the boat, waving at it.

[0077] There can be multiple scene deduction prompts. In the process of generating the plot text, the server can select a scene deduction prompt from multiple scene deduction texts and input it into the large language model to generate the plot text.

[0078] In some embodiments, the selected scenario deduction prompt is a first scenario deduction prompt. The first scenario deduction prompt is used to prompt the large language model to infer, based on the image description text, an event that will occur after the scenario shown in the image. Accordingly, step 302 includes: the server inputs the image description text and the first scenario deduction prompt into the large language model to obtain a first plot text. The first plot text is used to describe the event that will occur after the scenario shown in the image. The solution provided by the embodiments of this disclosure, through the first scenario deduction prompt, instructs the large language model to infer, based on the image description text, a possible event that will occur after the scenario shown in the image occurs. Since this event is deduced based on the image description text and is strongly correlated with the scenario in the image, it is beneficial to ensure that the subsequent video generated based on the plot text describing the event and the image is also strongly correlated with the scenario in the image. The scenario presented in the video is more reasonable, conforms to the user's intention, and can improve the accuracy of video generation and video quality.

[0079] In other embodiments, the selected scenario deduction prompt is a second scenario deduction prompt. The second scenario deduction prompt is used to prompt the large language model to infer events that occurred before the scenario shown in the image based on the image description text. Accordingly, step 302 includes: the server inputting the image description text and the second scenario deduction prompt into the large language model to obtain a second plot text. The second plot text is used to describe events that occurred before the scenario shown in the image. The solution provided by this disclosure, through the second scenario deduction prompt, instructs the large language model to infer events that might have occurred before the scenario shown in the image could occur based on the image description text. Since this event is deduced based on the image description text and is strongly correlated with the scenario in the image, it is beneficial to ensure that the subsequent plot text and video generated based on the image describing the event are also strongly correlated with the scenario in the image. The scenario presented in the video is more reasonable, conforms to the user's intent, and can improve the accuracy and quality of video generation.

[0080] In other embodiments, the selected scenario deduction prompt is a third scenario deduction prompt. The third scenario deduction prompt is used to prompt the large language model to infer events occurring before and after the scenario shown in the image based on the image description text. Accordingly, step 302 includes: the server inputs the image description text and the third scenario deduction prompt into the large language model to obtain a third plot text. The third plot text is used to describe the events occurring before and after the scenario shown in the image. The solution provided by this disclosure, through the third scenario deduction prompt, instructs the large language model to infer possible events before and after the scenario shown in the image based on the image description text. Since this event is deduced based on the image description text and is strongly correlated with the scenario in the image, it is beneficial to ensure that the subsequent plot text and video generated based on the image describing the event are also strongly correlated with the scenario in the image. The scenario presented in the video is more reasonable, conforms to the user's intent, and can improve the accuracy and quality of video generation.

[0081] The third scenario inference prompt can include a first sub-prompt and a second sub-prompt. The first sub-prompt indicates that the large language model infers events that will occur after the scenario shown in the image based on the image description text. The second sub-prompt indicates that the large language model infers events that occurred before the scenario shown in the image based on the image description text. The server then generates the third scenario text by processing the image description text and the third scenario inference prompt using the large language model, based on the weights of the first and second sub-prompts. The third scenario text includes a first sub-text generated based on the first sub-prompt and a second sub-text generated based on the second sub-prompt. The first sub-text describes events that will occur after the scenario shown in the image. The second sub-text describes events that occur before and after the scenario shown in the image. The amount of data in the sub-text generated based on each sub-prompt is positively correlated with the weight of the sub-prompt.

[0082] The solution provided in this embodiment of the present disclosure, for the deduction of events before and after the scene shown in the image, can assign weights to the sub-tips used to deduce each event in the third scene deduction prompt, so as to perform different degrees of deduction of events before and after the scene shown in the image. The higher the weight of the sub-tips used to deduce the events, the more data of text used to describe the events are obtained, that is, the more or more detailed the events are deduced, and the more space (or frames) they occupy in the subsequently generated video. This makes it easier to control the degree of event deduction, and thus can control the position (or timing of appearance) of the image in the video, that is, better controllability, which is conducive to improving the accuracy of video generation and video quality.

[0083] The weight of each sub-hint is positively correlated with the user's duration requirement for the corresponding event. Specifically, before generating the plot text, the terminal can display the first scenario deduction hint, the second scenario deduction hint, and the third scenario deduction hint for the user to choose from. If the third scenario deduction hint is selected by the user, the terminal can display a first duration setting control and a second duration setting control. The first duration setting control is used to obtain the user's duration requirement for events occurring after the scene shown in the image, and the second duration setting control is used to obtain the user's duration requirement for events occurring before the scene shown in the image. Therefore, the server can determine the weight of the first sub-hint and the weight of the second sub-hint based on the durations in the first and second duration setting controls. The weight of the first sub-hint is positively correlated with the duration in the first duration setting control, and the weight of the second sub-hint is positively correlated with the duration in the second duration setting control. For example, the weight of the first sub-hint is equal to the proportion of the duration in the first duration setting control to the total duration; the weight of the second sub-hint is equal to the proportion of the duration in the second duration setting control to the total duration. The total duration is the total duration of the video, which is equal to the sum of the duration set in the first duration setting control and the duration set in the second duration setting control.

[0084] The solution provided in this disclosure, in determining the weight of sub-cues used to infer each event, can be based on the user's duration requirement for the event. The longer the user's duration requirement for the event, the higher the weight of the sub-cues used to infer that event. This helps to instruct the large language model to perform deeper inferences, obtain more or more detailed events, so that they can occupy more time in the subsequently generated video, meet the user's video generation requirements, and improve the accuracy and quality of video generation.

[0085] There can be multiple scenario deduction prompts. During the generation of the plot text, the server can select one scenario deduction prompt from multiple scenario deduction texts and input it into the large language model to generate the plot text. The selection of the scenario deduction prompt can be random, specified by the user, or based on certain rules; this embodiment does not limit the selection in this way.

[0086] In some embodiments, the scenario deduction prompts used to generate the plot text are specified by the user. Accordingly, when any of the multiple scenario deduction prompts is selected by the user, the server inputs the image description text and the scenario deduction prompt selected by the user into the large language model to obtain the plot text. The solution provided by this disclosure embodiment can provide multiple scenario deduction prompts to the user, allowing the user to select one according to their needs to instruct the large language model to deduce reasonable events based on the image description text, thereby generating a video containing reasonable events. This meets the user's video generation needs, offers greater controllability and flexibility, and improves the accuracy and quality of video generation.

[0087] In other embodiments, the scenario deduction prompts used to generate the plot text are automatically determined based on the characteristics of the user object. Accordingly, the server determines the target scenario deduction prompt from multiple scenario deduction prompts based on the user object's behavioral characteristics. Behavioral characteristics are used to indicate the scenario deduction prompts that the user object habitually uses within a historical event segment. Then, the server inputs the image description text and the target scenario deduction prompt into a large language model to obtain the plot text. The solution provided by the embodiments of this disclosure can determine the target scenario deduction prompt based on the user object's behavioral characteristics, making the target scenario deduction prompt conform to the user object's habitual scenario deduction prompts, thus increasing the likelihood of conforming to the user object's intent, thereby facilitating the subsequent generation of videos that meet user needs and ensuring the accuracy of video generation; and it does not require manual operation by the user object, thus improving the efficiency of video generation.

[0088] In step 303, the server inputs images and narrative text into the video generation model to obtain a video containing the events from the images and narrative text.

[0089] In this embodiment of the disclosure, after generating the plot text, the server inputs the image and plot text from the user into the video generation model. The video generation model processes the image and plot text to obtain the video. Different plot texts generate different videos.

[0090] For example, for the above-mentioned question text A, the video generation model needs to generate a "glowing sprite," create an animation of it "jumping between puddles," and subtly adjust the cat's facial expression to make its "eyes widen" and its tail "start wagging." For the above-mentioned question text B, the video generation model needs to "render a dynamic, lion's reflection on the glass."

[0091] In some embodiments, the server generates a video whose first frame is an image, based on an image and first plot text. In the video generated by this method, the image input by the user is the first video frame, and subsequent video frames are used to present events that may occur as described in the first plot text. This not only enables the generation of videos containing different scenarios based on different images, increasing the diversity of video generation, but also makes the scenarios presented in the video more reasonable and consistent with the user's intent, thereby improving the accuracy and quality of video generation.

[0092] In other embodiments, the server generates a video whose last frame is an image, based on an image and second plot text. In the video generated by this method, the image input by the user object is the last video frame, and the preceding video frames are used to present events that might have occurred previously as described in the second plot text. This not only enables the generation of videos containing different scenarios based on different images, increasing the diversity of video generation, but also makes the scenarios presented in the video more reasonable and consistent with the user's intent, thereby improving the accuracy and quality of video generation.

[0093] In other embodiments, the server generates a video with intermediate frames consisting of images and third-party plot text. In this method, the user-input image is the intermediate video frame, and subsequent video frames present events that might occur later as described in the third-party plot text, while earlier video frames present events that might occur earlier as described in the third-party plot text. This not only enables the generation of videos containing different scenarios based on different images, increasing the diversity of video generation, but also makes the scenarios presented in the videos more reasonable and consistent with user intent, thereby improving the accuracy and quality of video generation.

[0094] In some embodiments, the large language model generates multiple plot texts. When any one of these plot texts is selected by the user, the server generates a video based on the image and the selected plot text. The solution provided by this disclosure can generate multiple plot texts for the user to choose from using a large language model. If the user selects any plot text, a video is then generated based on that plot text. This ensures that the generated video matches the user's intent, improves the accuracy of video generation, and avoids generating redundant videos, thus saving resources.

[0095] This disclosure provides a video generation method. In the process of generating video based on images, the image is first identified to obtain image description text to determine the scene in the image. Then, the image description text and scene deduction prompts are input into a large language model. Through the scene deduction prompts, the large language model is instructed to deduce the possible events that may occur when the scene shown in the image occurs, based on the image description text. Since the event is deduced based on the image description text, it is strongly correlated with the scene in the image. This makes the video generated based on the plot text describing the event and the image also strongly correlated with the scene in the image. Furthermore, the events that occur in the video match the scene in the image. Therefore, not only can videos containing different scenes be generated based on different images, improving the diversity of video generation, but the scenes presented in the video are also more reasonable and in line with user intent, which can improve the accuracy and quality of video generation.

[0096] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0097] Figure 4 This is a block diagram illustrating a video generation apparatus according to an exemplary embodiment. See also Figure 4 The video generation device includes: a recognition unit 401, a plot generation unit 402, and a video generation unit 403.

[0098] The recognition unit 401 is configured to perform recognition of the input image to obtain image description text, which is used to describe the scene in the image; The plot generation unit 402 is configured to input image description text and scenario inference prompts into the large language model to obtain plot text. The scenario inference prompts are used to prompt the large language model to infer events containing the scenario shown in the image based on the image description text, and the plot text is used to describe the inferred events. The video generation unit 403 is configured to generate a video based on images and plot text, wherein the video contains events from the images and plot text.

[0099] In some embodiments, the image description text includes at least one of the following: The main object in the image; The scene in the image; The atmosphere in the image; The pose of the main object in the image, which includes at least one of the main object's behavior, expression, and appearance; The location of the main object in the image; The relationships between main objects in an image, including at least one of positional relationships and interaction relationships.

[0100] In some embodiments, the plot generation unit 402 is configured to input image description text and a first scenario inference prompt into a large language model to obtain a first plot text. The first scenario inference prompt is used to prompt the large language model to infer an event that will occur after the scenario shown in the image based on the image description text. The first plot text is used to describe the event that will occur after the scenario shown in the image. The video generation unit 403 is configured to generate a video whose first frame is an image, based on an image and a first plot text.

[0101] In some embodiments, the plot generation unit 402 is configured to input image description text and a second scenario inference prompt into a large language model to obtain a second plot text. The second scenario inference prompt is used to prompt the large language model to infer events that occurred before the scenario shown in the image based on the image description text. The second plot text is used to describe events that occurred before the scenario shown in the image. The video generation unit 403 is configured to perform image-based and second-story text-based video generation with an image as the last frame.

[0102] In some embodiments, the plot generation unit 402 is configured to input image description text and third scenario inference prompts into a large language model to obtain third plot text. The third scenario inference prompts are used to prompt the large language model to infer events that occur before and after the scenario shown in the image based on the image description text. The third plot text is used to describe the events that occur before and after the scenario shown in the image. The video generation unit 403 is configured to perform video generation based on images and third-party narrative text, with intermediate frames being images.

[0103] In some embodiments, the third scenario inference prompt includes a first sub-prompt and a second sub-prompt. The first sub-prompt is used to prompt the large language model to infer events that will occur after the scenario shown in the image based on the image description text, and the second sub-prompt is used to prompt events that have occurred before the scenario shown in the image based on the image description text. The plot generation unit 402 is configured to perform weighting based on the first sub-cue and the second sub-cue, and to process the image description text and the third scenario inference cue using a large language model to obtain the third plot text; The third plot text includes a first sub-text generated based on the first sub-hint and a second sub-text generated based on the second sub-hint. The first sub-text describes events that will occur after the scene shown in the image, and the second sub-text describes events that will occur before and after the scene shown in the image. The amount of data in the sub-text generated based on each sub-hint is positively correlated with the weight of the sub-hint.

[0104] In some embodiments, the weight of each sub-prompt is positively correlated with the duration requirement of the user object for the event corresponding to the sub-prompt.

[0105] In some embodiments, there are multiple scenario simulation prompts; The plot generation unit 402 is configured to, when any of the multiple scenario deduction prompts is selected by the user object, input image description text and the scenario deduction prompt selected by the user object into the large language model to obtain plot text.

[0106] In some embodiments, there are multiple scenario simulation prompts; The plot generation unit 402 is configured to perform behavior features based on user objects, determine the target scenario inference prompt from multiple scenario inference prompts, and use the behavior features to indicate the scenario inference prompts commonly used by user objects within the historical event segment; input the image description text and the target scenario inference prompt into the large language model to obtain the plot text.

[0107] In some embodiments, the plot generation unit 402 is configured to input image description text and scenario deduction prompts into a large language model to obtain multiple plot texts; The video generation unit 403 is configured to generate a video based on an image and the selected plot text when any one of the multiple plot texts is selected by a user object.

[0108] This disclosure provides a video generation method. In the process of generating video based on images, the image is first identified to obtain image description text to determine the scene in the image. Then, the image description text and scene deduction prompts are input into a large language model. Through the scene deduction prompts, the large language model is instructed to deduce the possible events that may occur when the scene shown in the image occurs, based on the image description text. Since the event is deduced based on the image description text, it is strongly correlated with the scene in the image. This makes the video generated based on the plot text describing the event and the image also strongly correlated with the scene in the image. Furthermore, the events that occur in the video match the scene in the image. Therefore, not only can videos containing different scenes be generated based on different images, improving the diversity of video generation, but the scenes presented in the video are also more reasonable and in line with user intent, which can improve the accuracy and quality of video generation.

[0109] It should be noted that the video generation apparatus provided in the above embodiments is only illustrated by the division of the above functional units when generating video based on images. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the video generation apparatus and the video generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0110] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0111] When an electronic device is provided as a terminal, Figure 5 This is a block diagram illustrating a terminal 500 according to an exemplary embodiment. The terminal... Figure 5 A structural block diagram of a terminal 500 provided in an exemplary embodiment of this disclosure is shown. The terminal 500 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 500 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0112] Typically, terminal 500 includes a processor 501 and a memory 502.

[0113] Processor 501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 501 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0114] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 502 are used to store at least one computer program, which is executed by the processor 501 to implement the video generation method provided in the method embodiments of this application.

[0115] In some embodiments, the terminal 500 may also optionally include a peripheral device interface 503 and at least one peripheral device. The processor 501, memory 502, and peripheral device interface 503 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 503 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507, and a power supply 508.

[0116] Peripheral device interface 503 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 501 and memory 502. In some embodiments, processor 501, memory 502 and peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 501, memory 502 and peripheral device interface 503 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0117] The radio frequency (RF) circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 504 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 504 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0118] Display screen 505 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 505 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 501 for processing. In this case, display screen 505 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 505, disposed on the front panel of terminal 500; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 500 or in a folded design; in other embodiments, display screen 505 may be a flexible display screen, disposed on a curved or folded surface of terminal 500. Furthermore, display screen 505 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 505 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0119] The camera assembly 506 is used to acquire images or videos. In some embodiments, the camera assembly 506 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash is a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0120] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 501 for processing, or input to the radio frequency circuit 504 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 500. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 507 may also include a headphone jack.

[0121] Power supply 508 is used to power the various components in terminal 500. Power supply 508 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 508 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0122] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on terminal 500, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0123] When electronic devices are provided as servers, Figure 6 This is a block diagram illustrating a server 600 according to an exemplary embodiment. The server 600 can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 601 and one or more memories 602. The memories 602 store at least one line of program code, which is loaded and executed by the processor 601 to implement the video generation methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 600 may also include other components for implementing device functions, which will not be elaborated upon here.

[0124] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as memory 502 or memory 602 including instructions, which can be executed by processor 501 of terminal 500 or processor 601 of server 600 to complete the video generation method described above. Optionally, the computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0125] A computer program product includes a computer program / instructions that, when executed by a processor, implement the video generation method described above.

[0126] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0127] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video generation method, characterized in that, The method includes: The input image is identified to obtain image description text, which is used to describe the scene in the image; The image description text and scenario inference prompts are input into the large language model to obtain the plot text. The scenario inference prompts are used to prompt the large language model to infer an event containing the scenario shown in the image based on the image description text. The plot text is used to describe the inferred event. Based on the image and the story text, a video is generated, which includes the events described in the image and the story text.

2. The video generation method according to claim 1, characterized in that, The image description text includes at least one of the following: The main object in the image; The scene in the image; The atmosphere in the image; The pose of the main object in the image, wherein the pose includes at least one of the main object’s behavior, expression, and appearance; The location of the main object in the image; The relationships between the main objects in the image include at least one of positional relationships and interaction relationships.

3. The video generation method according to claim 1, characterized in that, The process of inputting the image description text and scenario inference prompts into the large language model to obtain the plot text includes: The image description text and the first scenario inference prompt are input into the large language model to obtain the first plot text. The first scenario inference prompt is used to prompt the large language model to infer the events that will happen after the scenario shown in the image based on the image description text. The first plot text is used to describe the events that will happen after the scenario shown in the image. The process of generating a video based on the image and the narrative text includes: Based on the image and the first plot text, a video with the image as the first frame is generated.

4. The video generation method according to claim 1, characterized in that, The process of inputting the image description text and scenario inference prompts into the large language model to obtain the plot text includes: The image description text and the second scenario inference prompt are input into the large language model to obtain the second plot text. The second scenario inference prompt is used to prompt the large language model to infer events that occurred before the scenario shown in the image based on the image description text. The second plot text is used to describe the events that occurred before the scenario shown in the image. The process of generating a video based on the image and the narrative text includes: Based on the image and the second plot text, a video with the image as the last frame is generated.

5. The video generation method according to claim 1, characterized in that, The process of inputting the image description text and scenario inference prompts into the large language model to obtain the plot text includes: The image description text and the third scenario inference prompt are input into the large language model to obtain the third plot text. The third scenario inference prompt is used to prompt the large language model to infer the events that occurred before and after the scenario shown in the image based on the image description text. The third plot text is used to describe the events that occurred before and after the scenario shown in the image. The process of generating a video based on the image and the narrative text includes: Based on the image and the third plot text, a video with the image as the intermediate frame is generated.

6. The video generation method according to claim 5, characterized in that, The third scenario inference prompt includes a first sub-prompt and a second sub-prompt. The first sub-prompt is used to prompt the large language model to infer events that will occur after the scenario shown in the image based on the image description text. The second sub-prompt is used to prompt the large language model to infer events that have occurred before the scenario shown in the image based on the image description text. The process of inputting the image description text and the third scenario deduction prompt into the large language model to obtain the third plot text includes: Based on the weights of the first sub-cue and the second sub-cue, the large language model is used to process the image description text and the third scenario inference cue to obtain the third plot text. The third plot text includes a first sub-text generated based on the first sub-hint and a second sub-text generated based on the second sub-hint. The first sub-text describes events that will occur after the scene shown in the image, and the second sub-text describes events that will occur before and after the scene shown in the image. The amount of data in the sub-text generated based on each sub-hint is positively correlated with the weight of the sub-hint.

7. The video generation method according to claim 6, characterized in that, The weight of each sub-hint is positively correlated with the duration requirement of the user object for the event corresponding to the sub-hint.

8. The video generation method according to claim 1, characterized in that, The scenario simulation prompts are multiple; The process of inputting the image description text and scenario inference prompts into the large language model to obtain the plot text includes: If any scenario deduction prompt among multiple scenario deduction prompts is selected by the user object, the image description text and the scenario deduction prompt selected by the user object are input into the large language model to obtain the plot text.

9. The video generation method according to claim 1, characterized in that, The scenario simulation prompts are multiple; The process of inputting the image description text and scenario inference prompts into the large language model to obtain the plot text includes: Based on the behavioral characteristics of user objects, a target scenario deduction prompt is determined from multiple scenario deduction prompts. The behavioral characteristics are used to indicate the scenario deduction prompts that the user objects commonly use within the historical event segment. The image description text and the target scenario deduction prompt are input into the large language model to obtain the plot text.

10. The video generation method according to claim 1, characterized in that, The process of inputting the image description text and scenario inference prompts into the large language model to obtain the plot text includes: Input the image description text and scenario deduction prompts into the large language model to obtain multiple plot texts; The process of generating a video based on the image and the narrative text includes: If any one of the multiple plot texts is selected by the user object, a video is generated based on the image and the plot text selected by the user object.

11. A video generation apparatus, characterized in that, The device includes: The recognition unit is configured to perform recognition on an input image to obtain image description text, which is used to describe the scene in the image; The plot generation unit is configured to input the image description text and the scenario inference prompt into the large language model to obtain plot text. The scenario inference prompt is used to prompt the large language model to infer an event containing the scenario shown in the image based on the image description text. The plot text is used to describe the inferred event. The video generation unit is configured to generate a video based on the image and the plot text, the video containing events from the image and the plot text.

12. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the video generation method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the video generation method as described in any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method according to any one of claims 1 to 10.

Images