Video generation method and device, computer readable storage medium, and program product

Abstract

The present disclosure relates to a video generation method and device, a computer-readable storage medium, and a program product, and relates to the field of computer technology. The video generation method includes: receiving input content from a user, wherein the input content includes multiple reference images and sequence information of the multiple reference images; generating target prompt information with video content logic according to the sequence information and the change process information of objects and scenes in the multiple reference images; generating a video including the multiple reference images according to the target prompt information. According to the present disclosure, the controllability of video generation can be improved, the usability of generated videos can be improved, and the user experience can be enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video generation method and device, a computer-readable storage medium, and a program product. Background Art

[0002] With the development of artificial intelligence technology, related applications have penetrated into every aspect of our lives, such as intelligent video generation.

[0003] In related technologies, an intelligent agent provides users with feedback on videos corresponding to the text input by the user. Summary of the Invention

[0004] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] According to a first aspect of some embodiments of the present disclosure, there is provided a video generation method, including:

[0006] receiving input content from a user, wherein the input content includes a plurality of reference images and sequence information of the plurality of reference images;

[0007] generating target prompt information having video content logic according to the sequence information and information on changes in objects and scenes in the plurality of reference images;

[0008] A video including the multiple reference images is generated according to the target prompt information.

[0009] In some embodiments, generating target prompt information having video content logic according to the sequence information and information about changes in objects and scenes in the plurality of reference images includes:

[0010] Identify objects and scenes in each reference image;

[0011] Performing semantic understanding on the multiple reference images according to the sequence information to obtain change process information of the object and change process information of the scene;

[0012] Target prompt information having video content logic is generated according to the object, the scene and the change process information.

[0013] In some embodiments, performing semantic understanding on the plurality of reference images based on the sequence information to obtain information about the change process of the object and the scene includes:

[0014] extracting object features of objects and scene features of scenes in each reference image;

[0015] determining, based on the sequence information, first difference information between object features of the same object and second difference information between scene features of the same scene in two adjacent reference images;

[0016] determining change process information of the object according to the first difference information;

[0017] According to the second difference information, change process information of the scene is determined.

[0018] In some embodiments, performing semantic understanding on the plurality of reference images based on the sequence information to obtain information about the change process of the object and the scene includes:

[0019] performing multimodal fusion on the objects and scenes in the multiple reference images, the sequence information, and the multiple reference images;

[0020] According to the result of the multimodal fusion, the change process information of the object and the change process information of the scene are generated.

[0021] In some embodiments, performing multimodal fusion on the objects and scenes in the plurality of reference images, the sequence information, and the plurality of reference images includes:

[0022] extracting text features from the objects and scenes in the plurality of reference images and the sequence information;

[0023] Extracting visual features from the multiple reference images;

[0024] The text feature and the visual feature are fused to obtain a fused feature as a result of the multimodal fusion.

[0025] In some embodiments, generating target prompt information having video content logic according to the object, the scene, and the change process information includes:

[0026] generating first picture description information describing a video picture of the object according to the object and the change process information of the object;

[0027] generating second picture description information describing a video picture of the scene according to the scene and the change process information of the scene;

[0028] Target prompt information having video content logic is generated according to the first picture description information and the second picture description information.

[0029] In some embodiments, generating a video including the plurality of reference images according to the target prompt information includes:

[0030] determining, according to the change process information, a duration of a video segment based on the plurality of reference images;

[0031] A video including the multiple reference images is generated according to the target prompt information and the duration of the video clip.

[0032] In some embodiments, generating a video including the plurality of reference images according to the target prompt information and the duration of the video clip includes:

[0033] determining, according to the change process information, a visual effect of the video clip based on the plurality of reference images;

[0034] A video including the multiple reference images is generated according to the target prompt information, the duration of the video clip and the visual effect.

[0035] In some embodiments, the input content further includes at least one reference audio segment, and generating a video including the plurality of reference images according to the target prompt information includes:

[0036] determining, according to the change process information, an audio effect of the video clip based on the plurality of reference images;

[0037] Performing audio processing on the at least one reference audio segment according to the audio effect;

[0038] A video including the plurality of reference images and at least one reference audio segment after the audio processing is generated according to the target prompt information.

[0039] In some embodiments, generating a video including the plurality of reference images according to the target prompt information includes:

[0040] generating at least one reference audio segment according to the sequence information and the plurality of reference images;

[0041] A video including the multiple reference images and the at least one reference audio segment is generated according to the target prompt information.

[0042] In some embodiments, the object's change process information includes motion change information and form change information; and / or

[0043] The scene change process information includes scene angle change information and switching information between different scenes.

[0044] In some embodiments, the plurality of reference images include a first frame image, a last frame image, and at least one intermediate frame image between the first frame image and the last frame image of the video to be generated.

[0045] According to a second aspect of some embodiments of the present disclosure, there is provided a video generating apparatus, including:

[0046] a receiving module configured to receive input content from a user, wherein the input content includes a plurality of reference images and sequence information of the plurality of reference images;

[0047] A first generating module is configured to generate target prompt information having video content logic according to the sequence information and information about changes in objects and scenes in the plurality of reference images;

[0048] The second generating module is configured to generate a video including the multiple reference images according to the target prompt information.

[0049] According to a third aspect of some embodiments of the present disclosure, a video generating device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the video generating method of any embodiment described in the present disclosure based on instructions stored in the memory.

[0050] According to a fourth aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the video generation method of any embodiment described in the present disclosure is performed.

[0051] According to a fifth aspect of some embodiments of the present disclosure, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to implement the video generation method of any of the embodiments.

[0052] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings described herein are used to provide a further understanding of the present disclosure. Each of the drawings, together with the following detailed description, is included in this specification and forms a part of the specification to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation of the present disclosure. In the drawings:

[0054] Figure 1 is a flowchart illustrating a video generation method according to some embodiments of the present disclosure;

[0055] Figure 2 is a flowchart illustrating a video generation method according to other embodiments of the present disclosure;

[0056] Figure 3 is a flowchart illustrating a video generation method according to other embodiments of the present disclosure;

[0057] Figure 4 is a flowchart illustrating a video generation method according to other embodiments of the present disclosure;

[0058] Figure 5 is a flowchart illustrating a video generation method according to other embodiments of the present disclosure;

[0059] Figure 6 is a flowchart illustrating a video generation method according to other embodiments of the present disclosure;

[0060] Figure 7 is a flowchart illustrating a video generation method according to other embodiments of the present disclosure;

[0061] Figure 8 is a flowchart illustrating a video generation method according to other embodiments of the present disclosure;

[0062] Figure 9 is a flowchart illustrating a video generation method according to other embodiments of the present disclosure;

[0063] Figure 10A is a schematic diagram illustrating an interface for video generation according to some embodiments of the present disclosure;

[0064] Figure 10B is a schematic diagram illustrating an interface for video generation according to other embodiments of the present disclosure;

[0065] Figure 11 is a block diagram illustrating a video generating apparatus according to some embodiments of the present disclosure;

[0066] Figure 12 is a block diagram illustrating a video generating apparatus according to other embodiments of the present disclosure;

[0067] Figure 13 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0068] It should be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not necessarily drawn to scale. The same or similar reference numerals are used throughout the drawings to indicate the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. DETAILED DESCRIPTION

[0069] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. However, it is obvious that the embodiments described are only some embodiments of the present disclosure, rather than all embodiments. The following description of the embodiments is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein.

[0070] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in different orders and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0071] As used in this disclosure, the term "include" and its variations are intended to be open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to." Furthermore, the term "comprise" and its variations are intended to be open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to." Therefore, "include" and "include" are synonymous. The term "based on" means "based, at least in part, on."

[0072] Reference throughout this specification to "one embodiment," "some embodiments," or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. For example, the term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Furthermore, the appearances of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but may.

[0073] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules, or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules, or units. Unless otherwise specified, concepts such as "first" and "second" are not intended to imply that the objects described in such a manner must be in a given order in time, space, ranking, or any other manner.

[0074] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0075] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0076] The following detailed description of the embodiments of the present disclosure is provided in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. In addition, in one or more embodiments, specific features, structures, or characteristics may be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0077] In the related art, a video is generated based on text input by a user, but the controllability of the video generation is poor.

[0078] The present disclosure provides a technical solution that can improve the controllability of video generation, improve the usability of generated videos, and enhance user experience.

[0079] Figure 1 is a flowchart illustrating a video generation method according to some embodiments of the present disclosure.

[0080] like Figure 1 As shown, the video generation method includes: step S10, receiving input content from a user, wherein the input content includes multiple reference images and sequence information of the multiple reference images; step S20, generating target prompt information with video content logic based on the sequence information and information about the change process of objects and scenes in the multiple reference images; and step S30, generating a video including the multiple reference images based on the target prompt information. The video generation method can be executed by an intelligent agent.

[0081] The sequence information of the multiple reference images, for example, includes position information of the multiple reference images as image frames in the video to be generated.

[0082] The object in each reference image can be a person or object, and the scene can be the environment in which the object resides. Object change information describes how the object changes between different reference images (e.g., adjacent reference images). Scene change information describes how the scene changes between different reference images.

[0083] The video content logic may also be referred to as video story logic, and may include at least one of the video's story theme, background information, character information, and plot development. The target prompt information with the video content logic may describe the details of the story in the video to be generated.

[0084] In the above embodiment, the video generation process takes into account the multiple reference images entered by the user and their sequence information. Prompt information with video content logic is first generated, and then a video including the multiple reference images entered by the user is generated based on the prompt information. Throughout the video generation process, the reference images are considered as part of the video, and the sequence information is configured by the user. This makes the video generation process more controllable and the generated video more consistent with user expectations. This improves the controllability and accuracy of video generation, enhances the usability of the generated video, and enhances the user experience.

[0085] The following will be combined Figures 2 to 9 The video generation method in other embodiments of the present disclosure is described in detail.

[0086] Figure 2 It is a flowchart illustrating a video generation method according to some other embodiments of the present disclosure. Figure 2 and Figure 1 The difference is that Figure 2 Steps S21 to S23 in Figure 1 An implementation of step S20 in FIG. Figure 2 and Figure 1 The differences are as follows, and the similarities are not repeated here.

[0087] like Figure 2 As shown, in step S21, objects and scenes in each reference image are identified.

[0088] In step S22, semantic understanding is performed on the multiple reference images according to the sequence information to obtain change process information of the object and change process information of the scene.

[0089] In step S23, target prompt information having video content logic is generated according to the object, the scene and the change process information.

[0090] For example, object recognition can be used to identify objects and scenes in each reference image.

[0091] In this embodiment, the order of multiple reference images reflects to a certain extent the video content logic of the video to be generated, such as the development of the plot. By combining the sequence information of multiple reference images and performing semantic understanding on the multiple reference images, the change process information of the objects and the scene representing the development of the plot can be accurately obtained, thereby further guiding the accurate generation of target prompt information with video content logic, thereby further improving the controllability and accuracy of video generation and further enhancing the user experience.

[0092] Figure 3 It is a flowchart illustrating a video generation method according to some other embodiments of the present disclosure. Figure 3 and Figure 2 The difference is that Figure 3 Steps S220 to S226 in Figure 2 An implementation of step S22 in FIG. Figure 3 and Figure 2 The differences are as follows, and the similarities are not repeated here.

[0093] like Figure 3 As shown, in step S220, object features of objects and scene features of scenes in each reference image are extracted.

[0094] In step S222 , first difference information between object features of the same object and second difference information between scene features of the same scene in two adjacent reference images are determined based on the sequence information.

[0095] In step S224, the change process information of the object is determined according to the first difference information.

[0096] In step S226, the change process information of the scene is determined according to the second difference information.

[0097] For example, taking a kitten as an example, the kitten's changing process information can be determined by analyzing the differences in the kitten's movements, expressions, etc. in two adjacent reference images.

[0098] For another example, taking a coffee shop as an example, the changing process information of the coffee shop can be determined by analyzing the presence or absence of the coffee shop in two adjacent reference images, the perspective and other convenient differences.

[0099] In this embodiment, the difference changes of the same object and the difference changes of the same scene in two adjacent reference images are analyzed based on the sequential information, and then the change process information of the object and scene is determined according to the difference changes. This method is simple and fast, which can improve the efficiency of video generation and enhance the user experience.

[0100] Figure 4is a flowchart illustrating a video generation method according to some other embodiments of the present disclosure. Figure 4 and Figure 2 The difference is that Figure 4 Steps S221 and S223 in Figure 2 An implementation of step S22 in FIG. Figure 4 and Figure 2 The differences are as follows, and the similarities are not repeated here.

[0101] like Figure 4 As shown, in step S221, multimodal fusion is performed on the objects and scenes in the multiple reference images, the sequence information, and the multiple reference images.

[0102] In step S223 , according to the result of the multimodal fusion, the change process information of the object and the change process information of the scene are generated.

[0103] The object, scene, and sequence information is textual information, while the multiple reference images are image information. For example, machine learning models can be used to implement multimodal fusion of textual and image information. The result of multimodal fusion can be feature-level multimodal fusion features, such as feature vectors, or textual descriptions describing the visual features of objects and scenes across multiple reference images.

[0104] In this embodiment, through multimodal fusion, the changing process of objects and scenes in the generated video caused by multiple reference images can be learned more deeply, so as to more accurately generate the changing process information of objects and scenes, further improve the accuracy of video generation, and further enhance the user experience.

[0105] Figure 5 is a flowchart illustrating a video generation method according to some other embodiments of the present disclosure. Figure 5 and Figure 4 The difference is that Figure 5 Steps S2211 and S2213 in Figure 4 An implementation of step S221 in FIG. Figure 5 and Figure 4 The differences are as follows, and the similarities are not repeated here.

[0106] like Figure 5 As shown, in step S2211, text features are extracted from the objects and scenes in the multiple reference images and the sequence information.

[0107] In step S2212, visual features are extracted from the multiple reference images.

[0108] In step S2213, the text feature and the visual feature are fused to obtain a fused feature as the result of the multimodal fusion.

[0109] For example, both text features and visual features can be represented by feature vectors.

[0110] In this embodiment, the multimodal feature-level fusion of text features and visual features can improve the accuracy of multimodal fusion, thereby further improving the accuracy of video generation and enhancing user experience.

[0111] Figure 6 is a flowchart illustrating a video generation method according to some other embodiments of the present disclosure. Figure 6 and Figure 2 The difference is that Figure 6 Steps S231 to S233 in Figure 2 An implementation of step S23 in FIG. Figure 6 and Figure 2 The differences are as follows, and the similarities are not repeated here.

[0112] like Figure 6 As shown, in step S231, first picture description information describing the video picture of the object is generated according to the object and the change process information of the object.

[0113] In step S232, second picture description information describing the video picture of the scene is generated according to the scene and the change process information of the scene.

[0114] In step S233, target prompt information having video content logic is generated according to the first screen description information and the second screen description information.

[0115] For example, taking a kitten as an example, the first picture description information may include description information of changes in the kitten's movements, expressions, etc. over time.

[0116] For example, taking the scene of a campus playground as an example, the second picture description information may include description information of changes in the size, shape, light and shadow effects, etc. of the campus playground under different viewing angles.

[0117] In this embodiment, first picture description information of the object's video screen and second picture description information of the scene's video screen are generated based on the object's change process information and the scene's change process information, respectively, and then target prompt information is generated based on the picture description information. This can improve the accuracy of the target prompt information, thereby improving the accuracy of video generation and enhancing user experience.

[0118] Figure 7It is a flowchart illustrating a video generation method according to some other embodiments of the present disclosure. Figure 7 and Figure 2 The difference is that Figure 7 Steps S31 to S33 in Figure 2 An implementation of step S30 in FIG. Figure 7 and Figure 2 The differences are as follows, and the similarities are not repeated here.

[0119] like Figure 7 As shown, in step S31, the duration of the video clip based on the multiple reference images is determined according to the change process information.

[0120] In step S33, a video including the multiple reference images is generated according to the target prompt information and the duration of the video clip.

[0121] For example, when the change process information indicates that the change process is complex, a longer duration may be determined; and when the change process information indicates that the change process is simple, a shorter duration may be determined.

[0122] In this embodiment, the duration of the video segment is determined based on the change process information, and then the video is generated according to the target prompt information and the duration of the video segment, thereby further improving the controllability and accuracy of video generation and enhancing the user experience.

[0123] In some embodiments, the visual effect of a video clip based on the multiple reference images can be determined based on the change process information; and then, a video including the multiple reference images can be generated based on the target prompt information, the length of the video clip and the visual effect.

[0124] For example, visual effects can be the brightness and darkness of colors, dynamic effects, switching methods, etc.

[0125] In this embodiment, during the video generation process, the visual effect is determined based on the change process information, which can further improve the controllability and accuracy of the video generation and enhance the user experience.

[0126] Figure 8 It is a flowchart illustrating a video generation method according to some other embodiments of the present disclosure. Figure 8 and Figure 2 The difference is that Figure 8 Steps S32 to S36 in Figure 2 An implementation of step S30 in FIG. Figure 8 and Figure 2 The differences are as follows, and the similarities are not repeated here.

[0127] like Figure 8 As shown, in step S32, the audio effect of the video clip based on the multiple reference images is determined according to the change process information.

[0128] In step S34, audio processing is performed on the at least one reference audio segment according to the audio effect.

[0129] In step S36, a video including the plurality of reference images and the at least one reference audio segment after the audio processing is generated according to the target prompt information.

[0130] For example, audio effects can include effects on sound quality, spatial perception, pitch changes, and other aspects.

[0131] In this embodiment, the user can input reference audio. During the video generation process, the audio effect is determined based on the change process information to process the reference audio input by the user, so that the generated video is more realistic, which can further improve the controllability, accuracy and usability of video generation and enhance the user experience.

[0132] Figure 9 is a flowchart illustrating a video generation method according to some other embodiments of the present disclosure. Figure 9 and Figure 2 The difference is that Figure 9 Steps S32' to S34' in Figure 2 An implementation of step S30 in FIG. Figure 9 and Figure 2 The differences are as follows, and the similarities are not repeated here.

[0133] like Figure 9 As shown, in step S32', at least one reference audio segment is generated according to the sequence information and the multiple reference images.

[0134] In step S34 ′, a video including the plurality of reference images and the at least one reference audio segment is generated according to the target prompt information.

[0135] In this embodiment, reference audio can be generated based on the sequence information of multiple reference images and multiple reference images. Even if the user does not provide audio, audio that better meets the user's needs can be generated, and then video can be generated, further improving the accuracy and usability of video generation and enhancing the user experience.

[0136] In some embodiments, the object's change process information includes motion change information and morphological change information. For example, motion change information may include position change, posture change, gesture change, etc. For example, morphological change information may include shape change, size change, color or texture change, etc.

[0137] In some embodiments, the scene change process information includes scene angle change information and information about switching between different scenes. The scene angle change information may include, for example, a change in viewing angle height or viewing angle distance. The switching information between different scenes may include, for example, the process of switching from a first scene to a second scene.

[0138] In some embodiments, the multiple reference images include the first and last frames of the video to be generated, and at least one intermediate frame between the first and last frames. In this embodiment, the first, last, and intermediate frames are used to describe the process of switching from the first frame to the intermediate frames and then to the last frame, making the video generation process more controllable, further improving the controllability and accuracy of video generation, and enhancing the user experience.

[0139] In some embodiments, the user input content may also include initial prompt information input by the user, which represents the user's needs or expectations for video generation. The initial prompt information can be used to generate target prompt information throughout the entire process of video generation.

[0140] The following will be combined Figure 10A and Figure 10B The video generation process in some embodiments of the present disclosure is described.

[0141] Figure 10A It is a schematic diagram of an interface showing video generation according to some embodiments of the present disclosure.

[0142] Figure 10B It is a schematic diagram of an interface for video generation according to other embodiments of the present disclosure.

[0143] like Figure 10A As shown, the user enters the interactive interface for interacting with the intelligent agent, and can perform settings for video generation by operating the controls 10 of the interactive interface, such as inputting multiple reference images, inputting initial prompt information, and inputting audio.

[0144] For example, the control 10 is a button labeled "video generation", and the user can call up the following Figure 10B The video shown generates the panel. Figure 10B The video generation panel shown includes one or more controls, such as control 101 and control 102. The one or more controls include a control for a user to upload one or more reference images, and may also include controls for a user to set certain properties of the video to be generated (such as aspect ratio, playback speed, etc.).

[0145] For the sequence information of multiple reference images, the one or more controls can be identified as different frame identifiers, so that the frame position of the reference image in the video to be generated can be defined by uploading the reference image using the control with the corresponding frame identifier. For example, one or more controls include multiple image upload buttons, and these multiple image upload buttons are distinguished by words such as "first frame image" and "last frame image". The frame positions of the reference images corresponding to different image upload buttons in the video to be generated. For another example, the order between multiple reference images can also be determined according to the order in which the images are uploaded, and the frame position of the reference image in the video to be generated can be set using other controls.

[0146] Taking the image upload button of control 101 for users to upload reference images as an example, the user clicks the image upload button to call out the user's local or cloud album, and the user selects and checks an image in the album as a reference image to upload.

[0147] like Figure 10A and 10B As shown, the interactive interface also includes controls 20 and 30. For example, control 20 is an input box and control 30 is a send button. Figure 10A Enter the message indicating the video generation in the input box and click the send button to trigger the agent to call Figure 10B The video shown generates the panel.

[0148] After the user completes the settings in the video generation panel, he can click Figure 10B The send button shown triggers the intelligent agent to execute the video generation method in any embodiment of the present disclosure to generate a video.

[0149] For example, using control 20 as an input box, the user can also enter original prompt information in the input box, where the original prompt information represents the user's needs or expectations for video generation. For example, the original prompt information can be a complete narrative text, including but not limited to information such as characters, scenes, and behaviors.

[0150] The above is a video generation method provided by some embodiments of the present disclosure. Figure 11 The video generating apparatus in some embodiments of the present disclosure is described.

[0151] Figure 11 is a block diagram illustrating a video generating apparatus according to some embodiments of the present disclosure.

[0152] like Figure 11 As shown, the video generating device 11 includes a receiving module 111 , a first generating module 112 and a second generating module 113 .

[0153] The receiving module 111 is configured to receive input content from a user, wherein the input content includes multiple reference images and sequence information of the multiple reference images.

[0154] The first generating module 112 is configured to generate target prompt information having video content logic according to the sequence information and information about changes in objects and scenes in the plurality of reference images.

[0155] The second generating module 113 is configured to generate a video including the multiple reference images according to the target prompt information.

[0156] The video generating device 11 can be used to perform Figure 1 In some embodiments, the video generating device 11 may also perform the following steps: Figures 2 to 9 Any of the steps shown.

[0157] It should be noted that the aforementioned modules are merely logical modules divided according to the specific functions they implement, and are not intended to limit specific implementation methods. For example, they may be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the aforementioned modules may be implemented as independent physical entities, or they may be implemented by a single entity (e.g., a processor (CPU or DSP, etc.), an integrated circuit, etc.). Furthermore, the aforementioned modules are depicted with dashed lines in the accompanying drawings to indicate that these modules may not actually exist, and that the operations / functions they implement may be implemented by the processing circuit itself.

[0158] Figure 12 is a block diagram illustrating a video generating apparatus according to other embodiments of the present disclosure.

[0159] like Figure 12 As shown, the video generating device 12 includes: a memory 121; and a processor 122 coupled to the memory 121, and the processor 122 is configured to execute the video generating method described in any of the above embodiments based on instructions stored in the memory 121.

[0160] Memory 121 is used to store one or more computer-readable instructions. Memory 121 may include any combination of various forms of computer-readable storage media, such as volatile and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 121 may store, for example, an operating system, applications, a boot loader, databases, and other programs, as well as various applications and data.

[0161] The processor 122 is configured to execute computer-readable instructions to implement the video generation method described in any of the aforementioned embodiments. Detailed implementations of the various steps of the video generation method can be found in the aforementioned embodiments, and the repetitive details will not be repeated here.

[0162] The processor 122 and the memory 121 can communicate with each other directly or indirectly. For example, the processor 122 and the memory 121 can communicate with each other via a network. The network can include a wireless network, a wired network, and / or any combination of wireless networks and wired networks. The processor 122 and the memory 121 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0163] It should be noted that Figure 12 The components of the video generating device 12 shown are merely exemplary and non-limiting. The video generating device 12 may also have other components according to actual application requirements. The processor 122 may control other components in the video generating device 12 to perform desired functions.

[0164] The video generating device may be implemented by software, firmware and / or hardware, and may be integrated into an electronic device installed with a related application program.

[0165] The above is a video generating device in some embodiments of the present disclosure.

[0166] Figure 13 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0167] Figure 13 The electronic device 13 shown may be a computer system with a dedicated hardware structure, which can execute corresponding functions when a relevant application program is installed.

[0168] Electronic devices include, but are not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet personal computers (Tablet PCs), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., as well as fixed terminals such as digital televisions and desktop computers, etc.

[0169] like Figure 13As shown, the central processing unit (CPU) 131 executes various processes according to the program stored in the read-only memory (ROM) 132 or the program loaded from the storage unit 138 to the random access memory (RAM) 133. In the RAM 133, data required when the CPU 131 executes various processes is stored as needed. The central processing unit is merely an example, and it may also be other types of processors, such as the various processors described above. The ROM 132, RAM 133 and the storage unit 138 may be various forms of computer-readable storage media. It should be noted that although Figure 13 ROM 132, RAM 133 and storage portion 138 are shown separately in FIG, but one or more of them may be combined or located in the same or different memory or storage modules.

[0170] The CPU 131, the ROM 132, and the RAM 133 are connected to one another via a bus 134. An input / output interface 135 is also connected to the bus 134.

[0171] The following components are connected to the input / output interface 135: an input portion 136 such as a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output portion 137 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage portion 138 including a hard disk, a magnetic tape, etc.; and a communication portion 139 including a network interface card such as a LAN card, a modem, etc. The communication portion 139 allows communication processing to be performed via a network such as the Internet. As will be readily understood, although Figure 13 The various devices or modules in the electronic device 13 are shown to communicate via the bus 134, but they can also communicate via a network or other means, where the network can include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.

[0172] A drive 1310 is also connected to the input / output interface 135 as needed. A removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1310 as needed so that a computer program read therefrom is installed in the storage section 138 as needed.

[0173] When the above-described series of processing is implemented by software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 1311 .

[0174] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product that, when the computer program product is run on a computer, causes the computer to implement the video generation method described in any of the aforementioned embodiments. The computer program product includes a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 139, or installed from the storage part 138, or installed from the ROM 132. When the computer program is executed by the CPU 131, the video generation method of the embodiment of the present disclosure is executed.

[0175] It should be noted that, in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus, or device or for use in conjunction with an instruction execution system, apparatus, or device.

[0176] The computer readable medium may be a computer readable storage medium, or a computer readable signal medium, or any combination of the two.

[0177] Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. A computer program is stored on a computer-readable storage medium that, when executed by a processor, implements the video generation method described in any of the aforementioned embodiments.

[0178] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0179] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0180] In some embodiments, a computer program product is further provided. When the computer program product is run on a computer, the computer is enabled to implement the video generation method described in any of the above embodiments.

[0181] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to perform the video generation method of any of the above embodiments. For example, the instructions may be embodied as computer program codes.

[0182] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] The functions described above may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0185] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A video generation method, characterized in that: include: Receiving input content from a user, wherein the input content includes a plurality of reference images and sequence information of the plurality of reference images; According to the sequence information and the change process information of the objects and scenes in the multiple reference images, target prompt information with video content logic is generated, including: identifying the objects and scenes in each reference image; according to the sequence information, semantically understanding the multiple reference images to obtain the change process information of the objects and the change process information of the scenes; according to the objects and the change process information of the objects, first picture description information describing the video picture of the objects is generated; according to the scenes and the change process information of the scenes, second picture description information describing the video picture of the scenes is generated; according to the first picture description information and the second picture description information, target prompt information with video content logic is generated; A video including the multiple reference images is generated according to the target prompt information.

2. The video generation method according to claim 1, characterized in that: According to the sequence information, semantic understanding is performed on the multiple reference images to obtain the change process information of the object and the scene, including: extracting object features of the objects and scene features of the scene in each reference image; Determining, according to the sequence information, first difference information between object features of the same object in two adjacent reference images and second difference information between scene features of the same scene; Determining change process information of the object according to the first difference information; According to the second difference information, change process information of the scene is determined.

3. The video generation method according to claim 1, characterized in that: According to the sequence information, semantic understanding is performed on the multiple reference images to obtain the change process information of the object and the scene, including: Performing multimodal fusion on the objects and scenes in the multiple reference images, the sequence information, and the multiple reference images; According to the result of the multimodal fusion, the change process information of the object and the change process information of the scene are generated.

4. The video generation method according to claim 3, characterized in that: Performing multimodal fusion on the objects and scenes in the multiple reference images, the sequence information, and the multiple reference images includes: Extracting text features from the objects and scenes in the plurality of reference images and the sequence information; Extracting visual features from the multiple reference images; The text feature and the visual feature are fused to obtain a fused feature as a result of the multimodal fusion.

5. The video generation method according to claim 1, characterized in that: Generating a video including the plurality of reference images according to the target prompt information includes: Determining, according to the change process information, the duration of the video clips based on the multiple reference images; A video including the multiple reference images is generated according to the target prompt information and the duration of the video clip.

6. The video generation method according to claim 5, characterized in that: Generating a video including the plurality of reference images according to the target prompt information and the duration of the video clip includes: Determining, according to the change process information, a visual effect of the video clip based on the plurality of reference images; A video including the multiple reference images is generated according to the target prompt information, the duration of the video clip and the visual effect.

7. The video generation method according to claim 1, characterized in that: The input content also includes at least one reference audio segment, and generating a video including the plurality of reference images according to the target prompt information includes: Determining, according to the change process information, an audio effect of the video clip based on the plurality of reference images; Performing audio processing on the at least one reference audio segment according to the audio effect; A video including the plurality of reference images and at least one reference audio segment after the audio processing is generated according to the target prompt information.

8. The video generation method according to claim 1, characterized in that: Generating a video including the plurality of reference images according to the target prompt information includes: Generate at least one reference audio segment according to the sequence information and the plurality of reference images; A video including the plurality of reference images and the at least one reference audio segment is generated according to the target prompt information.

9. The video generation method according to claim 2, characterized in that: The object's change process information includes action change information and form change information; and / or The scene change process information includes scene angle change information and switching information between different scenes.

10. The video generation method according to claim 1, characterized in that: The multiple reference images include a first frame image, a last frame image, and at least one intermediate frame image between the first frame image and the last frame image of the video to be generated.

11. A video generating device, characterized in that: include: A receiving module, configured to receive input content from a user, wherein the input content includes a plurality of reference images and sequence information of the plurality of reference images; The first generation module is configured to generate target prompt information with video content logic according to the sequence information and the change process information of the objects and scenes in the multiple reference images, including: identifying the objects and scenes in each reference image; performing semantic understanding on the multiple reference images according to the sequence information to obtain the change process information of the objects and the change process information of the scenes; generating first picture description information describing the video picture of the objects according to the objects and the change process information of the objects; generating second picture description information describing the video picture of the scenes according to the scenes and the change process information of the scenes; generating target prompt information with video content logic according to the first picture description information and the second picture description information; The second generating module is configured to generate a video including the multiple reference images according to the target prompt information.

12. A video generating device, characterized in that: include: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the video generating method according to any one of claims 1 to 10 based on instructions stored in the memory.

13. A computer-readable storage medium, characterized in that: Computer program instructions are stored thereon, and when the instructions are executed by a processor, the video generation method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The invention comprises a computer program, which implements the video generation method according to any one of claims 1 to 10 when being executed by a processor.

Images