Video generation method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a video generation method and device, electronic equipment and a computer readable storage medium, and can be applied to the technical field of artificial intelligence and computer vision. The method comprises the steps that a scene prompt text is generated according to a video content description text input by a user, and the scene prompt text is used for limiting a generation scene of a video; inputting the scene prompt text into the large language model, and outputting at least one video split text; determining a video clip matched with each video split text from a database according to the keyword of each video split text; and generating a target video matched with the video content description text according to the video split text and the at least one video clip.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence and computer vision technology, and in particular to a video generation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the popularization of mobile video technology, video has gradually become an important way for the general public to understand the world, and video content creation has also ushered in explosive growth. At present, the vast majority of video producers adopt a more traditional video generation method: creating video texts, searching and screening video materials, editing videos, and post-processing videos. The traditional video generation method is very time-consuming. It often takes up to several weeks to produce a video of a few hours.

[0003] Therefore, there are at least the following technical problems in the relevant technology: the video generation process requires real-time participation of video producers, and the video generation method is not intelligent enough. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a video generation method, device, electronic device and computer-readable storage medium.

[0005] According to a first aspect of the present disclosure, a video generation method is provided, comprising: generating a scene prompt text according to a video content description text input by a user, wherein the scene prompt text is used to limit a generation scene of a video; inputting the scene prompt text into a large language model, and outputting at least one video storyboard text; determining a video clip matching each video storyboard text from a database according to keywords of each video storyboard text; and generating a target video matching the video content description text according to the video storyboard text and at least one video clip.

[0006] According to an embodiment of the present disclosure, the method further includes: inputting each video storyboard text into a large language model, outputting at least one keyword in each video storyboard text

[0007] According to an embodiment of the present disclosure, based on the keywords of each video storyboard text, a video clip matching each video storyboard text is determined from a database, including: based on the keywords, searching the database for multiple initial video clips matching the keywords; determining the similarity between each initial video clip and the video storyboard text; and based on the similarity, determining a video clip from the multiple initial video clips.

[0008] According to an embodiment of the present disclosure, determining the similarity between each initial video clip and a video storyboard text includes: determining a first vector corresponding to the initial video clip, wherein the first vector is used to characterize image features of the initial video clip; converting the video storyboard text into a second vector; and determining the similarity between the initial video clip and the video storyboard text based on the first vector and the second vector.

[0009] According to an embodiment of the present disclosure, each initial video segment includes at least one video frame or a material description text for at least one video frame. Determining a first vector corresponding to the initial video segment includes: determining a key video frame from at least one video frame; encoding the key video frame into a first vector; or encoding each video frame in at least one video frame into a third vector; using an average vector of at least one third vector as the first vector; or encoding the material description text into the first vector.

[0010] According to an embodiment of the present disclosure, a scene prompt text is generated based on a video content description text input by a user, including: determining a video generation scene related to the video content description text based on the video content description text; obtaining a prompt template that matches the video generation scene, wherein the prompt template includes video restriction information for the video generation scene; and generating a scene prompt text based on the video content description text and the prompt template.

[0011] According to an embodiment of the present disclosure, the method also includes: in response to detecting a preset interactive operation of a user, obtaining at least one custom video restriction information corresponding to the preset interactive operation; updating a prompt template using at least one custom video restriction information; and generating a scene prompt text based on the updated prompt template and the video content description text.

[0012] According to an embodiment of the present disclosure, a target video matching a video content description text is generated based on a video storyboard text and at least one video segment, including: combining at least one video segment into a target video according to an arrangement order of at least one video storyboard text.

[0013] According to an embodiment of the present disclosure, generating a target video that matches a video content description text based on a video storyboard text and at least one video clip also includes: converting each video storyboard text in the at least one video storyboard text into an initial audio clip; combining a preset smooth audio clip and at least one initial audio clip into audio data according to the arrangement order of the at least one video storyboard text; and generating a target video that matches the video content description text based on the audio data and at least one video clip.

[0014] According to an embodiment of the present disclosure, the method also includes: adding background audio data to the audio data to obtain updated audio data; wherein the background audio data is predetermined; or, the background audio data is extracted from a video content description text input by a user; or, the background audio data is determined based on a preset interactive operation of the user.

[0015] According to an embodiment of the present disclosure, a target video corresponding to a user video content description text is generated based on audio data and at least one video clip, including: cropping each video clip to obtain a target video clip that matches the length of the audio clip corresponding to the video clip; and generating a target video corresponding to the user video content description text based on the audio data and at least one target video clip.

[0016] According to an embodiment of the present disclosure, the method further includes: generating video subtitle data corresponding to each video segment; and generating a target video corresponding to the user video content description text based on the audio data, at least one video segment and at least one video subtitle data.

[0017] The second aspect of the present disclosure provides a video generation device, including: a first generation module, used to generate a scene prompt text according to a video content description text input by a user, wherein the scene prompt text is used to limit the generation scene of the video; a second generation module, used to input the scene prompt text into a large language model and output at least one video storyboard text; a determination module, used to determine a video clip corresponding to each video storyboard text from a database according to keywords of each video storyboard text; and a third generation module, used to generate a target video matching the video content description text according to the video storyboard text and at least one video clip.

[0018] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein when the one or more processors execute the one or more computer programs, the steps of the above-mentioned video generation method are implemented.

[0019] The fourth aspect of the present disclosure further provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above-mentioned video generation method are implemented.

[0020] The fifth aspect of the present disclosure also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned video generation method when executed by a processor.

[0021] The embodiments of the present disclosure provide an intelligent video generation method, in which the user only needs to input a video content description text to obtain a target video that matches the video content description text, without the need for the user to independently create a video storyboard text, search and filter video clips, edit videos, and perform post-processing, thereby achieving end-to-end content creation and video generation, thereby improving the intelligence and generation efficiency of the video generation method. In addition, since the user does not need to independently create a video storyboard text, edit videos, and perform post-processing, the embodiments of the present disclosure can also reduce the user's video creation skill requirements, facilitate professional or non-professional users to generate video content, and thus improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 The application scenario of the video generation method according to the embodiment of the present disclosure is schematically shown;

[0024] Figure 2 The flowchart of the video generation method according to the embodiment of the present disclosure is schematically shown;

[0025] Figure 3A A flowchart for determining keywords of a video storyboard text according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 3B Schematically shows a flow chart for determining keywords of a video storyboard text according to another embodiment of the present disclosure;

[0027] Figure 4A A flowchart for determining a video segment according to a specific embodiment of the present disclosure is schematically shown;

[0028] Figure 4B Schematically shows a flow chart of determining a video segment according to another specific embodiment of the present disclosure;

[0029] Figure 5 A schematic diagram schematically shows a plurality of initial video clips acquired according to an embodiment of the present disclosure;

[0030] Figure 6 A flowchart of generating a target video according to a video content description text input by a user according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 7 The structure block diagram of the video generating device according to the embodiment of the present disclosure is schematically shown;

[0032] Figure 8A block diagram of an electronic device suitable for a video generating method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0034] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0035] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0036] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0037] For traditional video generation methods, the entire process is operated by the video producer (user), resulting in a long video generation time. In other video generation methods, although users can use a variety of search algorithms to improve the efficiency of video material search, other generation processes still require user participation.

[0038] For example, in the video script creation stage, in order to ensure that the video content conforms to the user's creative concept, the user usually writes a video storyboard script based on a specific theme. In the material collection stage, although the user can use the algorithm to search for video materials, the user is still required to screen the video materials, process the video materials, and even shoot or customize the video materials based on the video storyboard script. In the video editing stage, the user generally uses the editing tools to combine and edit the video materials. It can be seen that the current video generation process is cumbersome and the generation efficiency is low; in addition, the video content is also defined by the user throughout the process, and the video generation method is not intelligent enough.

[0039] In order to at least partially solve the above technical problems, an embodiment of the present disclosure provides a video generation method, comprising: generating a scene prompt text according to a video content description text input by a user, wherein the scene prompt text is used to limit a generation scene of a video; inputting the scene prompt text into a large language model and outputting at least one video storyboard text; determining a video clip matching each video storyboard text from a database according to keywords of each video storyboard text; and generating a target video matching the video content description text according to the video storyboard text and at least one video clip.

[0040] Figure 1 The application scenario of the video generation method according to the embodiment of the present disclosure is schematically illustrated.

[0041] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0042] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0043] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0044] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0045] It should be noted that the video generation method provided in the embodiment of the present disclosure can be executed by the server 105; it can also be executed by one of the first terminal device 101, the second terminal device 102, and the third terminal device 103. Alternatively, it can also be executed by a system consisting of the server 105 and one of the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the video generation device provided in the embodiment of the present disclosure is correspondingly set in the server 105, or set in one of the first terminal device 101, the second terminal device 102, and the third terminal device 103, or set in the above system.

[0046] For example, the user can input the video content description text on the terminal device, generate the scene prompt text locally on the terminal device, and output at least one storyboard text with the help of the local large language model. Then, through the local offline database or the real-time online database, obtain the video clips that match the video storyboard text, and then generate the target video. Alternatively, the user can input the video content description text on the local terminal device, and the local terminal device sends the video content description text to the server, which generates the scene prompt text, generates the video storyboard text, searches for video materials, and generates the target video.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0048] The following will be based on Figure 1 The scene described by Figure 2~Figure 6 The video generation method of the disclosed embodiment is described in detail.

[0049] Figure 2 The flowchart of the video generation method according to the embodiment of the present disclosure is schematically shown.

[0050] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0051] In operation S210, a scene prompt text is generated according to the video content description text input by the user, wherein the scene prompt text is used to define a generation scene of the video;

[0052] In operation S220, the scene prompt text is input into the large language model, and at least one video storyboard text is output.

[0053] In operation S230, according to the keywords of each video storyboard text, a video segment matching each video storyboard text is determined from a database.

[0054] In operation S240, a target video matching the video content description text is generated according to the video storyboard text and at least one video clip.

[0055] According to an embodiment of the present disclosure, a user may input video content description text in a variety of input modes, such as voice input, input with the aid of an external hardware device, or input through interaction with a display screen of a terminal device.

[0056] The video content description text can have various lengths. For example, the video content description text input by the user can be a long text describing the generation requirements, content requirements and / or generation scenes of the video. Alternatively, the video content description text input by the user can be a simple theme text, item text or scene text.

[0057] In the process of generating the scene prompt text, the video content description text input by the user can be subjected to information extraction and information reorganization to obtain the scene prompt text. For example, in the case where the video content description text is a long text describing the generation requirements, content requirements and / or generation scenes of the video, the video generation requirement information related to the generation requirements, the video content requirement information related to the content requirements, and the scene requirement information related to the generation scene can be extracted from the video content description text. The scene requirement text is generated based on the video generation requirement information, the video content requirement information, and / or the scene requirement information.

[0058] For example, the video content description text may be "I want a video showing the advantages of a currently popular computer, preferably with a manual explanation section, and the video length should not be too long", and the video generation requirement information extracted therefrom is "manual explanation" and "video length should not be too long", the video content requirement information may be "advantage display" and "popular computer", and the scene requirement video may be a selling point display theme extracted from "advantage display". The above embodiment is only used as an illustrative display, and the video generation requirement information, video content requirement information, and / or scene requirement information, etc. can be divided according to actual needs, and are not limited to the above information and the above information distinction form.

[0059] In the process of generating the scene prompt text, the video content description text can also be directly processed into the scene prompt text based on a predefined prompt template. The above prompt template can be a general template or multiple templates related to the scene.

[0060] According to an embodiment of the present disclosure, the scene prompt text can be used as the input of a large language model (Large Language Models, LLM), that is, prompt. For example, the large language model can be one or more of GPT (Generative Pre-Trained Transformer, generative pre-trained encoding and decoding model), ChatGPT (Chat Generative Pre-Trained Transformer, chat generative pre-trained encoding and decoding model), GLM (General Language Model, general language model), etc., but is not limited to this.

[0061] Since the scene prompt text is generated based on the video content description text input by the user, the large language model can analyze the content requirements, generation requirements and scene requirements of the video from the scene prompt text, and then generate at least one video storyboard text that meets the user's needs, without the user having to write the video storyboard text themselves.

[0062] The video storyboard text output by the large language model, also known as the video storyboard script, is used to guide video shooting and video generation. Each video storyboard text can be used as the smallest unit for shooting a person, action, or event. For example, a video storyboard text output by the large language model can be: a close-up display and dynamic effect display of the selling points of a best-selling computer.

[0063] According to an embodiment of the present disclosure, after obtaining at least one video storyboard text using a large language model, a search for video material can be performed based on the at least one video storyboard text, wherein each video material can be understood as a video clip.

[0064] Since multiple video storyboard texts may be generated for multiple user requirements in the scene-limited text, the video clips used by each video storyboard text may be different. Therefore, for each video storyboard text, according to the keywords of the video storyboard text, a video clip matching the video storyboard text is determined from the database.

[0065] Searching for video clips matching the video storyboard text includes: searching for video clips related to the keywords from a database through a variety of search algorithms. The database may be an open source video material library or a pre-built video material library.

[0066] According to an embodiment of the present disclosure, after searching for a video segment matching each video storyboard text, at least one video segment may be combined based on the video storyboard text to obtain a target video.

[0067] Through the embodiments of the present disclosure, starting from the video content description text input by the user, a scene prompt text for limiting the video generation scene is generated, and the scene prompt text is processed using a large language model to realize the video storyboard text creation of the video generation process. Afterwards, the video storyboard text output by the large language model is used to search for at least one video clip, and end-to-end content creation and video generation are realized based on the video storyboard text and at least one video clip.

[0068] The embodiments of the present disclosure provide an intelligent video generation method, in which the user only needs to input a video content description text to obtain a target video that matches the video content description text, without the need for the user to independently create a video storyboard text, search and filter video clips, edit videos, and perform post-processing, thereby achieving end-to-end content creation and video generation, thereby improving the intelligence and generation efficiency of the video generation method. In addition, since the user does not need to independently create a video storyboard text, edit videos, and perform post-processing, the embodiments of the present disclosure can also reduce the user's video creation skill requirements, facilitate professional or non-professional users to generate video content, and thus improve the user experience.

[0069] According to an embodiment of the present disclosure, before performing operation S230, keywords in each video storyboard text may also be determined by the following method: inputting each video storyboard text into a large language model, and outputting at least one keyword in each video storyboard text.

[0070] According to an embodiment of the present disclosure, the large language model may include multiple model components, for example, the first model component is used to generate and output at least one video storyboard text according to the scene prompt text; the second model component is used to extract keywords from the video storyboard text. Thus, each video storyboard text is input into the large language model, and at least one keyword in each video storyboard text is outputted by the first model component of the large language model, and at least one video storyboard text is respectively input into the second model component of the large language model, and at least one keyword in each video storyboard text is outputted.

[0071] In one embodiment, the large language model can have two different inputs and two different outputs. The large language model obtains different inputs by analyzing the input prompt and using different model components. For example, after the scene prompt text is input into the large language model, the large language model outputs at least one video storyboard text using the first model component; then, after each video storyboard text is input into the large language model, the large language model outputs at least one keyword in each video storyboard text using the second model component.

[0072] Figure 3A The flowchart of determining keywords of a video storyboard text according to an embodiment of the present disclosure is schematically shown.

[0073] like Figure 3A As shown, in embodiment 300A, a scene prompt text 302 can be generated for a video content description text 301 input by a user. After the scene prompt text 302 is input into the large language model M10, the large language model M10 outputs a plurality of video storyboard texts 303 according to the scene prompt text, such as video storyboard text 1 303_1.... The plurality of video storyboard texts 303 are input into the large language model M10 again, and a plurality of keywords 304 are output, such as keyword 1 304_1.... Each video storyboard text may include at least one keyword, for example, the large language model M10 may extract keyword 1 304_1 from video storyboard text 1 303_1.

[0074] In this embodiment, before inputting the multiple video storyboard texts 303 into the large language model M10, the multiple video storyboard texts 303 can be adaptively modified, for example, new model processing requirements can be added, such as the model processing requirement can be "extract keywords in XX", XX is each video storyboard text.

[0075] In another embodiment, the final output of the large language model can be at least one video storyboard text and at least one keyword of each video storyboard text. That is, after the scene prompt text is input into the large language model, the first model component and the second model component are used in sequence to obtain at least one video storyboard text and at least one keyword of each video storyboard text. Finally, the output component of the large language model integrates the outputs of the first model component and the second model component, and outputs at least one video storyboard text and at least one keyword in each video storyboard text in a predetermined format.

[0076] Figure 3B The flowchart of determining keywords of a video storyboard text according to another embodiment of the present disclosure is schematically shown.

[0077] like Figure 3BAs shown, in embodiment 300B, a scene prompt text 302 may be generated for a video content description text 301 input by a user. After the scene prompt text 302 is input into the large language model M10, the large language model M10 outputs a plurality of video storyboard texts and their corresponding keywords according to the scene prompt text, such as video storyboard text 1 3031_1 ..., keyword 1 3032_1 .... Each video storyboard text may include at least one keyword, such as keyword 1 3032_1 extracted from video storyboard text 1 3031_1.

[0078] In this embodiment, based on the video content description text 301 input by the user, the generated scene prompt text 302 may include model processing requirements, such as "generating video storyboard text" and "extracting keywords from each video storyboard text".

[0079] The embodiments of the present disclosure not only utilize the large language model to generate video storyboard text, but also utilize the large language model to extract keywords from each video storyboard text, thereby reducing data transmission and interactive operations between multiple functional modules at the back end, and the text processing function is relatively centralized, which facilitates reducing the size of the end-to-end video generation software.

[0080] According to another embodiment of the present disclosure, multiple existing natural language processing (NLP) technologies can also be used to extract keywords from each video storyboard text. For example, keywords can be extracted using the keyword extraction model KeyBert based on the Bert model, keywords can be extracted using the topic model (Latent Dirichlet Allocation, LDA), keywords can be extracted based on the TF-IDF algorithm, keywords can be extracted based on the TextRank algorithm, etc.

[0081] According to an embodiment of the present disclosure, for operation S230, determining a video segment matching each video storyboard text from a database based on keywords of each video storyboard text includes: searching the database for multiple initial video segments matching the keywords based on the keywords; determining the similarity between each initial video segment and the video storyboard text; and determining a video segment from the multiple initial video segments based on the similarity.

[0082] According to an embodiment of the present disclosure, each video storyboard text may include at least one keyword, multiple initial video clips may be extracted based on each keyword, and multiple initial video clips may be extracted based on at least one keyword. The lengths of the multiple initial video clips may be the same or different, that is, the multiple initial video clips may include the same or different numbers of video frames.

[0083] Since the initial video clip is determined based on the keywords of the video storyboard text, the accuracy of keyword extraction of the video storyboard text will affect the accuracy of the initial video clip searched, so there may be a deviation between the extracted initial video clip and the video storyboard text. Therefore, when searching for the initial video clip based on keywords, as many as possible can be searched from the database, and then the initial video clip and the video storyboard text are screened by the similarity, so as to reduce the error between the initial video clip and the video storyboard text.

[0084] For example, the initial video segment with the highest similarity to the video storyboard text among the multiple initial video segments may be used as the video segment matching the video storyboard text.

[0085] The embodiments of the present disclosure calculate the similarity between the initial video clip and the video storyboard text, and screen multiple initial video clips based on the similarity, thereby effectively reducing the content difference between the initial video clip searched in the video material collection stage and the video storyboard text creation stage, thereby helping to improve the content relevance between the finally generated target video and user needs, and improving the video generation quality.

[0086] In order to more clearly describe the search operation of multiple initial video clips, two embodiments are explained below.

[0087] Figure 4A The flowchart of determining a video segment according to a specific embodiment of the present disclosure is schematically shown.

[0088] like Figure 4A As shown, embodiment 400A can extract N keywords from video storyboard text 401, such as keyword 1 402_1 ... keyword N 402_N. Figure 4A In the embodiment 400A, for each keyword, at least one initial video segment can be searched from the database. For example, according to keyword 1 402_1, P initial video segments are searched from the database, namely initial video segment 1 4031_1, ... initial video segment P 4031_P; ...; according to keyword N 402_N, Q initial video segments are searched from the database, namely initial video segment N1 403N_1, ... initial video segment NQ 403N_Q.

[0089] For the video clips searched according to each keyword, a deduplication operation may be performed first, so as to obtain as many video clips as possible that match the video storyboard text while reducing the amount of data for subsequent similarity calculations.

[0090] Subsequently, by calculating the similarity between each initial video segment and the video storyboard text 401 , a video segment 404 that best matches the video storyboard text 401 is obtained from the multiple initial video segments.

[0091] Figure 4B The flowchart of determining a video segment according to another specific embodiment of the present disclosure is schematically shown.

[0092] Figure 4B The video storyboard text 401 of embodiment 400B and the N keywords extracted from the video storyboard text 401 and Figure 4A Similar, no further elaboration is required here.

[0093] The N keywords extracted from the video storyboard text 401 constitute a keyword set 403. In the keyword set 403, the N keywords can be arranged randomly or in a preset order, for example, in the order in which they appear in the video storyboard text 401. Afterwards, the keyword set 403 is used as a search item to search for V initial video clips from the database, such as initial video clip 1 404_1, ..., initial video clip V 404_V. Subsequently, by calculating the similarity between each initial video clip and the video storyboard text 401, a video clip 405 that best matches the video storyboard text 401 is obtained from the multiple initial video clips. The keyword set is mainly used to directly find the initial video clips that hit multiple keywords at the same time in the database, reduce the amount of data to be searched, and improve the search efficiency.

[0094] For example, for video storyboard text , using the keyword extraction model KeyBert to extract three keywords , , , forming a keyword set ,in, .

[0095] For database , based on a keyword set A collection of m initial video clips was searched ,in, .

[0096] According to an embodiment of the present disclosure, determining the similarity between each initial video clip and a video storyboard text includes: determining a first vector corresponding to the initial video clip, wherein the first vector is used to characterize image features of the initial video clip; converting the video storyboard text into a second vector; and determining the similarity between the initial video clip and the video storyboard text based on the first vector and the second vector.

[0097] According to an embodiment of the present disclosure, the first vector corresponding to the initial video clip may be obtained by converting text data, or may be obtained based on image data.

[0098] The text data may describe the image features of the initial video clip in the form of text, and thus the first vector converted from the text data may indirectly represent the image features of the initial video clip. The image data may be the initial video clip itself, or may be a video frame selected from at least one video frame included in the initial video clip, and thus the first vector converted from the image data may directly reflect the image features of the initial video clip.

[0099] For text data, a text encoder may be used to encode the text data into a first vector corresponding to the initial video segment; for image data, an image encoder may be used to encode the image data into a first vector corresponding to the initial video segment.

[0100] Similarly, for the video storyboard text, a text encoder may be used to encode the video storyboard text into a second vector.

[0101] In the embodiments of the present disclosure, the similarity between the first vector and the second vector can be calculated by various distance measurement algorithms, such as cosine similarity, Euclidean distance, etc. The similarity between the first vector and the second vector is the similarity between the initial video clip and the video storyboard text.

[0102] In the above embodiment, the text encoder can be implemented by an existing text encoding algorithm, and the image encoder can be implemented by an existing image encoding algorithm. In the case where the first vector is obtained by converting text data, the video storyboard text and the text data can be encoded using the same or different text encoders.

[0103] Alternatively, a multimodal model may be used to determine the similarity between each initial video clip and the video storyboard text. For example, each initial video clip and the video storyboard text may be input into a multimodal model of image + language (Contrastive Language-Image Pre-training, CLIP), and the CLIP model directly outputs the similarity between each initial video clip and the video storyboard text.

[0104] According to an embodiment of the present disclosure, determining a first vector corresponding to an initial video clip includes: determining a key video frame from at least one video frame; encoding the key video frame into a first vector; or, encoding each video frame of at least one video frame into a third vector; taking an average vector of at least one third vector as a first vector; or, encoding a material description text into a first vector.

[0105] In one embodiment, each initial video segment may include at least one video frame, and the at least one video frame is arranged in a time sequence according to the video shooting. The key video frame in the at least one video frame may be a cover video frame. Since the video segment arrangements and the definition algorithms of the cover video frames in different databases are different, the cover video frame defined in the database may be directly determined as the key video frame. Alternatively, for an initial video segment without a cover video frame, a video frame may be selected as the key video frame from at least one video frame according to actual needs or randomly. Afterwards, the key video frame may be encoded into a first vector using an image encoder.

[0106] In another embodiment, each video frame in at least one video frame is first encoded into a third vector, and then an average vector of at least one third vector is calculated to obtain a first vector capable of representing image features of the initial video clip.

[0107] In yet another embodiment, the material description text encoding for the initial video clip in the database may be directly encoded into the first vector, that is, the material description text is the text data mentioned above.

[0108] In addition, at least two of the above three embodiments may be combined to obtain a comprehensive vector by weighting or the like, and the comprehensive vector may be used as the first vector.

[0109] Figure 5 A schematic diagram schematically shows multiple initial video clips obtained according to an embodiment of the present disclosure.

[0110] For two keywords "dog" and "play" extracted from a certain video storyboard text, three initial video clips are searched through the keyword set, such as the first video clip 501, the second video clip 502, and the third video clip 503. The first video clip 501 includes a cover video frame 5011 and a material description text 5012. In addition, each initial video clip may also include information such as the length and clarity of the video. , the first vector corresponding to the first video clip 501 can be calculated based on the cover video frame 5011 or the material description text 5012.

[0111] According to an embodiment of the present disclosure, before determining the first vector according to the key video frame, the key video frame may be downsampled to facilitate calculating the similarity between the first vector and the second vector in the same spatial dimension. Similarly, before encoding each video frame in the at least one video frame into a third vector, each video frame may be downsampled.

[0112] The embodiments of the present disclosure filter and match the initial video clips based on similarity, thereby selecting the video clips that are most similar to the video storyboard text, which can at least partially optimize the video content of the generated target video and improve the efficiency and accuracy of the material collection stage.

[0113] According to an embodiment of the present disclosure, for operation S210, scene prompt text is generated based on the video content description text input by the user, including: determining a video generation scene related to the video content description text based on the video content description text; obtaining a prompt template matching the video generation scene, wherein the prompt template includes video restriction information for the video generation scene; and generating scene prompt text based on the video content description text and the prompt template.

[0114] For users who are not proficient in video editing skills or are just getting started, the video content description text entered by the user may not be detailed, and the video storyboard text generated based only on the video content description text entered by the user is relatively broad and may not fit the actual needs of the user. Therefore, the embodiments of the present disclosure optimize the input of the large language model.

[0115] In the embodiments of the present disclosure, prompt templates for various video generation scenarios can be pre-built based on actual research or scenario analysis. The prompt templates can include video restriction information related to the video generation scenario, such as the number of video storyboards, the length of storyboards, and other special requirements, such as video style.

[0116] Therefore, in the actual application process of video generation, the video content description text input by the user is analyzed to determine the video generation scene related to the video content description text, and then obtain the prompt template matching the video generation scene. The prompt templates of multiple video generation scenes can be the same or different.

[0117] For example, the video generation scene determined from the user's video content description text is , combine predefined and video generation scenes Combined with the matching prompt text, we can get the specific text description of the input large language model. ,in, Taking the large language model A as an example, the output of A is expressed as: ,in, , n represents the number of video storyboards, that is, the number of video storyboard texts.

[0118] For example, when the video description content input by the user is a short video theme A, a prompt template matching the video theme can be obtained. In this case, the video generation scene is the theme. Alternatively, when the video description content input by the user is a short name of an object, the video generation scene can be determined to be a product display / sale, and then a prompt template matching the product display / sale can be obtained. Alternatively, when the video content description text input by the user is a longer description text, the video generation scene can be extracted from the video content description text. The extracted video generation scene can be the video theme, the product display / sale, or other scenes.

[0119] After obtaining the prompt template that matches the video generation scene, the video content description text and the prompt template can be combined to generate the scene prompt text. For example, the specific video theme A and the specific object name B in the video content description text can be directly combined with the prompt template, or the video content description text can be combined with the prompt template after content cropping and keyword extraction.

[0120] Before using the large language model to output the video storyboard text, the embodiment of the present disclosure optimizes the data of the large language model based on the video generation scenario related to the video content description text, thereby obtaining a more accurate, concise, and more scene-specific scene prompt text, thereby improving the accuracy of the video storyboard text output by the large language model.

[0121] According to an embodiment of the present disclosure, the method also includes: in response to detecting a preset interactive operation of a user, obtaining at least one custom video restriction information corresponding to the preset interactive operation; updating a prompt template using at least one custom video restriction information; and generating a scene prompt text based on the updated prompt template and the video content description text.

[0122] Considering that the user may have more specific customization requirements, the information in the prompt template may not be comprehensive or may not meet the user's actual customization requirements. Therefore, the user's actual customization requirements can also be obtained by providing an interactive page to the user. The actual customization requirements can be reflected in the prompt template in the form of customized video restriction information.

[0123] For example, when there is video restriction information corresponding to the custom video restriction information in the prompt template, such as the number of video storyboards, the user's custom video restriction information shall prevail. When there is no video restriction information corresponding to the custom video restriction information in the prompt template, the custom video restriction information can be imported through the custom item in the prompt template to realize the customized update of the prompt template.

[0124] The embodiments of the present disclosure obtain the user's customized video restriction information through interactive pages and the like, and can flexibly generate a video customization requirement that better meets a specific user during the video generation process, thereby improving the user's experience.

[0125] In one embodiment, for operation S240, a target video may be generated directly based on the video storyboard text and at least one video segment. Specifically, the at least one video segment may be combined into a target video according to the arrangement order of the at least one video storyboard text. The arrangement order of the at least one video storyboard text may be the order in which the large language model outputs the at least one video storyboard text. The target video generated by this embodiment may be a video that includes only elements in the video segment.

[0126] In another embodiment, for operation S240, at least one video storyboard text output by the large language model may be made into audio data, and the audio data and at least one video clip may be combined into a target video. The target video generated by this embodiment may be a multimedia video including a video clip and audio data.

[0127] Generating a target video matching a video content description text according to a video storyboard text and at least one video clip also includes: converting each video storyboard text in the at least one video storyboard text into an initial audio clip; combining a preset smooth audio clip and at least one initial audio clip into audio data according to the arrangement order of the at least one video storyboard text; and generating a target video matching the video content description text according to the audio data and the at least one video clip.

[0128] In this embodiment, an audio processing model can be used to convert each video storyboard text into an initial audio segment. For example, an open source text-to-speech model (TTS) is used to convert each video storyboard text into an initial audio segment. For example, at least one initial audio segment can constitute an audio set , where n represents the number of initial audio segments.

[0129] As an illustrative example, the converted initial audio clip may be a human voice, and the timbre, pitch, and other information of the human voice may be determined based on the video storyboard text, or may also be determined from the scene prompt text. For example, the timbre, pitch, and other information of the human voice may be predetermined in the prompt template, or may be determined based on the user's customized video restriction information.

[0130] Then, at least one initial audio segment is combined according to the arrangement order of at least one video storyboard text, and a preset smooth audio segment is added before / after each initial audio segment, so that the overall transition of the generated audio data is more natural. For example, the preset smooth audio segment can be a silent segment with a fixed length of τ.

[0131] For example, audio data ,in, represents the i-th initial audio segment, Represents a preset smooth audio clip. Can be a predefined starting audio segment.

[0132] The embodiments of the present disclosure generate multiple initial audio clips based on the video storyboard text, and generate a target video with smooth and natural transitions based on at least one initial audio data and at least one video clip, thereby expanding the video generation scenario, generating a target video with richer and more diverse content, and improving the user experience.

[0133] According to an embodiment of the present disclosure, in the case where the target video includes audio data and at least one video segment, the audio and video synchronization problem of the video is taken into consideration, and thus, a target video corresponding to a user video content description text is generated based on the audio data and the at least one video segment, including: cropping each video segment to obtain a target video segment that matches the length of the audio segment corresponding to the video segment; and generating a target video corresponding to the user video content description text based on the audio data and the at least one target video segment.

[0134] According to the embodiment of the present disclosure, a video segment can be searched for and an initial audio segment can be generated based on each video storyboard text. For each initial audio segment, a new audio segment can be generated after adding a preset smooth audio segment. Therefore, the new audio segment generated above can be used as the audio segment corresponding to the video segment.

[0135] According to an embodiment of the present disclosure, for each video clip, it can be cropped into a target video clip that matches the length of its corresponding audio clip, ensuring that the audio data is synchronized with the video clip. In another embodiment, the cropping operation may not be performed on each video clip, and the audio data is synchronized with the video clip by adjusting the playback speed of the video clip.

[0136] In yet another embodiment, in the process of generating a target video including audio data and at least one video clip, background audio data may also be added to the audio data to obtain updated audio data, that is, audio data including BGM.

[0137] The background audio data may be predetermined, for example, may be a certain fixed BGM.

[0138] The background audio data can also be extracted from the video content description text input by the user. For example, the video content description text can be "I want a fierce equestrian competition video", from which keywords such as "fierce" and "competitive" related to the video style can be extracted. Therefore, when adding background audio data, a BGM can be randomly selected from multiple BGMs such as "fierce" and "competitive".

[0139] The background audio data can also be determined based on the user's preset interactive operation. For example, the user can input the video style of the background audio data through the interactive page. On the one hand, the video style can be updated to the prompt template as custom video restriction information, so that the video storyboard text output by the large language model can reflect the video style, which is convenient for the subsequent selection of background audio data based on the video style in the video storyboard text. Alternatively, the video style input by the user through the interactive page can be directly obtained, and the background audio data can be selected based on the video style.

[0140] For example, the updated audio data ,in, is the audio data before updating. is the background audio data, Refers to the add operation.

[0141] According to an embodiment of the present disclosure, whether to add background audio data to the audio data can also be predetermined, or extracted from the video content description text, or determined based on a preset interactive operation of the user. This operation is similar to the style selection of the background audio data described above, and will not be repeated here.

[0142] The embodiments of the present disclosure not only include audio data of human voice narration in the target video, but also add background audio data, that is, background music, so as to generate a richer target video, meet the user's needs from an audio perspective, and improve the user's experience.

[0143] According to an embodiment of the present disclosure, in order to further enrich the content of the generated target video, subtitles can also be added to the video clips to optimize the visual display effect of the target video. Specifically, the method also includes: generating video subtitle data corresponding to each video clip; and generating a target video corresponding to the user video content description text based on the audio data, at least one video clip and at least one video subtitle data.

[0144] In this embodiment, each initial audio segment can be converted into a voice-to-text format to obtain video subtitle data that is consistent with the human voice in the initial audio segment. Afterwards, the video subtitle data is added to each video segment of the video storyboard text to obtain a video stream consisting of at least one video segment including subtitles. After that, the audio data / the audio data of the newly added background audio data and the video stream are superimposed to generate the target video. ,in, .

[0145] For example, the open source ffmpeg video processing library may be used to superimpose the audio data / audio data of the newly added background audio data and the video stream to obtain the target video.

[0146] The embodiments of the present disclosure integrate some common video post-processing contents into the video generation process during the material combination and post-processing stages, eliminating the need for users to participate in video post-processing and reducing workload.

[0147] Figure 6 The flowchart of generating a target video according to a video content description text input by a user according to an embodiment of the present disclosure is schematically shown.

[0148] In this embodiment 600, the user can obtain a complete target video by simply inputting a video content description text.

[0149] like Figure 6 As shown, the video content description text 601 input by the user can be optimized according to the frequency generated scene extracted from the video content description text 601 to generate the scene prompt text 602. The scene prompt text 602 is input into the large language model M10, and a video storyboard text 603 can be output. The large language model M10 can output one or more video storyboard texts 603. Then, the audio data and at least one video clip are processed respectively through two branches.

[0150] In the left branch, the video storyboard text 603 can be converted into an initial audio segment using an audio processing model M20. The audio processing model M20 can be a TTS model. In addition, the left branch can also perform smoothing and combination processing on at least one initial audio segment to obtain audio data. Alternatively, background audio data can also be added to the audio data to obtain updated audio data.

[0151] In the right branch, the large language model M10 or the keyword extraction model can be used to extract keywords 604 from the video storyboard text 603, and one or more keywords can be extracted from each video storyboard text 603. The keyword 604 is input into the search engine M30, and the initial video clip matching the keyword 604 can be searched from the database. The search engine M30 supports searching using one keyword and also supports searching using a keyword set of multiple keywords.

[0152] At least one initial video clip 605 is formed into a video sequence, and is input into the multimodal processing model M40 together with the video storyboard text 603, and the similarity between each initial video clip 605 and the video storyboard text 603 is output. The multimodal processing model M40 can be a CLIP model. Based on the similarity between each initial video clip 605 and the video storyboard text 603, a video clip 606 matching the video storyboard text 603 can be screened out from at least one initial video clip 605. For each video storyboard text 603 output by the large language model M10, the matching video clip can be obtained in the above manner.

[0153] Afterwards, the audio data or updated audio data output by the left branch is merged with at least one video segment 606 output by the right branch to obtain a merged video 607 .

[0154] In this embodiment, the fused video 607 can be directly output, and special effects / video post-processing can be performed on the fused video to generate a target video 608. For example, editing, video length processing, simple special effects processing, etc. can be performed.

[0155] Thus, for the video content description text input by the user, a target video corresponding to the video content description text can be generated and output to the user.

[0156] This paper carefully analyzes each process of intelligent video content production, and optimizes the problems in each process in combination with existing technical conditions, so as to greatly save the time and efficiency of content production, while improving the quality of video content generation. Considering that our tool is end-to-end, only the user needs to input a specific topic (that is, the video content description text) to output the target video without intervention in a short time, which provides a way for some users who lack professional knowledge of video editing to produce their own video content.

[0157] In addition, of course, the various processes involved in the present disclosure can also be split out to provide assistance to professional users in different video production processes, reflecting the effectiveness and flexibility of the embodiments of the present disclosure.

[0158] Figure 7The structural block diagram of a video generating device according to an embodiment of the present disclosure is schematically shown.

[0159] like Figure 7 As shown, the video generating device 700 of this embodiment includes a first generating module 710 , a second generating module 720 , a determining module 730 and a third generating module 740 .

[0160] The first generating module 710 is used to generate a scene prompt text according to the video content description text input by the user, wherein the scene prompt text is used to limit the generation scene of the video.

[0161] The second generation module 720 is used to input the scene prompt text into the large language model and output at least one video storyboard text.

[0162] The determination module 730 is used to determine the video clip corresponding to each video storyboard text from the database according to the keywords of each video storyboard text.

[0163] The third generating module 740 is used to generate a target video matching the video content description text according to the video storyboard text and at least one video clip.

[0164] According to an embodiment of the present disclosure, the video generation device 700 further includes a keyword extraction module, which is used to input each video storyboard text into a large language model and output at least one keyword in each video storyboard text.

[0165] According to an embodiment of the present disclosure, the determination module 730 includes a search submodule, a similarity determination submodule and a screening submodule. The search submodule is used to search for multiple initial video clips matching the keyword from the database according to the keyword. The similarity determination submodule is used to determine the similarity between each initial video clip and the video storyboard text. The screening submodule is used to determine the video clip from the multiple initial video clips according to the similarity.

[0166] According to an embodiment of the present disclosure, the similarity determination submodule includes a first encoding unit, a second encoding unit and a similarity determination unit. The first encoding unit is used to determine a first vector corresponding to an initial video segment, wherein the first vector is used to characterize image features of the initial video segment. The second encoding unit is used to convert the video storyboard text into a second vector. The similarity determination unit is used to determine the similarity between the initial video segment and the video storyboard text based on the first vector and the second vector.

[0167] According to an embodiment of the present disclosure, each initial video segment includes at least one video frame or a material description text for at least one video frame. The first encoding unit includes a first encoding subunit, a second encoding subunit, and a third encoding subunit. The first encoding subunit is used to determine a key video frame from at least one video frame; and encode the key video frame into a first vector. The second encoding subunit is used to encode each video frame in at least one video frame into a third vector; and use the average vector of at least one third vector as the first vector. The third encoding subunit is used to encode the material description text into a first vector.

[0168] According to an embodiment of the present disclosure, the first generation module 710 includes a scene determination submodule, a template acquisition submodule and a text generation submodule. The scene determination submodule is used to determine a video generation scene related to the video content description text according to the video content description text. The template acquisition submodule is used to obtain a prompt template matching the video generation scene, wherein the prompt template includes video restriction information for the video generation scene. The text generation submodule is used to generate a scene prompt text according to the video content description text and the prompt template.

[0169] According to an embodiment of the present disclosure, the video generation device 700 further includes: an interaction module, a template update module and a fourth generation module. The interaction module is used to obtain at least one custom video restriction information corresponding to the preset interaction operation in response to detecting a preset interaction operation of the user. The template update module is used to update the prompt template using the at least one custom video restriction information. The fourth generation module is used to generate a scene prompt text according to the updated prompt template and the video content description text.

[0170] According to an embodiment of the present disclosure, the third generation module 740 includes a first generation submodule, which is used to combine at least one video segment into a target video according to the arrangement order of at least one video storyboard text.

[0171] According to an embodiment of the present disclosure, the third generation module 740 further includes: a first audio processing submodule, a second audio processing submodule, and a second generation submodule. The first audio processing submodule is used to convert each video storyboard text in at least one video storyboard text into an initial audio segment. The second audio processing submodule is used to combine a preset smooth audio segment and at least one initial audio segment into audio data according to the arrangement order of at least one video storyboard text. The second generation submodule is used to generate a target video that matches the video content description text based on the audio data and at least one video segment.

[0172] According to an embodiment of the present disclosure, the third generation module 740 also includes a background audio processing submodule for adding background audio data to the audio data to obtain updated audio data; wherein the background audio data is predetermined; or, the background audio data is extracted from the video content description text input by the user; or, the background audio data is determined based on the user's preset interactive operation.

[0173] According to an embodiment of the present disclosure, the second generation submodule includes a video processing unit and a merging unit. The video processing unit is used to crop each video segment to obtain a target video segment that matches the length of the audio segment corresponding to the video segment. The merging unit is used to generate a target video corresponding to the user video content description text based on the audio data and at least one target video segment.

[0174] According to an embodiment of the present disclosure, the third generation module 740 further includes a first subtitle processing submodule and a second subtitle processing submodule. The first subtitle processing submodule is used to generate video subtitle data corresponding to each video segment. The second subtitle processing submodule is used to generate a target video corresponding to the user video content description text according to the audio data, at least one video segment and at least one video subtitle data.

[0175] According to an embodiment of the present disclosure, any multiple modules among the first generation module 710, the second generation module 720, the determination module 730 and the third generation module 740 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0176] According to an embodiment of the present disclosure, at least one of the first generation module 710, the second generation module 720, the determination module 730, and the third generation module 740 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, at least one of the first generation module 710, the second generation module 720, the determination module 730, and the third generation module 740 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function may be executed.

[0177] It should be noted that the device part in the embodiments of the present disclosure corresponds to the method part in the embodiments of the present disclosure. The description of the device part specifically refers to the method part and will not be repeated here.

[0178] Figure 8 A block diagram of an electronic device suitable for a video generating method according to an embodiment of the present disclosure is schematically shown.

[0179] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0180] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0181] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the input / output I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage portion 808 as needed.

[0182] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0183] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0184] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the above method provided by the embodiment of the present disclosure.

[0185] The above functions defined in the system / device of the embodiment of the present disclosure are executed when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0186] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0187] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0188] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0189] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0191] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above description is only a specific embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A video generation method, comprising: Generate a scene prompt text according to the video content description text input by the user, wherein the scene prompt text is used to limit the generation scene of the video; Inputting the scene prompt text into a large language model, and outputting at least one video storyboard text; Determining, from a database, a video clip matching each of the video storyboard texts according to the keywords of each of the video storyboard texts; and A target video matching the video content description text is generated based on the video storyboard text and at least one of the video clips.

2. The method according to claim 1, further comprising: Each of the video storyboard texts is input into the large language model, and at least one keyword in each of the video storyboard texts is output.

3. The method according to claim 1, wherein: The step of determining a video segment matching each of the video storyboard texts from a database according to the keywords of each of the video storyboard texts comprises: According to the keyword, searching the database for a plurality of initial video clips matching the keyword; Determining the similarity between each of the initial video segments and the video storyboard text; and The video segment is determined from the multiple initial video segments according to the similarity.

4. The method according to claim 3, wherein: The determining of the similarity between each of the initial video segments and the video storyboard text comprises: Determine a first vector corresponding to the initial video segment, wherein the first vector is used to characterize an image feature of the initial video segment; Converting the video storyboard text into a second vector; and The similarity between the initial video clip and the video storyboard text is determined based on the first vector and the second vector.

5. The method according to claim 4, wherein: Each of the initial video segments includes at least one video frame or a material description text for the at least one video frame; the determining a first vector corresponding to the initial video data includes: Determine a key video frame from the at least one video frame; encode the key video frame into the first vector; or, Encode each video frame in the at least one video frame into a third vector; use an average vector of at least one of the third vectors as the first vector; or, The material description text is encoded into the first vector.

6. The method according to claim 1, wherein: The step of generating scene prompt text according to the video content description text input by the user includes: Determining a video generation scene related to the video content description text according to the video content description text; Acquire a prompt template matching the video generation scenario, wherein the prompt template includes video restriction information for the video generation scenario; and The scene prompt text is generated according to the video content description text and the prompt template.

7. The method according to claim 6, further comprising: In response to detecting a preset interactive operation of a user, obtaining at least one customized video restriction information corresponding to the preset interactive operation; Updating the prompt template using the at least one custom video restriction information; as well as The scene prompt text is generated according to the updated prompt template and the video content description text.

8. The method according to claim 1, wherein: The step of generating a target video matching the video content description text according to the video storyboard text and the at least one video clip comprises: According to the arrangement order of the at least one video storyboard text, at least one of the video segments is combined into the target video.

9. The method according to claim 1, wherein: The step of generating a target video matching the video content description text according to the video storyboard text and the at least one video clip further includes: Converting each video storyboard text in the at least one video storyboard text into an initial audio clip; Combining a preset smooth audio segment and at least one of the initial audio segments into audio data according to the arrangement order of the at least one video storyboard text; and A target video matching the video content description text is generated according to the audio data and at least one of the video clips.

10. The method according to claim 9, further comprising: Adding background audio data to the audio data to obtain updated audio data; The background audio data is predetermined; or, the background audio data is extracted from a video content description text input by a user; or, the background audio data is determined based on a preset interactive operation of the user.

11. The method according to claim 9, wherein: The step of generating a target video corresponding to the user video content description text according to the audio data and at least one of the video clips includes: cropping each of the video segments to obtain a target video segment that matches the length of the audio segment corresponding to the video segment; and A target video corresponding to the user video content description text is generated according to the audio data and at least one of the target video segments.

12. The method according to claim 9, further comprising: Generating video subtitle data corresponding to each of the video clips; as well as A target video corresponding to the user video content description text is generated according to the audio data, at least one of the video clips and at least one of the video subtitle data.

13. A video generating device, comprising: A first generating module, configured to generate a scene prompt text according to a video content description text input by a user, wherein the scene prompt text is used to define a generation scene of the video; A second generating module is used to input the scene prompt text into a large language model and output at least one video storyboard text; a determination module, configured to determine, from a database, a video segment corresponding to each of the video storyboard texts according to a keyword of each of the video storyboard texts; and The third generating module is used to generate a target video matching the video content description text according to the video storyboard text and at least one of the video clips.

14. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images