Text-based automatic video generation method

Abstract

The invention provides a text-based automatic video generation method. The text-based automatic video generation method comprises the following steps: creating a video vector library; constructing a video description document, wherein the video description document comprises third-party website acquisition and manual writing; executing a paragraph separation instruction according to the video description document to obtain text fragments; executing the vector model operation instruction in S1-5 according to the text fragment to obtain a vector fragment list; executing a video retrieval instruction according to the vector fragment list and the video vector library to obtain an output video fragment list, wherein the video retrieval instruction comprises executing a search instruction of the video vector library to obtain a closest video picture content description vector and the corresponding video fragment list; and executing a video merging instruction according to the output video clip list to obtain a complete video. The problem of low efficiency of artificially synthesizing the video is solved.

Description

Technical Field

[0001] The present invention belongs to the field of AI, and in particular relates to a text-based automatic video generation method. Background Art

[0002] A complete video is usually composed of multiple video clips. For a given text, it is often necessary to obtain multiple video clips related to the text content from the video material library and synthesize them in order. However, at present, when synthesizing videos for a given text, it still depends on manually providing video segments that match the text, or manually annotating the video material library and then using keyword matching to obtain video segments related to the given text. The former requires high labor costs and is not efficient. The latter uses keyword matching to obtain video segments, which cannot make full use of the video screen content and is not accurate and effective enough. In addition, manually shot videos often need to be cropped of irrelevant content before they can be used as materials. Therefore, an automatic and accurate method is needed to match content-related video clips for a given text, so as to synthesize high-quality videos to meet the needs of various film and television creation tasks. Summary of the invention

[0003] The purpose of this embodiment is to provide a text-based automatic video generation method to solve the problem of low efficiency of artificial video synthesis.

[0004] A text-based automatic video generation method, comprising:

[0005] S1: Create a video vector library, including:

[0006] S1-1: Get the original video file;

[0007] S1-2: Construct a content description script, including role setting, task objectives, and video description;

[0008] S1-3: Execute multimodal large model operation to obtain a video information description script according to the original video file and the content description script; wherein the content of the information description script includes timestamp and narration fields;

[0009] S1-4: Execute a video cutting instruction according to the timestamp and the original video file to obtain multiple video clips;

[0010] S1-5: Execute a vector model operation instruction according to the video clip to obtain a vector list;

[0011] S1-6: Execute vector library operation according to the vector list to construct the video vector library, wherein the video vector library is an object variable and has a built-in search instruction;

[0012] S2: Construct video description documents, including obtaining from third-party websites and manual writing;

[0013] S3: Execute a paragraph separation instruction according to the video description document to obtain a text segment;

[0014] S4: Execute the vector model operation instruction as described in S1-5 according to the text segment to obtain a vector segment list;

[0015] S5: executing a video search instruction according to the vector segment list and the video vector library to obtain an output video segment list, wherein the video search instruction includes executing a search instruction of the video vector library to obtain the closest video picture content description vector and the corresponding video segment list;

[0016] S6: Execute a video merging instruction according to the output video clip list to obtain a complete video, wherein the video merging instruction includes a concatenate_videoclips function of a moviepy library.

[0017] Furthermore, the content description script includes a json script.

[0018] Furthermore, multimodal large models include gemini models.

[0019] Furthermore, the video cutting instruction includes executing the VideoFileClip function by traversing the video information description script content using the moviepy tool.

[0020] Furthermore, the video cutting instruction also includes constructing a sub_videos_info list according to the narration and the video clip; wherein the narration and the video clip are in a one-to-one correspondence.

[0021] Furthermore, the vector model operation includes executing text replacement instructions and OpenAI.embeddings.create instructions according to the text-embedding-3-small model.

[0022] Furthermore, the vector library operation includes executing IndexFlatIP operation according to the faiss vector database to obtain the video vector library.

[0023] Furthermore, paragraph separation instructions include the splite instruction of the article library.

[0024] The invention provides a text-based automatic video generation method which solves the problem of low efficiency of artificially synthesized videos.

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 : A step diagram of the text-based automatic video generation method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] This embodiment provides a text-based automatic video generation method (see Figure 1 ),include:

[0029] The specific steps are as follows:

[0030] S1. Prepare an original video library Video_files, which stores multiple original video files video_file in mp4 format. These video files can be pre-shot or downloaded public video content. The specific steps in this embodiment are as follows:

[0031] S1-1. Obtain the original video file, which is composed of multiple video files and placed in a specified disk directory. The video files are downloaded from an existing public network.

[0032] S1-2. Design prompt to prompt the multimodal large model to describe the video content. A design example in the embodiment is as follows:

[0033] prompt="""

[0034] #Character settings:

[0035] You are a film and television commentary expert who is good at generating interesting and attractive commentary based on the video footage.

[0036] #Mission Objectives:

[0037] -Describe the given video footage in detail and expand the narrative, especially the important scenes.

[0038] -The final result is output in JSON format, and the fields include:

[0039] "timestamp": timestamp (indicates the time when the picture appears - the time when the picture ends)

[0040] "narration": the video content description of the clip

[0041] #Video Description:

[0042] -The generated commentary copy should be concise and powerful, in line with the preferences of users of short video platforms.

[0043] - Be specific in your description and mention the key elements of the video.

[0044] -The copywriting language is: Chinese """.

[0045] S1-3. Use the multimodal large model gemini to process each original video file video_file in step 1. Input each video file video_file in step 1 and the prompt in step 2 to obtain the video content description information script. The code for calling the gemini model interface is as follows:

[0046] import google.generativeai as gemini

[0047] model=gemini.GenerativeModel(model_name='gemini-1.5-pro')

[0048] gemini_video_file=gemini.upload_file(video_file)

[0049] script=model.generate_content([prompt,gemini_video_file])

[0050] The output video content description information script is in JSON format, containing one or more data, each of which consists of a start and end time timestamp and a narration of the video content within the time period. An example of a generated video content information script is as follows:

[0051] script="'[

[0052] {"timestamp":"00:00-00:07",

[0053] "narration":"The sun sets, golden light fills the earth, a car drives on a winding road, heading into the distance."},

[0054] {"timestamp":"00:07-00:13",

[0055] "narration":"The mountains in the distance are still covered with snow, and the sunlight shining on them is magnificent! This scenery is simply a paradise on earth!"}

[0056] ]”'

[0057] S1-4. Using the python moviepy toolkit, combined with the start and end timestamps in the video content information script output in step S1-3, the original video file video_file is divided into multiple more refined video segments sub_video, and matched with the video screen content description narration in step S1-3 to obtain the sub_videos_info list. In the embodiment, the content description narration and the video segment sub_videos are in a one-to-one correspondence; the code is as follows:

[0058] from moviepy.video.io.VideoFileClip import VideoFileClip

[0059] sub_videos_info={}

[0060] for itemin json.loads(script):

[0061] start,stop=item["timestamp"].split("-")

[0062] sub_video=VideoFileClip(origin_video).subclip(start,end)

[0063] sub_videos_info[item["narration"]]=sub_video

[0064] S1-5. Use AI's vector model text-embedding-3-small to vectorize the content description narration of each video screen, obtain the vector narration_embedding of the video screen content description, and collect each obtained vector to obtain a vector list narration_embeddings. The Python sample execution code of this process is:

[0065]

[0066] S1-6. Further use the faiss vector database to construct the obtained vector list narration_embeddings into a vector library Embedding_Database. The execution code of this process is:

[0067] import faiss

[0068] faiss.normalize_L2(narration_embeddings)

[0069] dimension=narration_embedding.shape[1]

[0070] Embedding_Database=faiss.IndexFlatIP(dimension)

[0071] Embedding_Database.add(standard_embeddings)

[0072] S2. Construct a video description document. For a given text article, the description document in this embodiment can be obtained from a third-party website or manually written. The third-party website can obtain, for example, a news website or a novel website by parsing the page through Python language to obtain the text description and use it as the article in this embodiment; the example content is as follows:

[0073] article="'From an aerial perspective, a straight road stretches into the distance, with endless green grasslands on both sides of the road and rolling mountains in the distance. The clouds in the sky glow orange-red, and the sun shines through the clouds, forming beams of light that look like stairs to heaven.

[0074] The camera moves slowly, and you can see the hillside covered with snow, and in the distance are rolling mountains. The sun shines on the peaks, forming golden light and shadows. '

[0075] S3. Segment it into multiple text fragments by sentence; in this embodiment, the sentence segmentation is based on the character "\n", and the character "." can also be used to separate sentences, where the character "." is in Chinese encoding format; the code is:

[0076] sentences=article.splite("\n" or ".")

[0077] S4. Each text segment sentence obtained in step S3 is vectorized using the vectorization method get_embedding defined in step S1-5 to obtain the vector sentence_embedding of the text segment, and each text segment vector is collected to obtain a list of text segment vectors sentence_embeddings:, the code is as follows:

[0078]

[0079] S5. Use the vector sentence_embedding of each text segment in step S4 to search from the vector database Embedding_Database constructed in step S1, and obtain the vector narration_embedding of the video screen content description closest to the text segment, and the corresponding video segment sub_video. As described in steps S1-6, the vector database is constructed using the faiss module of python and is an instance of the IndexFlatL2 class in faiss. During the retrieval process, the search method of this class can be directly used to query the first k elements close to the text segment vector sentence_embedding, where K is 1 in the embodiment; then the relevant video content description sim_narration is obtained from the content description list narrations of the video vector. The video content description obtained after the retrieval of each text segment vector sentence_embedding is collected to obtain a description list sim_narrations. The python execution code of this process is:

[0080] k=1

[0081] sim_narrations = []

[0082] for sentence_embedding in sentence_embeddings:

[0083] D, I=Embedding_Database.search(sentence_embedding,k)

[0084] score = D[0][0]

[0085] sim_narration=narrations[I[0][0]]

[0086] sim_narrations.append(sim_narration)

[0087] And obtain the corresponding video segment list sub_videos from the sub_videos_info list of the video content description and the video segment that match in S1-4; the code is as follows:

[0088] sub_videos=[sub_videos_info[sim_narration]for sim_narration in sim_narrations]

[0089] S6. Finally, use the Python moviepy toolkit to merge the obtained video clip list sub_videos in order to obtain the synthetic video final_video that is highly relevant to the given text. The code for this process is:

[0090] from moviepy.editor import VideoFileClip,concatenate_videoclips

[0091] clips=[VideoFileClip(video)for video in sub_videos]

[0092] final_clip=concatenate_videoclips(clips,method="compose")

[0093] final_clip.write_videofile("final_video.mp4",codec="libx264",fps=24)

[0094] The invention provides a text-based automatic video generation method which solves the problem of low efficiency of artificially synthesized videos.

[0095] The above description is only an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A text-based automatic video generation method, characterized in that: include: S1: Create a video vector library, including: S1-1: Get the original video file; S1-2: Construct a content description script, including role setting, task objectives, and video description; S1-3: Execute multimodal large model operation to obtain a video information description script according to the original video file and the content description script; wherein the content of the information description script includes timestamp and narration fields; S1-4: Execute a video cutting instruction according to the timestamp and the original video file to obtain multiple video clips; S1-5: Execute a vector model operation instruction according to the video clip to obtain a vector list; S1-6: Execute vector library operation according to the vector list to construct the video vector library, wherein the video vector library is an object variable and has a built-in search instruction; S2: Construct video description documents, including obtaining from third-party websites and manual writing; S3: Execute a paragraph separation instruction according to the video description document to obtain a text segment; S4: Execute the vector model operation instruction as described in S1-5 according to the text segment to obtain a vector segment list; S5: executing a video search instruction according to the vector segment list and the video vector library to obtain an output video segment list, wherein the video search instruction includes executing a search instruction of the video vector library to obtain the closest video picture content description vector and the corresponding video segment list; S6: Execute a video merging instruction according to the output video clip list to obtain a complete video, wherein the video merging instruction includes a concatenate_videoclips function of a moviepy library.

2. The text-based automatic video generation method according to claim 1, characterized in that: The content description script includes a json script.

3. The text-based automatic video generation method according to claim 1, characterized in that: The multimodal large model includes a gemini model.

4. The text-based automatic video generation method according to claim 1, characterized in that: The video cutting instruction includes executing the moviepy tool to traverse the video information description script content and run the VideoFileClip function.

5. The text-based automatic video generation method according to claim 1, characterized in that: The video cutting instruction further includes constructing a sub_videos_info list according to the narration and the video fragment; wherein the narration and the video fragment are in a one-to-one correspondence.

6. The text-based automatic video generation method according to claim 1, characterized in that: The vector model operation includes executing text replacement instructions and OpenAI.embeddings.create instructions according to the text-embedding-3-small model.

7. The text-based automatic video generation method according to claim 1, characterized in that: The vector library operation includes executing IndexFlatIP operation according to the faiss vector database to obtain the video vector library.

8. The text-based automatic video generation method according to claim 1, characterized in that: The paragraph separation instruction includes the splite instruction of the article library.

Images