Method for realizing video reconstruction agent based on large language model

Abstract

The invention provides a method for realizing a video reconstruction agent based on a large language model, and the method comprises the following steps: S1, uploading a video to a platform, and filling a video reconstruction production requirement; s2, analyzing and processing the video; s3, judging whether intelligent dubbing is needed or not according to the content of the officinal draft and the requirements of video reconstruction and production; s4, cross-modal shot matching: vectorizing the oral broadcast draft content obtained by text analysis, and carrying out shot vector matching in a vector database by using text content vectors to obtain a matched shot vector, namely a raw material required for video reconstruction; s5, generating a video synthesis script according to the raw materials and the audio file required by video reconstruction in combination with a preset packaging template; s6, the synthesis server synthesizes the video according to the video synthesis script to form a work; through an intelligent means, the existing video content is automatically reconstructed, and the video modification and production efficiency is greatly improved while the video quality and content requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of video production, and in particular to a method for realizing a video reconstruction intelligent agent based on a large language model. Background Art

[0002] According to the "China Internet Audiovisual Development Research Report (2024)", as of December 2023, the scale of online audiovisual users in my country will reach 1.074 billion, and the Internet user utilization rate will be 98.3%. In 2023, the average daily usage time of mobile Internet users will be 435 minutes, of which the average daily usage time of mobile audiovisual applications will be 187 minutes, and the average daily usage time of short videos will be 151 minutes. As of December 2023, the total number of short video accounts on the entire network will reach 1.55 billion, the number of professional anchors has reached 15.08 million, the average daily short video updates on major short video platforms are nearly 80 million, and the number of daily live broadcasts exceeds 3.5 million.

[0003] However, in order to publish their works on various short video platforms, short video operators have to publish the same content to different platforms, resulting in frequent problems such as serious homogeneity of content on various platforms, inconsistency between content and platform characteristics, and content flow restrictions on short video platforms. In order to solve problems such as content that is consistent with short video characteristics and multiple sending of content of different styles, short video operators or producers need to spend a lot of time modifying videos before publishing, which is inefficient.

[0004] Therefore, there is an urgent need for a method to realize video reconstruction intelligent agent based on a large language model. An intelligent agent is constructed to drive the entire process of video intelligent reconstruction through the operation of the intelligent agent. In the process of building the entire intelligent agent, the current mature speech recognition, speech synthesis, text recognition, large language model, cross-modal vector retrieval and other intelligent capabilities can be used. Summary of the invention

[0005] The purpose of the present invention is to provide a method for realizing video reconstruction intelligent agent based on a large language model to solve the existing problems.

[0006] To achieve the above object, the present invention provides the following solutions: The present invention provides a method for realizing a video reconstruction agent based on a large language model, comprising the following steps: S1. Upload the video to the platform and fill in the requirements for video reconstruction; S2. Analyze and process the video; S3. Determine whether intelligent dubbing is needed based on the content of the oral script and the requirements of video reconstruction and production; S4. Cross-modal shot matching: vectorize the spoken script content obtained by text analysis, and use these text content vectors to the vector database for shot vector matching. The matched shot vectors are the raw materials required for video reconstruction. S5. Generate a video synthesis script based on the raw materials and audio files required for video reconstruction and combined with a pre-set packaging template; S6. The synthesis server synthesizes the video according to the video synthesis script to form a work.

[0007] Preferably, in step S1, the requirements for video reconstruction production include duration, viewpoint, whether dubbing is required, and style.

[0008] Preferably, step S2 comprises: S21. Video text processing; S22. Video lens processing.

[0009] Preferably, step S21 includes: S211. Obtain the text of the video content through intelligent means of voice recognition, image recognition, and text recognition; S212. Unify the text of the video content and the requirements for video reconstruction and production to the large language model, and at the same time provide the large language model with necessary fine-tuned parameters and prompts, so that the large language model can generate the spoken script content required for video reconstruction according to the requirements.

[0010] Preferably, step S22 includes: S221. Obtain important lens images in the video through lens analysis; S222. Performing vector analysis on the lens to form a lens vector; S223. Store the lens vector into a vector database.

[0011] Preferably, in step S3, if intelligent dubbing is required, speech synthesis is called to generate a spoken script audio file.

[0012] Preferably, in step S5, the packaging template includes a title and subtitles.

[0013] Compared with the prior art, the present invention has achieved the following beneficial technical effects: The present invention provides a method for realizing a video reconstruction intelligent agent based on a large language model, comprising the following steps: S1. uploading a video to a platform and filling in the requirements for video reconstruction and production; S2. analyzing and processing the video; S3. judging whether intelligent dubbing is required according to the oral script content and the requirements for video reconstruction and production; S4. cross-modal shot matching, vectorizing the oral script content obtained by text analysis, and using these text content vectors to a vector database for shot vector matching, where the matched shot vectors are the raw materials required for video reconstruction; S5. generating a video synthesis script according to the raw materials and audio files required for video reconstruction and in combination with a pre-set packaging template; S6. a synthesis server synthesizes the video according to the video synthesis script to form a work; automatically reconstructing the existing video content through intelligent means, while meeting the video quality and content requirements, greatly improving the efficiency of video modification and production. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0015] Figure 1 A flow chart of a method for realizing video reconstruction intelligent agent based on a large language model provided by the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The purpose of the present invention is to provide a method for realizing video reconstruction intelligent agent based on a large language model to solve the problems existing in the prior art.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Embodiment 1: This embodiment provides a method for realizing a video reconstruction agent based on a large language model. Figure 1 As shown, the following steps are included: S1. Upload the video to the platform and fill in the requirements for video reconstruction; the requirements for video reconstruction include duration, viewpoint, whether dubbing is required, style, etc. S2. Analyze and process the video; It consists of two branches: S21. Video text processing; S211. Obtain the text of the video content through intelligent means of voice recognition, image recognition, and text recognition; S212. Unify the text of the video content and the requirements of the video reconstruction production to the large language model, and at the same time provide the large language model with necessary fine-tuned parameters and prompts, so that the large language model generates the oral script content required for video reconstruction according to the requirements; S22. Video lens processing; S221. Obtain important lens images in the video through lens analysis; S222. Performing vector analysis on the lens to form a lens vector; S223. Storing the lens vector in a vector database; S3. Determine whether intelligent dubbing is required based on the content of the oral script and the requirements of video reconstruction production; if intelligent dubbing is required, call speech synthesis to generate an oral script audio file; S4. Cross-modal shot matching: vectorize the spoken script content obtained by text analysis, and use these text content vectors to the vector database for shot vector matching. The matched shot vectors are the raw materials required for video reconstruction. S5. Generate a video synthesis script based on the raw materials and audio files required for video reconstruction and combined with a pre-set packaging template, which includes a title, subtitles, and other content; S6. The synthesis server synthesizes the video according to the video synthesis script to form a work.

[0020] The technologies applied in the present invention include: Large Language Model: LLM (Large Language Model) is an artificial intelligence model designed to understand and generate natural human language. After being trained with a large amount of text data, the large language model can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. At the same time, the large language model can perform a variety of human thinking capabilities including context learning, command understanding and execution, and reasoning. In the present invention, the large language model is mainly used to extract (or transcribe, expand, or rewrite) the content to form the text content of the script.

[0021] Agent: AI Agent is a computer program based on a large language model that has the ability to plan, think, remember, and use tool functions, and can autonomously complete a given task. The video reconstruction process drive and various intelligent capability drives in the process of the present invention are all completed by the agent.

[0022] Vectorization: The vectorization technology used in the present invention is mainly image vectorization and text vectorization.

[0023] Text vectorization: TE (Text Embedding) converts text data (including words, sentences, and document-level text) into vector representations, and converts the data into a digital form that can be processed by computers through algorithms. In the present invention, the oral script generated by the large model needs to be vectorized for use.

[0024] Image vectorization: IE (Image Embedding) is a method of converting image data into a continuous, low-dimensional vector representation after reading the image content. In the present invention, it is necessary to first image the video content through audio-visual technology, and then vectorize the content and save it in a vector database through image vectorization technology.

[0025] Vector Database: The core function of VectorDB is to store vectors in the database. When a user queries, the query is converted into a vector, and then the most similar vector and context are searched in the database, and the result vector is finally returned. In the application of the present invention, there are three links that require the use of vectors and vector databases: Shot analysis vectorization: vectorize the shot analysis transitions in the video and save them in the vector database; Text vectorization: vectorize speech recognition, text recognition and other content and save them in the vector database; Shot matching: vectorize the spoken script generated by the large model, and query and compare it with the shot vectors in the vector database to obtain the shots required to reconstruct the video.

[0026] Speech recognition: ASR (Automatic Speech Recognition) is a technology that converts speech into text. After separating the audio track in the video, the language in the audio is converted into a text format with video time code. In the present invention, the speech part of the audio and video content is converted into timeline text through speech recognition.

[0027] Shot Analysis: VSA (Video Shot Analysis) is a process of vectorizing the shot content and storing it in a vector database by analyzing the transition process in the video content. The present invention uses the shot analysis technology to vectorize the shot content and store it in a vector database.

[0028] Text recognition: OCR (Optical Character Recognition) technology extracts text information appearing in the video file and converts it into a text format with a time code. The present invention uses text recognition technology to extract the title, lyrics, name items, etc. (if any) appearing in the video to form a timeline text.

[0029] Shot Matching: VSM (Video Shot Matching) is a process of converting the oral script organized by the large model into vectors, and then searching for the same or similar vectors in the shot vector database to obtain the found video shots. The present invention vectorizes the question text content generated by the large model, and performs query matching with the shot vectors in the vector database to obtain the original shot content that can be used as video reconstruction.

[0030] Speech synthesis: TTS (Text To Speech) is a technology that converts text into anthropomorphic speech. It also requires that speech synthesis support SSML markup language and support custom polyphonic characters, volume, speech speed and other parameters. Through the application of speech synthesis, the present invention uses a virtual person with specified timbre and morphology to speak the oral script generated in the process, which is closer to the requirements of video production reconstruction.

[0031] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for realizing a video reconstruction agent based on a large language model, characterized in that: The following steps are involved: S1. Upload the video to the platform and fill in the requirements for video reconstruction; S2. Analyze and process the video; S3. Determine whether intelligent dubbing is needed based on the content of the oral script and the requirements of video reconstruction and production; S4. Cross-modal shot matching: vectorize the spoken script content obtained by text analysis, and use these text content vectors to the vector database for shot vector matching. The matched shot vectors are the raw materials required for video reconstruction. S5. Generate a video synthesis script based on the raw materials and audio files required for video reconstruction and combined with a pre-set packaging template; S6. The synthesis server synthesizes the video according to the video synthesis script to form a work.

2. The method for realizing video reconstruction agent based on large language model according to claim 1, characterized in that: In step S1, the requirements for video reconstruction production include duration, viewpoint, whether dubbing is required, and style.

3. The method for realizing video reconstruction agent based on large language model according to claim 1, characterized in that: Step S2 includes: S21. Video text processing; S22. Video lens processing.

4. The method for realizing video reconstruction agent based on large language model according to claim 3, characterized in that: Step S21 includes: S211. Obtain the text of the video content through intelligent means of voice recognition, image recognition, and text recognition; S212. Unify the text of the video content and the requirements for video reconstruction and production to the large language model, and at the same time provide the large language model with necessary fine-tuned parameters and prompts, so that the large language model can generate the spoken script content required for video reconstruction according to the requirements.

5. The method for realizing video reconstruction agent based on large language model according to claim 3, characterized in that: Step S22 includes: S221. Obtain important lens images in the video through lens analysis; S222. Performing vector analysis on the lens to form a lens vector; S223. Store the lens vector into the vector database.

6. The method for realizing video reconstruction agent based on large language model according to claim 1, characterized in that: In step S3, if intelligent dubbing is required, speech synthesis is called to generate an audio file of the spoken script.

7. The method for realizing video reconstruction agent based on large language model according to claim 1, characterized in that: In step S5, the packaging template includes a title and subtitles.

Images