An intelligent video editing method based on large language model

Abstract

The present invention discloses an intelligent video editing method based on a large language model. Based on the material of the institutional resource library, the method combines AI artificial intelligence technologies such as a large language model, a cross-modal engine, speech recognition and synthesis, and an audio-visual language model with audio-visual production. The text manuscripts in the production of professional media content are processed through the large language model, and the audio-visual materials are subjected to comprehensive intelligent analysis through the cross-modal engine and the speech recognition engine. Through semantic matching, intelligent shot matching is performed in the cross-modal index library and the synchronous sound index library respectively, so that editors can manually and quickly adjust and modify the intelligent editing results. The present invention can be used for the intelligent production of new media short videos, event broadcast video news, secondary creation of TV programs, film and television drama trailers and highlights, etc., providing various media organizations and professional content producers with a new video production method to meet the production requirements of massive video content under the video push system in the Internet era.

Description

Technical Field

[0001] The present invention relates to the technical field of video editing, and in particular to an intelligent video editing method based on a large language model. Background Art

[0002] With the continuous advancement of technologies such as computer vision and speech recognition, the application of large cross-modal models is becoming increasingly mature. These models can process data from different modalities (such as text, images, and speech), enabling the fusion and interaction of multimodal information, and providing a richer range of possibilities for AI applications. From the scale of large language models, the diversification of application scenarios, advanced technological innovations, to the in-depth development of large cross-modal models, these trends and achievements not only reflect the tremendous progress made in the field of AI, but also indicate that large model technology will demonstrate its unique value and capabilities in even more fields in the future.

[0003] In the process of producing and creating audio-visual content, the information contained in the image is often not limited to visual information, but may also involve data in other modalities, such as text, voice, etc. Therefore, the intelligent production of audio-visual content through cross-modal data fusion technology has become an important research and improvement direction of the present invention. With the in-depth development of media convergence, various media organizations are facing changes in media content dissemination channels and the resulting massive demand for audio-visual content. Traditional media organizations broadcast at fixed times through radio and television and other release forms, while the new Internet media can release content anytime and anywhere, and is no longer restricted by traditional broadcast channels and broadcast times. The audience's viewing channels and viewing methods have also changed significantly. Obtaining news information through the Internet has become the most common way. At the same time, because most viewers watch through fragmented time, the demand for short videos has also increased significantly.

[0004] Against the backdrop of the rapid development of new Internet media, new requirements have been put forward for the output and production efficiency of audio-visual content. However, traditional audio-visual content production methods can no longer meet the needs of current media convergence development. Professional media organizations urgently need a new production process that is fast, efficient and can ensure content quality to meet the content production needs in the converged media environment. Summary of the Invention

[0005] The present invention proposes an intelligent video editing method based on a large language model. The method is based on AI technologies such as large language models and cross-modal analysis to automatically generate professional media content from text manuscripts. In combination with the characteristics of audio-visual language, the method performs a series of processing such as duplicate detection and length optimization on intelligently matched shots, thereby meeting the efficiency and quality requirements of professional media for video production.

[0006] The present invention is implemented as follows: an intelligent video editing method based on a large language model, the method steps comprising:

[0007] Step 1: Perform cross-modal analysis on the material: When the material is stored, AI engines such as cross-modal models and intelligent voice will automatically perform multi-dimensional comprehensive intelligent analysis on the material;

[0008] Step 2: Select the material or material group required for creation from the institution's resource library;

[0009] Step 3: Import the transcript and rewrite and classify it: Import the video transcript and use the large language model to rewrite the video transcript into the required transcript, and classify and mark the synchronous sound and text types.

[0010] Step 4: Automatically use different intelligent matching models to match shots based on different categories of the manuscripts;

[0011] Step 5: Adjust the intelligent shot matching results to generate matching candidate shot groups: save the matching results of each sentence / paragraph as a shot group, sort them by similarity, define the maximum number of shots in a shot group, and select the shot with the highest matching similarity in each shot group as the preferred result for the next step;

[0012] Step 6: Generate a sequence and adjust it based on the audio-visual language model. This includes intelligently merging scenes based on the temporal order of the previous and next shots in the original footage, and using the audio-visual language model to analyze and process the shot matching results.

[0013] Step 7: Generate the accompaniment and subtitles, and add music;

[0014] Step 8: Complete intelligent editing and conduct manual proofreading to meet the review requirements for final release.

[0015] Furthermore, the comprehensive intelligent analysis in step 1 includes:

[0016] 1.1, perform transition frame detection on the video and split the continuous video material into multiple scene segments based on the detection results;

[0017] 1.2, extract key frames for each video scene;

[0018] 1.3, keyframe cross-modal detection and analysis;

[0019] 1.4, cross-modal analysis of key frames, generating vectors and saving them to the index library;

[0020] 1.5. Perform audio synchronous sound analysis, generate synchronous sound index and save it to the index library.

[0021] Further, in step 4, for the text content marked as simultaneous interpretation, the system performs similarity matching on the simultaneous interpretation index based on the semantic understanding of the text manuscript, including that the text in the manuscript is not exactly the same as the text result of speech recognition, to ensure intelligent matching between the written text in the manuscript and the spoken language in the material interview and dialogue; for the body text type, the system performs matching between the text and the video-audio content in terms of vector dimension based on the semantic understanding of the text, forms similarity data according to the matching comparison results, and performs intelligent matching of shots according to the similarity.

[0022] Further, the method for intelligent scene merging according to the time sequence of the front and rear shots in the original material in step 6 is as follows:

[0023] Each shot matching result contains information such as its original material id ClipID, in point IN, out point OUT, etc.

[0024] Assume that the matching shot results of consecutive multiple sentences / paragraphs of simultaneous interpretation text are C0, C1, C2... respectively; their corresponding original material ids are ClipID1, ClipID2, ClipID3... respectively, the corresponding original material in points are IN1, IN2, IN3... respectively, and the corresponding original material out points are OUT1, OUT2, OUT3... respectively;

[0025] First, compare the material information of the two shots C1 and C0 in the first group, and compare whether the corresponding original material IDs of the two shots are the same;

[0026] If ClipID2 is different from ClipID1, the two shot matching results are from different materials, and no scene merging is required, then proceed to compare the next group of materials C2 and C1;

[0027] If ClipID2 is the same as ClipID1, it is also necessary to compare the continuity of the two shots. Compare the material in point IN2 of shot C1 and the material out point OUT1 of shot C0;

[0028] If IN2 - OUT1 < t, where t is a system predefined value, it indicates that the second matching shot result is continuous with the first matching shot result in terms of time, and scene merging is performed;

[0029] If IN2 - OUT1 ≥ t, where t is a system predefined value, it indicates that the second matching shot result is not continuous with the first matching shot result in terms of time, and no scene merging is performed;

[0030] And so on until the last simultaneous interpretation matching shot result.

[0031] Furthermore, the method of using the audio-visual language model to analyze and process the matching shot results in step 6 includes: judging the duplicate shot results of the intelligent matching of each sentence / paragraph of text; processing the shot length of the intelligent matching shot results; analyzing the audio-visual voice such as the field of view and shooting method of the previous and next shots, and matching them with the montage sentence pattern in the audio-visual language model; fine-tuning the length of the video shot according to the audio length of the dubbing or synchronous sound; after the shot matching of each sentence / paragraph of text in the manuscript, arranging and splicing the shot matching results according to the order of the manuscript or the storyboard script.

[0032] Furthermore, in step 7, the voice-over part of the editing result can be generated through the speech recognition and speech synthesis engine, and subtitles can be automatically generated for the text and the speech in the same period. The music provided by the system can be emotionally classified, and the corresponding emotion can be automatically added to the generated intelligent editing result as needed.

[0033] Furthermore, the maximum number of lenses is less than or equal to 10.

[0034] Furthermore, the cross-modal analysis of key frames in step 1.4 is as follows: extracting the first frame of every 10 frames of the video content as a key frame, performing vector analysis on the key frame after the extraction, and performing difference calculation on the vectors of the two key frames before and after;

[0035] If the vector difference ∂ between every two consecutive frames is less than the preset value ∆, it is considered that the scene does not need to be further split, and the vector of each key frame analyzed is stored in the index library for preservation;

[0036] If the difference between two consecutive key frames is greater than or equal to the preset value ∆, the sixth frame in the segment between the two key frames is added as a key frame, and its analysis result vector is indexed.

[0037] The beneficial effects of the present invention are:

[0038] (1) Designed for the characteristics of professional media organizations, which have a large number of their own audiovisual resources and have established their own institutional resource libraries. Intelligent editing based on institutional resource library materials can ensure the authenticity, reliability and legality of the generated content, while avoiding copyright disputes that may arise from citing Internet materials.

[0039] (2) The industry's first intelligent editing mode is adopted to classify professional media manuscripts into different categories such as main text and concurrent content according to the characteristics of traditional video content production. Different AI models are used for intelligent matching according to different categories to improve the accuracy of intelligent editing.

[0040] (3) For the matching of synchronous sound, semantic matching is performed through a large language model to ensure the matching degree between written and spoken language. The present invention has designed a special semantic matching mode for synchronous sound. Unlike the traditional "text-speech" matching mode, the text content is first semantically understood by the large language model, and then the semantic understanding vector is matched with the audio vector of the synchronous sound, ensuring the matching tolerance of text to speech. For professional media organizations, their manuscripts often use more formal written language, while in interviews and daily conversations, oral expressions are inevitable. Through the semantic matching mode, not only the matching problem of written text to spoken speech is solved, but also the coherence of the previous and next sentences can be intelligently judged to ensure that the jump of the picture is avoided as much as possible in the result of text matching.

[0041] (4) Through the intelligent merging algorithm of the synchronous sound matching results, the intelligent scene merging of the synchronous sound intelligent matching lens results can be carried out, which can effectively avoid the problems of shot jump and discontinuity caused by sound matching according to separate text.

[0042] (5) Based on the large language model and cross-modal engine, intelligent shot matching of audio and video is performed. At the same time, the audio-visual language model is integrated to intelligently process the connection between the shots. This not only avoids the use of repeated shots in the same program, but also can perform secondary processing based on the audience's viewing habits, comprehensive information such as the length, angle of view, and scene of the shot, to form the final intelligent editing result.

[0043] (6) While AI is intelligently generating the editing results, it also automatically generates a series of matching candidate shot groups that have a high degree of match with the text content of the sentence / paragraph, so that editors can quickly adjust and modify the intelligent editing results manually.

[0044] (7) The core models of the large language model and cross-modal model used in this intelligent editing system support local private deployment, which can ensure that the original material content will not be leaked during the intelligent production and creation of videos, thereby ensuring data security.

[0045] The present invention can be widely used in the intelligent production of new media short videos, event broadcast video news, secondary creation of TV programs, trailers and highlights of film and television dramas, and other types of programs; through the application of AI intelligent technology, it provides various media organizations and professional content producers with a new way of video production to meet the production requirements of massive video content under the video push system in the Internet era. It saves editors' time in browsing materials, selecting required shots from materials, and time in picking lyrics from interview materials and shooting lyrics for finished films; with the help of AI dubbing, it can also save the link of professional dubbing actors dubbing, greatly improving the production efficiency of event reporting content. For the secondary creation of finished programs, the present invention can perform intelligent analysis on the finished programs, select points of interest in the finished programs that are suitable for the Internet platform, extract and transcribe them, and generate new short video manuscripts or scripts. It also intelligently generates short video versions based on new points of interest to meet the new needs of creation and push for different audience groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the steps of the method of the present invention;

[0047] Figure 2 This is a flow chart of the steps of the cross-modal analysis method for images according to the present invention;

[0048] Figure 3 Schematic diagram of video frame grouping according to the present invention. DETAILED DESCRIPTION

[0049] Explanation of terms:

[0050] Cross-modality: Cross-modality involves extracting information from different data modalities and interactively fusing them. This involves extracting features from data in various forms, such as text, audio, images, and video, and using these features to retrieve, understand, or generate information. Cross-modal interactive fusion aims to achieve more effective data analysis and understanding through joint feature extraction and cross-modal association. For example, by identifying and leveraging the inherent connections between data from different modalities, such as the relationship between image content and corresponding text descriptions, it is possible to handle asymmetric data, where data from one modality may be richer or more detailed than data from another.

[0051] This invention is based on AI technologies such as large language models and cross-modal analysis to automatically generate professional media content from text manuscripts. It also combines the characteristics of audio-visual language to perform a series of processing such as duplicate detection and length optimization on intelligently matched shots, thus meeting the requirements of professional media for video production efficiency and quality.

[0052] An intelligent video editing method based on a large language model, such as Figure 1 、 Figure 2 As shown, the method steps include:

[0053] Step 1: Perform cross-modal analysis on the material: When the material is stored, AI engines such as cross-modal models and intelligent voice will automatically perform multi-dimensional comprehensive intelligent analysis on the material.

[0054] In order to improve the efficiency of cross-modal intelligent analysis of materials and ensure the accuracy of analysis, this embodiment improves the cross-modal analysis method for images. The steps of the comprehensive intelligent analysis are as follows: Figure 2 Shown, including:

[0055] Perform transition frame detection on the video and split the continuous video material into multiple scene segments based on the detection results;

[0056] Extract key frames for each video scene;

[0057] Keyframe cross-modal detection and analysis;

[0058] Analyze key frames across modalities, generate vectors and save them to the index library;

[0059] Perform audio synchronous sound analysis, generate synchronous sound index and save it to the index library;

[0060] Complete material analysis.

[0061] The system transcodes audio and video footage to generate a low-bitrate proxy as the basis for cross-modal analysis, ensuring efficient intelligent cross-modal analysis. It also detects transition frames and, based on the results, splits the continuous video footage into multiple scene segments. Keyframes are extracted for each scene. In addition to the first frame detected for transition frames, keyframes are extracted for all other frames beyond the transition frame to ensure that the decomposition results include no missing critical information. Cross-modal analysis of keyframes generates vectors and saves them to an index library.

[0062] In this invention, a method is designed to effectively detect key frame content and ensure that the number of key frames does not increase as much as possible. First, the first frame of the video content is extracted as a key frame every 10 frames, and the key frame after the extraction is vector analyzed, and the vector difference between the two key frames is calculated; if the vector difference ∂ between each two consecutive frames is less than the preset value ∆, it is considered that the scene does not need to be further split, and the vector of each analyzed key frame is stored in the index library for preservation; if the difference between the two consecutive key frames is greater than or equal to the preset value ∆, an intermediate frame between the two key frames is added, and the 6th frame in the segment is used as the key frame, as shown in the following figure. Figure 3As shown in the video frame grouping diagram, the video key frames are first intelligently identified based on the AI ​​algorithm. The video frames between two intelligently identified key frames are grouped into groups of 10 frames each, and the first frame of each group is regenerated into a key frame for cross-modal vector analysis. The analysis result vector is indexed.

[0063] By using this method, the key content of each frame in a continuously changing scene can be analyzed, avoiding the problem that the key information of the continuous images may be discarded due to only analyzing the key frames detected by the transition frames.

[0064] Step 2: Select the material or material group required for creation from the institutional resource library. When starting intelligent video editing creation, first select the relevant materials required for this creation from the resource library. You can select one or more related materials for intelligent editing of audio and video content. Selecting materials from the institutional resource library can ensure the authenticity and reliability of the material source and the legality of the copyright. This method uses intelligent editing creation based on institutional resource libraries, local or cloud resource libraries, which can effectively avoid copyright disputes caused by grabbing materials from the Internet, as well as the need to verify the authenticity and reliability of the materials.

[0065] Step 3: Importing, Rewriting, and Categorizing Text: Video text can be directly imported and rewritten using a large language model. Generic text may not be suitable for expression in the language of video footage. With the help of a large language model, semantic understanding of the original text allows it to be rewritten into a video storyboard, generating content such as scene descriptions and narration. The technical solution of this invention also allows for categorized annotation of the text content within the storyboard.

[0066] For [Synchronous Sound], the present invention provides two different annotation modes: continuous and single sentence. The continuous mode is suitable for entire interviews or conversations, while the single sentence mode is suitable for precisely selecting entire sentences or phrases from interviews or conversations. For text not marked as [Synchronous Sound], the system defaults to processing it as [Main Text].

[0067] Step 4: Automatically use different intelligent matching models to match shots based on the different categories of the manuscripts:

[0068] For the text sections marked as [Synchronous Sound] and [Text] in the previous step, different shot-to-shot intelligent matching methods are used. For text content marked as [Synchronous Sound], the system will perform a similarity match against the synchronous sound index based on its semantic understanding of the text manuscript. Even if the text manuscript and the text results from speech recognition are not exactly the same, a match can be made based on the results of semantic understanding. This ensures that the written text in the manuscript can be intelligently matched with the spoken language in the interviews and conversations.

[0069] For the shot results of synchronous sound pattern matching, when the method of the present invention finally uses the shot segment, the original picture and sound corresponding to the shot will be used to enhance the sense of presence of the final result.

[0070] For the "Main Text" portion, the system uses a cross-modal index library to match text with audio and video content in the vector dimension based on semantic understanding of the text. Based on the matching results, similarity data is generated and intelligent shot matching is performed based on this similarity. For shots matched with the main text mode, only the corresponding visual portion of the shot will be used in the final use of the shot, and the audio portion will be replaced by the subsequent speech-synthesized audio content.

[0071] Step 5: Adjust the smart shot matching results to generate candidate shot groups. During the previous smart shot matching process, whether in [Synchronous Sound] mode or [Text] mode, save the matching results of each sentence / paragraph as a shot group and sort them by similarity. Define the maximum number of shots in a shot group and select the shot with the highest matching similarity in each group as the preferred result for the next step. It is generally recommended to set the maximum number of shots in a shot group to 10.

[0072] Generally, the present invention will provide the shot with the highest matching similarity in each group of shots (ie, the shot numbered 01) as the preferred result for the next step of processing.

[0073] Step 6: Generate a sequence and adjust it according to the audio-visual language model, including intelligently merging scenes based on the time sequence of the previous and next shots in the original material in the shot matching results, and using the audio-visual language model to analyze and process the matching shot results.

[0074] In the present invention, in addition to matching and recommending intelligent shots for each sentence / paragraph, the single sentence matching shot result of the previous step will be further corrected and adjusted according to the content of the entire manuscript or shot-by-shot script.

[0075] When using the [Synchronous Sound] mode for intelligent shot matching, in addition to using semantically-based text-to-speech matching, post-processing of the shot matching results is performed on synchronous sound manuscripts marked as "continuous." Specifically, for multiple sentences or paragraphs marked as "continuous," scenes are intelligently merged based on the chronological order of the preceding and following shots in the original footage as shown in the shot matching results.

[0076] The specific method is:

[0077] Each shot matching result contains information such as the original material ID ClipID, in point IN, and out point OUT;

[0078] Assume that the matching shot results of consecutive multi-sentence / paragraphs of simultaneous interpretation texts are C0, C1, C2... respectively; the corresponding original material IDs are ClipID1, ClipID2, ClipID3... respectively, the corresponding in-points of the original materials are IN1, IN2, IN3... respectively, and the corresponding out-points of the original materials are OUT1, OUT2, OUT3... respectively;

[0079] First, compare the material information of the two shots C1 and C0 in the first group;

[0080] Compare whether the corresponding original material IDs of the two shots are the same;

[0081] If ClipID2 is different from ClipID1, the matching results of the two shots are from different materials, and there is no need to perform scene merging. Proceed to compare the next group of materials C2 and C1;

[0082] If ClipID2 is the same as ClipID1, then it is also necessary to compare the continuity of the two shots. Compare the in-point IN2 of the material of shot C1 with the out-point OUT1 of the material of shot C0;

[0083] If IN2 - OUT1 < t, where t is a system-predefined value, it indicates that the second matching shot result is continuous in time with the first matching shot result, and scene merging is performed;

[0084] If IN2 - OUTI ≥ t, where t is a system-predefined value, it indicates that the second matching shot result is not continuous in time with the first matching shot result, and scene merging is not performed;

[0085] And so on until the last matching shot result of the simultaneous interpretation.

[0086] When the [Simultaneous Interpretation] mark in the manuscript ends, this comparison automatically ends; no scene merging analysis is performed between different [Simultaneous Interpretation] paragraphs.

[0087] When using the [Main Text] mode for intelligent shot matching, the intelligent analysis system will calculate and recommend the similarity of independent shot matching according to each sentence / paragraph of text; however, for the editing of audio-visual content, it does not process individual sentences, segments or shots separately, but needs to consider the whole according to the context.

[0088] When generating a video clip sequence, the present invention innovatively introduces an audio-visual language model to analyze and process the matching shot results. First, the intelligent matching shot results of each sentence / paragraph are judged to be duplicated; starting from the second sentence / paragraph, the first shot segment of the current matching candidate shot group is compared with the previous matching result to determine the material duplication; if there is a duplication, the next candidate shot of the matching candidate shot group is used until there is no duplication with the previous shot. If all the matching candidate shots corresponding to the sentence / paragraph are duplicated with the previous shot, the matching shot of the sentence / paragraph is left blank, and is subsequently replaced and filled in manually with other shots.

[0089] Secondly, the shot length of the intelligently matched shot results is processed. For shots that are too short, the length of the shot is extended forward or backward until the shot length meets the preset minimum shot length.

[0090] Next, the audiovisual and speech characteristics of the preceding and following shots, such as the camera angle and camera method, are analyzed to match them with the montage patterns in the audiovisual language model. If the analysis results of the preceding and following shots match any montage pattern, they are retained; otherwise, the current shot is replaced from the set of matching candidate shots until it matches the audiovisual language pattern.

[0091] Finally, fine-tune the length of the video shot based on the length of the dubbing or synchronous sound audio.

[0092] After matching the shots for each sentence / paragraph in the manuscript, the shot matching results are arranged and spliced ​​in the order of the manuscript or storyboard script.

[0093] Step 7: Generate the soundtrack and subtitles, and add the background music. In this invention, during intelligent editing, voice recognition and speech synthesis engines can be used to generate voiceovers for the narration of the editing results, and subtitles for the text and the speech in the same episode can be automatically generated. The music provided by the system can also be emotionally classified, and music with the corresponding emotion can be automatically added to the generated intelligent editing results as needed.

[0094] Step 8: Complete the intelligent editing and perform manual proofreading to meet the final release review requirements. After the above series of intelligent processing, the intelligent editing timeline result of the method of the present invention has been generated; after the system intelligently generates the timeline result, further manual proofreading can be performed to meet the final release review requirements.

[0095] Finally, it should be noted that the above is only used to illustrate the technical solution of the present invention and is not limiting. Although the present invention has been described in detail with reference to the preferred solution, those skilled in the art should understand that the technical solution of the present invention (such as changes in the name or quantity of categories, the sequence of steps, etc.) can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. An intelligent video editing method based on a large language model, characterized in that: The method steps include: Step 1: Perform cross-modal analysis on the material, including: Transcode audio and video materials to generate a low-bitrate proxy as the basis for cross-modal analysis; Perform transition frame detection on the video and split the continuous video material into multiple scene segments based on the detection results; The first frame of the video content is extracted as a key frame every 10 frames, and the key frames after the extraction are vector analyzed, and the vector difference between the two key frames is calculated; if the vector difference ∂ between each two consecutive frames is less than the preset value Δ, it is considered that the scene does not need to be further split, and the vector of each analyzed key frame is stored in the index library for preservation; if the difference between the two consecutive key frames is greater than or equal to the preset value Δ, the middle frame between the two key frames is added, and the 6th frame in the clip is used as the key frame. First, based on the AI ​​algorithm, the video key frames are intelligently identified; and the video frames between the two intelligently identified key frames are grouped into groups of every 10 frames, and the first frame of each group is generated as a key frame again for cross-modal vector analysis. The analysis result vector is indexed to perform cross-modal analysis on the key frames, generate vectors and save them to the index library; Perform audio synchronous sound analysis, generate synchronous sound index and save it to the index library; Step 2: Select the required material or material group from the institutional resource library, local or cloud resource library; Step 3: Import the transcript, rewrite it, and classify it: Import the video transcript and use the large language model to rewrite it into the required transcript. It is then categorized and annotated with the audio and text types. For audio, two different annotation modes are provided: continuous and single sentence. The continuous mode is used for the entire interview or conversation, while the single sentence mode is used to accurately select the entire sentence or phrase spoken in the interview or conversation. Text that is not marked as audio will be processed by the system as text by default. Step 4: Automatically use different intelligent matching models to match shots based on the different categories of the manuscript: For text content marked as synchronous sound, the system performs similarity matching on the synchronous sound index based on the semantic understanding of the manuscript. If the text results of the manuscript and the speech recognition are not exactly the same, matching is performed according to the semantic understanding results. For the shot results of synchronous sound pattern matching, the original picture and sound corresponding to the shot are used. For the main text part, the system matches the text with the audio and video content in the vector dimension based on the semantic understanding of the text. Based on the matching comparison results, similarity data is generated and intelligent shot matching is performed based on the similarity. For the shot results of text pattern matching, when the shot segment is finally used, only the picture portion corresponding to the shot is used, and the sound portion is replaced by the audio content of the subsequent speech synthesis. Step 5: Adjust the intelligent shot matching results to generate matching candidate shot groups: save the matching results of each sentence / paragraph as a shot group, sort them by similarity, define the maximum number of shots in a shot group, and select the shot with the highest matching similarity in each shot group as the preferred result for the next step; Step 6: Based on the time sequence of the previous and next shots in the original material, intelligent merging of scenes is performed and the audio-visual language model is used to analyze and process the matching shot results. The intelligent merging method of scenes is: Each shot matching result contains information such as the original material ID ClipID, in point IN, and out point OUT; Assume that the matching shot results of multiple consecutive sentences / paragraphs of synchronous sound text are C0, C1, C2, ...; their corresponding original material IDs are ClipID1, ClipID2, ClipID3, ..., their corresponding original material in points are IN1, IN2, IN3, ..., and their corresponding original material out points are OUT1, OUT2, OUT3, ...; First, compare the material information of the two shots C1 and C0 in the first group to see if the original material IDs corresponding to the two shots are the same; If ClipID2 is different from ClipID1, the two shot matching results come from different materials. There is no need to merge the scenes, and the next set of materials C2 and C1 will be compared. If ClipID2 is the same as ClipID1, the continuity of the two shots needs to be compared, comparing the material in point IN2 of shot C1 with the material out point OUT1 of shot C0; If IN2 —OUT1 < t, where t is a system-predefined value, it indicates that the second matching shot result is temporally continuous with the first matching shot result, and the scene is merged; If IN2 —OUT1 ≥ t, where t is a system-predefined value, it indicates that the second matching shot result is not temporally continuous with the first matching shot result, and scene merging is not performed; And so on, until the last synchronous sound matching shot result; Step 7: Generate the accompaniment and subtitles, and add music; Step 8: Complete intelligent editing and conduct manual proofreading to meet the review requirements for final release.

2. The intelligent video editing method based on a large language model according to claim 1, characterized in that The method for analyzing and processing the matching shot results using the audio-visual language model in step 6 includes: determining the duplicate shot results of each sentence / paragraph of text; processing the shot length of the intelligently matched shot results; analyzing the audio-visual speech such as the field of view and shooting method of the previous and next shots, and matching them with the montage sentence pattern in the audio-visual language model; fine-tuning the length of the video shot according to the audio length of the dubbing or synchronous sound; and after shot matching for each sentence / paragraph of text in the manuscript, arranging and splicing the shot matching results according to the order of the manuscript or the storyboard script.

3. The intelligent video editing method based on a large language model according to claim 1, characterized in that In step 7, a voice-over is generated for the narration part of the editing result through a speech recognition and speech synthesis engine, and subtitles are automatically generated for the text and the speech in the same period. The music provided by the system can be emotionally classified, and the corresponding emotion can be automatically added to the generated intelligent editing result according to needs.

4. The intelligent video editing method based on a large language model according to claim 1, characterized in that The maximum number of lenses is less than or equal to 10.

Images