Short video automatic editing method based on multi-path annotation recall

Abstract

The invention discloses an automatic short video editing method based on multipath annotation recall, and the method comprises the following steps: video knowledge base construction: constructing a video knowledge base with dubbing and a video knowledge base without dubbing; video clip generation recall, copywriting input, cutting into slices according to punctuations, circularly obtaining the slices until the number of Chinese characters reaches 40, taking a result as a copywriting clip, obtaining a knowledge block of Top 10 matching degree of each copywriting clip from a video knowledge base as a rough arrangement result, and performing LLM fine arrangement and formatting on the result according to time and content; meanwhile, the copywriting fragments are sent to TTS service to be converted into copywriting voice fragments, and the copywriting voice fragments are filtered; editing the video; according to the method, efficient management and accurate matching of the video materials are realized, the short video production efficiency and quality are remarkably improved, and the method has a wide application prospect.

Description

Technical Field

[0001] The invention relates to a short video automatic editing method based on multi-channel annotation recall, and belongs to the technical field of short video automatic editing applications. Background Art

[0002] Short video is a way of disseminating Internet content with uneven quality. It will take time to cure bad short videos. Short videos are generally less than 5 minutes long. With the popularization of mobile terminals and the acceleration of the Internet, they have gradually gained popularity on major platforms and a group of content producers have gradually emerged in the video industry. By 2017, competition in the short video industry has entered a white-hot stage, and content producers also tend to prefer PGC professional operations. Short video has gradually become an important carrier and dissemination method for netizens to obtain information, services, interactive communication, cultural entertainment.

[0003] Among existing video editing tools, applications such as Jianying allow users to mix and edit video clips. However, these tools mainly support static images and text as materials for processing, and do not support automated processing of dynamic video content. Users need to manually annotate materials, and cannot automatically annotate and process them through advanced large models. The lack of intelligent speech recognition and text-to-speech conversion functions results in low work efficiency. Summary of the invention

[0004] In this embodiment, a short video automatic editing method based on multi-channel annotation recall is provided to solve the problem in the prior art that in existing video editing tools, applications such as Jianying allow users to mix and edit video clips. However, these tools mainly support static images and texts as materials for processing, and do not support automatic processing of dynamic video content. Users need to manually annotate materials, and automatic annotation and processing cannot be performed through advanced large models. There is a lack of intelligent speech recognition and text-to-speech conversion functions, resulting in low work efficiency.

[0005] The present invention achieves the above-mentioned purpose through the following technical scheme, a short video automatic editing method based on multi-channel annotation recall, the short video automatic editing method comprises the following steps:

[0006] S1. Video knowledge base construction, video knowledge base construction includes video knowledge base construction with dubbing and video knowledge base construction without dubbing;

[0007] S2. Video segment generation recall: Input the text, divide it into slices according to punctuation, and loop to obtain slices until the number of Chinese characters reaches 40 (>10 seconds). The result is used as the text segment. The top 10 matching knowledge blocks of each text segment are obtained from the video knowledge base as the rough sorting result. The result is sorted and formatted by LLM according to time and content. At the same time, the text segment is sent to the TTS service and converted into a text voice segment.

[0008] S3. Filter, loop the blocks in the recall result of the previous step, accumulate the block length, ensure that the accumulated length is less than the current copy voice segment length, and output the matching block result to the video editing program;

[0009] S4. Video editing, loop block, obtain the original video according to the block file ID metadata, extract video clips from the original video according to the timestamp information, and splice them together to form a clip; calculate the length difference between the length of the copy voice clip and the length of the clip. If the length difference is <1 second, insert a blank at the end of the copy voice clip, otherwise the video clip will be slowed down, transitions will be added, and empty shots will be processed.

[0010] Preferably, the construction of the video knowledge base with dubbing includes subtitle generation and knowledge base construction.

[0011] Preferably, the subtitles are generated by inputting a video, calling an ASR service to generate video subtitles, and at the same time generating video metadata information, obtaining vehicle model information, and obtaining the duration of the subtitle segment.

[0012] Preferably, the knowledge base is created by inputting video subtitles, slicing and overlapping according to JSON objects, looping to obtain slices until the number of Chinese characters reaches 40 (>10 seconds), outputting the resulting subtitle fragments, and obtaining video metadata at the same time, defining them as custom fields (Custom Filed), and saving them together with the subtitle fragments to the video knowledge base, while performing text embedding and vectorized storage (milvus database) and inverted indexing (es); finally, the video knowledge base is output.

[0013] Preferably, the construction of the video knowledge base without dubbing includes scene metadata generation and scene knowledge base creation.

[0014] Preferably, the scene metadata is generated as an input video, and the scene is segmented through the TransNet model. At the same time, the large model is fine-tuned using the existing company's accumulated car picture vector form and labels so that it can accurately identify car parts, and the fine-tuned large model is used to generate video metadata information, including vehicle model information, scene timestamp JSON, and scene duration.

[0015] Preferably, the scene knowledge base is created by inputting scene source data, extracting frames according to JSON objects, looping to obtain frame screenshots until the duration is less than 5 seconds, using the Llama model to detect frame content, and saving the results as scene descriptions to the video knowledge base; at the same time, video metadata is obtained and saved as a custom field to the video knowledge base, and the video knowledge base is output.

[0016] Preferably, the LLama model is a large-scale language model developed by Meta (formerly Facebook) to improve the performance of natural language processing (NLP) tasks. The LLama model is a basic language model collection released by Meta AI in 2023, with parameters ranging from 7B (7 billion) to 65B (65 billion); the model is based on the Transformer architecture and is trained on large-scale data to perform well in multiple language tasks.

[0017] Preferably, the TransNet model is a video processing model based on deep learning, which focuses on identifying the boundaries of shot switching from a video stream. Shot switching is a common technical means in video production, which is used to connect different video clips to form a coherent storyline. TransNet can accurately detect these shot switching points by learning and identifying feature changes between video frames.

[0018] The beneficial effects of the present invention are as follows: the short video automatic editing method based on multi-channel annotation recall in the present invention, through intelligent identification of scenes and content in the video, automatically splits the video and performs semantic annotation, thereby improving the management efficiency and accuracy of video materials, achieving high-precision video clip matching and recall, and ensuring the quality and relevance of the generated video. A large language model, a fine-tuned image detection model and a predefined label group for the automotive industry are used to accurately annotate the video content through a multiple annotation system, thereby ensuring the accuracy and consistency of the video material, automatically aligning audio and video, and utilizing automatic speed regulation, supplementation of empty shots, transitions and the like to generate a smoother and more professional finished video, thereby achieving efficient management and precise matching of video materials, significantly improving the efficiency and quality of short video production, and having broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flow chart is constructed for the video knowledge base with dubbing of the present invention.

[0020] Figure 2 A flow chart for constructing a video knowledge base without dubbing according to the present invention.

[0021] Figure 3 The video recall and generation flow chart of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Embodiment 1:

[0024] A short video automatic editing method based on multi-channel annotation recall, the short video automatic editing method comprising the following steps:

[0025] S1. Video knowledge base construction, video knowledge base construction includes video knowledge base construction with dubbing and video knowledge base construction without dubbing;

[0026] S2. Video segment generation recall: Input the text, divide it into slices according to punctuation, and loop to obtain slices until the number of Chinese characters reaches 40 (>10 seconds). The result is used as the text segment. The top 10 matching knowledge blocks of each text segment are obtained from the video knowledge base as the rough sorting result. The result is sorted and formatted by LLM according to time and content. At the same time, the text segment is sent to the TTS service and converted into a text voice segment.

[0027] S3. Filter, loop the blocks in the recall result of the previous step, accumulate the block length, ensure that the accumulated length is less than the current copy voice segment length, and output the matching block result to the video editing program;

[0028] S4. Video editing, loop block, obtain the original video according to the block file ID metadata, extract video clips from the original video according to the timestamp information, and splice them together to form a clip; calculate the length difference between the length of the copy voice clip and the length of the clip. If the length difference is <1 second, insert a blank at the end of the copy voice clip, otherwise the video clip will be slowed down, transitions will be added, and empty shots will be processed.

[0029] Preferably, the construction of the video knowledge base with dubbing includes subtitle generation and knowledge base construction.

[0030] Preferably, the subtitles are generated by inputting a video, calling an ASR service to generate video subtitles, and at the same time generating video metadata information, obtaining vehicle model information, and obtaining the duration of the subtitle segment.

[0031] Preferably, the knowledge base is created by inputting video subtitles, slicing and overlapping according to JSON objects, looping to obtain slices until the number of Chinese characters reaches 40 (>10 seconds), outputting the resulting subtitle fragments, and obtaining video metadata at the same time, defining them as custom fields (Custom Filed), and saving them together with the subtitle fragments to the video knowledge base, while performing text embedding and vectorized storage (milvus database) and inverted indexing (es); finally, the video knowledge base is output.

[0032] Preferably, the construction of the video knowledge base without dubbing includes scene metadata generation and scene knowledge base creation.

[0033] Preferably, the scene metadata is generated as an input video, and the scene is segmented through the TransNet model. At the same time, the large model is fine-tuned using the existing company's accumulated car picture vector form and labels so that it can accurately identify car parts, and the fine-tuned large model is used to generate video metadata information, including vehicle model information, scene timestamp JSON, and scene duration.

[0034] Preferably, the scene knowledge base is created by inputting scene source data, extracting frames according to JSON objects, looping to obtain frame screenshots until the duration is less than 5 seconds, using the Llama model to detect frame content, and saving the results as scene descriptions to the video knowledge base; at the same time, video metadata is obtained and saved as a custom field to the video knowledge base, and the video knowledge base is output.

[0035] Preferably, the LLama model is a large-scale language model developed by Meta (formerly Facebook) to improve the performance of natural language processing (NLP) tasks. The LLama model is a basic language model collection released by Meta AI in 2023, with parameters ranging from 7B (7 billion) to 65B (65 billion); the model is based on the Transformer architecture and is trained on large-scale data to perform well in multiple language tasks.

[0036] Preferably, the TransNet model is a video processing model based on deep learning, which focuses on identifying the boundaries of shot switching from a video stream. Shot switching is a common technical means in video production, which is used to connect different video clips to form a coherent storyline. TransNet can accurately detect these shot switching points by learning and identifying feature changes between video frames.

[0037] The above-mentioned automatic short video editing method based on multi-channel annotation recall can intelligently identify the scenes and content in the video, automatically split the video and perform semantic annotation, thereby improving the management efficiency and accuracy of video materials, achieving high-precision video clip matching recall, and ensuring the quality and relevance of the generated video. It applies a large language model, a fine-tuned image detection model, and a predefined label group for the automotive industry, and accurately annotates the video content through a multiple annotation system, thereby ensuring the accuracy and consistency of the video material.

[0038] Embodiment 2:

[0039] A short video automatic editing method based on multi-channel annotation recall, the short video automatic editing method comprising the following steps:

[0040] S1. Video knowledge base construction, video knowledge base construction includes video knowledge base construction with dubbing and video knowledge base construction without dubbing;

[0041] S2. Video segment generation recall: Input the text, divide it into slices according to punctuation, and loop to obtain slices until the number of Chinese characters reaches 40 (>10 seconds). The result is used as the text segment. The top 10 matching knowledge blocks of each text segment are obtained from the video knowledge base as the rough sorting result. The result is sorted and formatted by LLM according to time and content. At the same time, the text segment is sent to the TTS service and converted into a text voice segment.

[0042] S3. Filter, loop the blocks in the recall result of the previous step, accumulate the block length, ensure that the accumulated length is less than the current copy voice segment length, and output the matching block result to the video editing program;

[0043] S4. Video editing, loop block, obtain the original video according to the block file ID metadata, extract video clips from the original video according to the timestamp information, and splice them together to form a clip; calculate the length difference between the length of the copy voice clip and the length of the clip. If the length difference is <1 second, insert a blank at the end of the copy voice clip, otherwise the video clip will be slowed down, transitions will be added, and empty shots will be processed.

[0044] Preferably, the construction of the video knowledge base with dubbing includes subtitle generation and knowledge base construction.

[0045] Preferably, the subtitles are generated by inputting a video, calling an ASR service to generate video subtitles, and at the same time generating video metadata information, obtaining vehicle model information, and obtaining the duration of the subtitle segment.

[0046] Preferably, the knowledge base is created by inputting video subtitles, slicing and overlapping according to JSON objects, looping to obtain slices until the number of Chinese characters reaches 40 (>10 seconds), outputting the resulting subtitle fragments, and obtaining video metadata at the same time, defining them as custom fields (Custom Filed), and saving them together with the subtitle fragments to the video knowledge base, while performing text embedding and vectorized storage (milvus database) and inverted indexing (es); finally, the video knowledge base is output.

[0047] Preferably, the construction of the video knowledge base without dubbing includes scene metadata generation and scene knowledge base creation.

[0048] Preferably, the scene metadata is generated as an input video, and the scene is segmented through the TransNet model. At the same time, the large model is fine-tuned using the existing company's accumulated car picture vector form and labels so that it can accurately identify car parts, and the fine-tuned large model is used to generate video metadata information, including vehicle model information, scene timestamp JSON, and scene duration.

[0049] Preferably, the scene knowledge base is created by inputting scene source data, extracting frames according to JSON objects, looping to obtain frame screenshots until the duration is less than 5 seconds, using the Llama model to detect frame content, and saving the results as scene descriptions to the video knowledge base; at the same time, video metadata is obtained and saved as a custom field to the video knowledge base, and the video knowledge base is output.

[0050] Preferably, the LLama model is a large-scale language model developed by Meta (formerly Facebook) to improve the performance of natural language processing (NLP) tasks. The LLama model is a basic language model collection released by Meta AI in 2023, with parameters ranging from 7B (7 billion) to 65B (65 billion); the model is based on the Transformer architecture and is trained on large-scale data to perform well in multiple language tasks.

[0051] Preferably, the TransNet model is a video processing model based on deep learning, which focuses on identifying the boundaries of shot switching from a video stream. Shot switching is a common technical means in video production, which is used to connect different video clips to form a coherent storyline. TransNet can accurately detect these shot switching points by learning and identifying feature changes between video frames.

[0052] The above-mentioned automatic short video editing method based on multi-channel annotation recall can intelligently identify the scenes and content in the video, automatically split the video and perform semantic annotation, thereby improving the management efficiency and accuracy of video materials, achieving high-precision video clip matching recall, and ensuring the quality and relevance of the generated video. It uses a large language model and a fine-tuned image detection model and a predefined label group for the automotive industry to accurately annotate the video content through a multiple annotation system, ensuring the accuracy and consistency of the video material, automatically aligning audio and video, and using automatic speed adjustment, supplementary empty shots, transitions and other means to generate a smoother and more professional finished video, achieving efficient management and precise matching of video materials, significantly improving the efficiency and quality of short video production, and has broad application prospects.

[0053] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0054] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A short video automatic editing method based on multi-channel annotation recall, characterized by: The short video automatic editing method comprises the following steps: S1. Video knowledge base construction, video knowledge base construction includes video knowledge base construction with dubbing and video knowledge base construction without dubbing; S2. Video segment generation recall: Input the text, divide it into slices according to punctuation, and loop to obtain slices until the number of Chinese characters reaches 40 (>10 seconds). The result is used as the text segment. The top 10 matching knowledge blocks of each text segment are obtained from the video knowledge base as the rough sorting result. The result is sorted and formatted by LLM according to time and content. At the same time, the text segment is sent to the TTS service and converted into a text voice segment. S3. Filter, loop the blocks in the recall result of the previous step, accumulate the block length, ensure that the accumulated length is less than the current copy voice segment length, and output the matching block result to the video editing program; S4. Video editing, loop block, obtain the original video according to the block file ID metadata, extract video clips from the original video according to the timestamp information, and splice them together to form a clip; calculate the length difference between the length of the copy voice clip and the length of the clip. If the length difference is <1 second, insert a blank at the end of the copy voice clip, otherwise the video clip will be slowed down, transitions will be added, and empty shots will be processed.

2. The method for automatically editing short videos based on multi-channel annotation recall according to claim 1, characterized in that: The construction of the video knowledge base with dubbing includes subtitle generation and knowledge base construction.

3. The method for automatically editing short videos based on multi-channel annotation recall according to claim 2, characterized in that: The subtitle generation is to input the video, call the ASR service to generate video subtitles, and at the same time generate video metadata information, obtain vehicle model information, and obtain the length of the subtitle segment.

4. The method for automatically editing short videos based on multi-channel annotation recall according to claim 2, characterized in that: The knowledge base is created by inputting video subtitles, slicing and overlapping according to JSON objects, looping to obtain slices until the number of Chinese characters reaches 40 (>10 seconds), outputting the resulting subtitle fragments, and obtaining video metadata at the same time, defining them as custom fields (Custom Filed), and saving them together with the subtitle fragments to the video knowledge base, while performing text embedding and vectorized storage (milvus database) and inverted indexing (es); finally, the video knowledge base is output.

5. The method for automatically editing short videos based on multi-channel annotation recall according to claim 1, characterized in that: The construction of the video knowledge base without dubbing includes scene metadata generation and scene knowledge base creation.

6. The method for automatically editing short videos based on multi-channel annotation recall according to claim 5, characterized in that: The scene metadata is generated as an input video, and the scene is segmented through the TransNet model. At the same time, the large model is fine-tuned using the existing company's accumulated car image vector form and labels so that it can accurately identify car parts. The fine-tuned large model is used to generate video metadata information, including vehicle model information, scene timestamp JSON, and scene duration.

7. The method for automatically editing short videos based on multi-channel annotation recall according to claim 5, characterized in that: The scene knowledge base is created by inputting scene source data, extracting frames according to JSON objects, looping to obtain frame screenshots until the duration is less than 5 seconds, using the Llama model to detect frame content, and saving the results as scene descriptions to the video knowledge base; at the same time, video metadata is obtained and saved as a custom field to the video knowledge base, and the video knowledge base is output.

8. The method for automatically editing short videos based on multi-channel annotation recall according to claim 5, characterized in that: The LLama model is a large-scale language model developed by Meta (formerly Facebook) to improve the performance of natural language processing (NLP) tasks. The LLama model is a collection of basic language models released by Meta AI in 2023, with parameters ranging from 7B (7 billion) to 65B (65 billion); the model is based on the Transformer architecture and is trained with large-scale data to perform well in multiple language tasks.

9. The method for automatically editing short videos based on multi-channel annotation recall according to claim 5, characterized in that: The TransNet model is a deep learning-based video processing model that focuses on identifying shot switching boundaries from video streams. Shot switching is a common technical means in video production, which is used to connect different video clips to form a coherent storyline. TransNet can accurately detect these shot switching points by learning and identifying feature changes between video frames.

Images