Full-automatic multi-mode machine labeling method for video with any length based on deep learning

Abstract

The invention relates to a deep learning-based full-automatic multi-mode machine labeling method for any-length videos, and the method comprises the following steps: obtaining a to-be-labeled-length video, carrying out the analysis and recognition of the lens change, obtaining a lens switching point, segmenting the to-be-labeled-length video based on the lens switching point, and obtaining a plurality of video segments with independent semantics; for each video clip, generating video description information by capturing a space time sequence of the video, translating the video description information into a language matched with video subtitles, and obtaining a classification label of the to-be-labeled long video through subject classification; for the to-be-marked long video, generating video subtitle information comprising a timestamp through audio transcription; and for each video clip, taking the video description information and the video caption information as prompts, reasoning to obtain a multi-modal annotation text, and realizing multi-modal machine annotation. The method has the advantages of being good in labeling effect, suitable for videos of any length, capable of fusing multi-modal information to achieve automatic labeling and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of data annotation, and in particular to a method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning. Background Art

[0002] As an important carrier in today's information age, the richness and quantity of video content are growing exponentially. The rapid popularization of applications such as online education, short video platforms, and social media has led to a diversified and massive development of video information. In these video data, information is not only reflected in static images, but also in multimodal features such as time series changes, sound, and text subtitles. These characteristics make video data more complex and dynamic than image data. However, because of this, video annotation has become a difficult task. Traditional video annotation work mainly relies on manual processing, which is time-consuming and labor-intensive, extremely inefficient, and limited by human subjective bias, making it difficult to maintain consistency. At the same time, faced with the high growth and rapid iteration of video content, manual annotation methods are difficult to meet the needs of practical applications. Therefore, automated annotation methods based on deep learning have gradually become a new solution in this field.

[0003] With the success of the Transformer architecture in natural language processing and computer vision, more and more video annotation works are trying to apply language models to video tasks. For example, multimodal large language models such as VideoChat and Video-LLaMa, which have cross-modal understanding and generation capabilities, can be used to generate video content descriptions or classify and annotate video clips. These models achieve the association between video content and semantic information by describing video frames. However, this end-to-end video annotation method has certain limitations: Limited processing capabilities for long videos: For long videos, the processing effect of such models is limited, mainly because the length of the processing sequence is limited and it is difficult to integrate global information. Insufficient extraction of speech information: Although these models can achieve cross-image text understanding, the direct understanding and processing of the audio content of the video is still weak, and it is difficult to accurately combine the audio content and visual content to generate a complete semantic description.

[0004] In summary, there is currently a lack of an efficient annotation method suitable for videos of a wide range of lengths. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a fully automatic multimodal machine annotation method for videos of arbitrary length based on deep learning, so as to solve or partially solve the problem of unsatisfactory annotation effect caused by the inconsistency of the language of video description and video subtitles.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] One aspect of the present invention provides a method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning, comprising the following steps:

[0008] Obtain a long video to be annotated, obtain shot switching points by analyzing and identifying shot changes, and segment the long video to be annotated based on the shot switching points to obtain multiple semantically independent video segments;

[0009] For each video clip, the video description information is generated by capturing the spatial time sequence of the video and translated into a language matching the video subtitles, and the classification label of the long video to be annotated is obtained by subject classification;

[0010] For the long video to be annotated, generating video subtitle information including a timestamp through audio transcription;

[0011] For each video clip, the video description information and the video subtitle information are used as prompts to infer multimodal annotated text, thereby realizing multimodal machine annotation.

[0012] As a preferred technical solution, the process of obtaining multiple independent video clips based on the video to be annotated includes the following steps:

[0013] Using the AutoShot model to perform shot boundary detection on the long video to be annotated, and obtaining a long video segmentation timestamp;

[0014] The ffmpeg library is used to segment the long video to be labeled according to the long video segmentation timestamps to obtain multiple semantically independent video segments.

[0015] As a preferred technical solution, the process of generating video description information and classification labels includes the following steps:

[0016] For each video clip, obtain video description information of the video clip through the video polymorphic large model;

[0017] Translate the language of the video description into a language that matches the video subtitles;

[0018] The long video to be labeled is subject-classified to obtain classification labels in the form of words.

[0019] As a preferred technical solution, the video polymorphic large model is the VideoChat2 polymorphic large model.

[0020] As a preferred technical solution, the Qwen2 model accelerated by the vllm library is used to achieve topic classification.

[0021] As a preferred technical solution, the process of generating video subtitle information includes the following steps:

[0022] Extracting the audio of the long video to be annotated and converting it into a preset format;

[0023] The audio is transcribed using an automatic speech recognition model to obtain video subtitle information with timestamps.

[0024] As a preferred technical solution, the process of obtaining multimodal annotated text through reasoning includes the following steps:

[0025] For each video clip, the video description information and the video subtitle information are temporally aligned, and a prompt is constructed by merging and serving as an input to the reasoning model to obtain a multimodal annotated text.

[0026] As a preferred technical solution, the inference model is the Vicuna 1.5-7B model.

[0027] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned deep learning-based fully automatic multimodal machine labeling method for videos of arbitrary length.

[0028] Another aspect of the present invention provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned deep learning-based fully automatic multimodal machine annotation method for videos of arbitrary length.

[0029] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0030] (1) Good annotation effect: To address the problem of unsatisfactory annotation effect caused by inconsistent language between video description and video subtitle, the present invention generates video description information and video subtitle information in matching languages ​​to ensure that the language of the video description is consistent with that of the voice content. This enables accurate understanding and annotation of video content in multilingual scenarios, ensures language consistency and content coherence of the video description, and greatly improves the readability and accuracy of the annotation.

[0031] (2) Applicable to videos of any length: The present invention ensures the integrity of video clips in terms of content and semantics by cutting long videos into pieces, reduces the complexity and computational overhead of subsequent processing, and enables videos of any length to be automatically processed through reasonable segmentation, thus ensuring flexibility and efficiency in annotation.

[0032] (3) Automatic annotation by integrating multimodal information: The present invention integrates multimodal information of video description and video subtitles to generate a natural, fluent and semantically complete final annotation text, achieving a high degree of integration and optimization of visual and voice information. It can generate multimodal annotation text with contextual relevance and coherence based on the visual and auditory content in the video, ensuring the comprehensive expression and high-quality presentation of the video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the process of the method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning in an embodiment. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] Example 1

[0036] In response to the problems existing in the aforementioned prior art, this embodiment provides a fully automatic multimodal machine annotation method for videos of arbitrary length based on deep learning, so as to achieve fully automatic multimodal machine annotation that can process videos of arbitrary length and combine video images and video subtitles.

[0037] See also Figure 1 , the method comprises the following steps:

[0038] Step S1: Use the AutoShot model to automatically identify switching points by analyzing shot changes in the video and segment the long video into multiple independent short video clips.

[0039] Specifically, this step includes:

[0040] Step S1-1, using the AutoShot model to detect shot boundaries of the long video to obtain timestamps for segmentation of the long video.

[0041] AutoShot is a shot boundary detection model based on deep learning. Its main goal is to identify the switching points between shots in a long video to achieve accurate cutting of the video content. The model extracts visual features from each frame in the long video, detects scene transitions and perspective changes in the video, and then generates segmentation timestamps. These timestamps mark the start and end positions of each shot in the video, dividing the video into multiple independent short video clips.

[0042] Step S1-2: Cut the video according to the generated timestamps through the ffmpeg library to generate video segments with independent semantics.

[0043] After obtaining the shot boundary timestamps generated by the AutoShot model, the multimedia processing library ffmpeg is used to traverse the timestamp list generated by AutoShot, and the long video is split into short video clips at the cutting point marked by each timestamp. The start and end time of each short video clip is determined by two adjacent timestamps, ensuring that the cut clips fully retain the shot semantics in the original video.

[0044] By adopting this cutting method, not only can the computational overhead of the overall processing of long videos be effectively reduced, but also the consistency of the content within the short video clips can be ensured, which is conducive to the accuracy and efficiency of subsequent video description and audio processing.

[0045] Step S2: Use the VideoChat2 model to capture the video spatial temporal information and generate a video caption for each short video clip.

[0046] Specifically, this step includes:

[0047] Step S2-1: perform video description annotation on the segmented video clips through the video multimodal large model VideoChat2.

[0048] VideoChat2 is a powerful open source multimodal model with strong video content understanding and description generation capabilities. Its model structure consists of three main components: video encoder, QFormer, and LLM (Large Language Model), which realize video understanding and description generation.

[0049] VideoChat2 has a strong ability to generate video descriptions. It can generate accurate and coherent text descriptions based on capturing the temporal information and spatial features in the video. This description covers all kinds of information in the video, including character behavior, action recognition, object detection, scene description, etc. In practical applications, VideoChat2 has demonstrated extremely high versatility and expressiveness, and can provide high-quality description results for different types of video clips.

[0050] In this embodiment, descriptions are generated for each of the cut short video clips to achieve accurate annotation of the video content. Due to its powerful description generation capability, VideoChat2 can provide a detailed and semantically rich description for each segment of the video, greatly improving the completeness and accuracy of the annotation, and laying a solid foundation for subsequent multimodal information fusion and annotation output.

[0051] Step S2-2: Use the open source model m2m100_418M to translate the video description, and translate the original English video description into the same language as the video subtitles to ensure the consistency of the language of the video description and the video subtitles.

[0052] In this embodiment, the video description (caption) generated by the VideoChat2 model is in English by default. However, in this embodiment, the video content may come from different language backgrounds, including Russian, Thai, Vietnamese and other languages. Therefore, in order to ensure that the video description is consistent with the voice content in the video, these English captions need to be translated to keep them consistent with the video subtitles when constructing multilingual video description data.

[0053] Step S2-3: Use the open source model Qwen2 to classify the video according to the video description and obtain the classification label of the long video.

[0054] After generating the video description (caption), the Qwen2 model is used to classify the topic of each video clip. First, the video description data is loaded and the summary information is extracted as input. Then, a prompt containing a description and predefined classification labels is constructed for each video, instructing the Qwen2 model to select a category that best matches the video description from labels such as "movies", "religion", and "technology". After the model infers the prompt, it outputs the classification label in the form of words and saves the result together with the ID of the video clip as a JSON file. In addition, we use the vllm library to accelerate the reasoning process of Qwen2 and improve the annotation efficiency.

[0055] By using Qwen2 to classify video descriptions, the multimodal annotation results have a clear semantic structure, which facilitates video retrieval, analysis and management, and further improves the practicality and completeness of the annotation results.

[0056] This embodiment uses the m2m100_418M translation model as a multilingual translation model, which supports direct translation between more than 100 languages ​​and has excellent translation quality and multilingual conversion capabilities. In this embodiment, the model can accurately translate the English video description generated by VideoChat2 into the corresponding language according to the language of the video subtitles.

[0057] Specifically, for each short video clip, the English video description (caption) generated by VideoChat2 is first translated using the m2m100_418M model to make it consistent with the language of the video subtitles, and then the English caption generated by VideoChat2 is translated to make it consistent with the language of the video subtitles. This translation process not only ensures the language synchronization of the video description and the voice content, but also makes the expression of multimodal information more unified and coordinated, providing semantically complete and coherent annotation results for videos with different language backgrounds.

[0058] Step S3: Use the fast-whisper-large-v3 model to perform audio transcription on the audio file of the video and generate video subtitles for automatic speech recognition (ASR) of the long video.

[0059] Specifically, the speech content (speech) in the video is an extremely important part of multimodal information, and often contains rich and critical semantic information. Voice elements such as character dialogues and scene explanations can not only supplement the details in the visual information, but also provide more accurate context and emotional expression for the video content. Therefore, the effective transcription and understanding of the audio in the video is crucial for generating complete and semantically rich multimodal annotations. To this end, this embodiment uses the Whisper model, which is an open source multilingual automatic speech recognition (ASR) model with excellent speech transcription capabilities, supports automatic recognition and transcription of multiple languages, and provides strong technical support for video transcription in different scenarios. The fast-whisper-large-v3 version is selected as the transcription model. The model not only maintains a very high recognition accuracy rate, but also has been greatly optimized in terms of inference speed, and can quickly respond to the transcription needs of large-scale long videos.

[0060] In this embodiment, an audio file is first extracted from the video and converted into .mp3 format. The extracted .mp3 audio file is automatically transcribed using the fast-whisper-large-v3 model to generate a speech transcription file with timestamps. These timestamps accurately mark the start and end time of the speech content in the video, so that the transcription result not only presents the complete speech text, but also maintains strict time synchronization with the video content. This processing method ensures the accuracy and coherence of the video subtitles.

[0061] Step S4: Integrate the video description and video subtitles through the open source large language model Vicuna 1.5-7B to output the multimodal annotation results of the video.

[0062] This step uses the open source large language model Vicuna 1.5-7B model to integrate the video description and ASR subtitles. In this embodiment, the generated video description and speech transcription files are first loaded, and the two are ensured to be accurately aligned in time. Subsequently, a merged prompt containing visual description and ASR subtitles is constructed for each video clip, and handed over to the Vicuna 1.5-7B model for reasoning to generate complete multimodal annotated text. In this embodiment, we use the vllm library to accelerate the reasoning of the Vicuna1.5-7B model to better handle large-scale data and ensure that the reasoning process is fast and efficient.

[0063] Based on the video image and voice information, the model outputs a natural and coherent description that covers the visual and auditory elements in the video. Finally, the generated multimodal annotation results are saved as JSON files to ensure complete expression of the information. The advantage of this method is that it efficiently integrates multimodal information and generates more natural and rich annotation text.

[0064] In summary, the deep learning-based fully automatic multimodal machine annotation method for videos of arbitrary length provided in this embodiment realizes accurate automatic annotation of long videos through steps such as shot cutting, video description generation, voice transcription and multimodal information integration, effectively overcoming the shortcomings of traditional video annotation methods such as low efficiency, high cost and poor semantic integrity.

[0065] First, the AutoShot model used in this embodiment performs shot cutting on long videos, ensuring the integrity of the video clips in terms of content and semantics, reducing the complexity and computational overhead of subsequent processing, so that videos of any length can be automatically processed through reasonable segmentation, ensuring the flexibility and efficiency of annotation.

[0066] Secondly, the VideoChat2 model is used to capture the temporal and spatial features of video clips and generate accurate visual descriptions for each video. The video classification is inferred based on the visual description of each long video to enrich the video annotation results. Then, the generated English caption is translated into the language corresponding to the video language in combination with the m2m100_418M translation model to ensure that the language of the video description is consistent with the voice content. This processing method can achieve accurate understanding and annotation of video content in multilingual scenarios, ensure the language consistency and content coherence of the video description, and greatly improve the readability and accuracy of the annotation.

[0067] In addition, the fast-whisper-large-v3 model is used for speech transcription in this embodiment to generate ASR subtitles with timestamps, which fully captures the audio content in the video and provides complete and accurate speech text for subsequent multimodal information integration.

[0068] Finally, the Vicuna 1.5-7B model integrates multimodal information of video description and ASR subtitles to generate natural, fluent, and semantically complete final annotation text, achieving a high degree of integration and optimization of visual and voice information. The model can generate contextually relevant and coherent multimodal annotation text based on the visual and auditory content in the video, and ensures the comprehensive expression and high-quality presentation of the video content by linking the information of the two modes of video (picture) description and video (voice) subtitles.

[0069] This embodiment achieves the advantages of flexible processing, rich semantics, multi-language support, and efficient processing in the annotation of video content. Thanks to the excellent performance of deep learning models in the fields of vision, language, and multimodality, the present invention can complete the fully automatic multimodal annotation of videos of any length at low cost and high efficiency, providing strong technical support for high-quality annotation of large-scale video data.

[0070] This method divides a long video into multiple independent short video clips, each of which can generate a video description independently. This not only greatly reduces the difficulty of overall processing of long videos and the consumption of computing resources, but also significantly improves the flexibility and scalability of the annotation process. Secondly, through multimodal fusion, the association and fusion of the semantic information of video images and audio are fully explored, effectively overcoming the limitations of single-modal annotation, and generating video annotation results with richer semantics and more coherent context. Through further translation and semantic integration, the language consistency of video description and subtitle information is ensured, thereby improving the accuracy and spatiotemporal consistency of annotation. Based on open source large language models and multimodal large models, low annotation costs and high efficiency are ensured. Compared with traditional manual annotation methods, the automated annotation process not only achieves significant improvements in efficiency, but can also cope with large-scale video data sets, significantly reducing labor costs.

[0071] Example 2

[0072] Based on Example 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the method for fully automatic multimodal machine labeling of videos of arbitrary length based on deep learning as described in Example 1.

[0073] Example 3

[0074] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning as described in Example 1.

[0075] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A fully automatic multimodal machine annotation method for videos of arbitrary length based on deep learning, characterized in that: The steps include: Obtain a long video to be annotated, obtain shot switching points by analyzing and identifying shot changes, and segment the long video to be annotated based on the shot switching points to obtain multiple semantically independent video segments; For each video clip, the video description information is generated by capturing the spatial time sequence of the video and translated into a language matching the video subtitles, and the classification label of the long video to be annotated is obtained by subject classification; For the long video to be annotated, generating video subtitle information including a timestamp through audio transcription; For each video clip, the video description information and the video subtitle information are used as prompts to infer multimodal annotated text, thereby realizing multimodal machine annotation.

2. According to claim 1, a method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning, characterized in that: The process of obtaining multiple independent video clips based on the video to be annotated includes the following steps: Using the AutoShot model to perform shot boundary detection on the long video to be annotated, and obtaining a long video segmentation timestamp; The ffmpeg library is used to segment the long video to be labeled according to the long video segmentation timestamps to obtain multiple semantically independent video segments.

3. According to claim 1, a method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning, characterized in that: The process of generating video description information and classification labels includes the following steps: For each video clip, obtain video description information of the video clip through the video polymorphic large model; Translate the language of the video description into a language that matches the video subtitles; The long video to be labeled is subject-classified to obtain classification labels in the form of words.

4. According to claim 3, a method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning is characterized in that: The video polymorphic large model is the VideoChat2 polymorphic large model.

5. According to claim 3, a method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning, characterized in that: Topic classification is achieved using the Qwen2 model accelerated by the vllm library.

6. According to claim 3, a method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning, characterized in that: The process of generating video subtitle information includes the following steps: Extracting the audio of the long video to be annotated and converting it into a preset format; The audio is transcribed using an automatic speech recognition model to obtain video subtitle information with timestamps.

7. The method of fully automatic multimodal machine annotation of videos of any length based on deep learning according to claim 1, characterized in that: The process of obtaining multimodal annotated text through reasoning includes the following steps: For each video clip, the video description information and the video subtitle information are temporally aligned, and a prompt is constructed by merging and serving as an input to the reasoning model to obtain a multimodal annotated text.

8. The method of fully automatic multimodal machine annotation of videos of any length based on deep learning according to claim 7, characterized in that: The inference model is the Vicuna 1.5-7B model.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, wherein the one or more programs include instructions for executing the method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the method for fully automatic multimodal machine annotation of videos of arbitrary length based on deep learning as described in any one of claims 1-8.

Images