Multi-modal large language model training data construction method and system
By correlating action time boundary marking and audio comments on table tennis videos, a training data set is constructed, and a multimodal large language model is fine-tuned, the problem of rough subtitles generation of table tennis videos is solved, and high-quality video commentary is achieved.
Patent Information
- Application Number
- CN202411192057.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2025-01-10
AI Technical Summary
The existing multimodal large language model is difficult to generate fine-grained comments in the generation of table tennis video subtitles, and the lack of appropriate training data makes the generated subtitles rough and unable to meet the needs of actual event commentary.
By obtaining the target video and marking it with action time boundaries, splitting the video into video clips, associating it with audio comments, building a training dataset, fine-tuning it with multimodal large language model and evaluating its performance, and generating high-quality video explanation .
The construction of refined narrative data sets for table tennis videos has been realized, the quality and granularity of video commentary have been improved, and the needs of actual event commentary have been met.
Smart Images
Figure CN119293468A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of sports video commentary, and in particular to a method and system for constructing multimodal large language model training data. Background Art
[0002] Ball sports such as table tennis are popular and demanding sports that require a high level of skill and involve fast-paced play and complex interactions between players, rackets, and balls. To enhance the intensity and entertainment value of the games, sports broadcasting platforms hire professional sports commentators, usually former athletes or professional commentators, to provide rich commentary. With the development of artificial intelligence, intelligent commentary on video platforms has also been widely used. However, the process of intelligent commentary using multimodal large models faces major challenges and requires accurate perception, domain-specific knowledge, and careful selection of vocabulary for the audience. Therefore, large language models can naturally adapt to automate this complex process. Despite the popularity and difficulty of table tennis, there is a relative lack of research on table tennis video subtitle generation. Existing sports video commentary research mainly focuses on football and basketball, while domain-specific tasks are unimodal, including table tennis ball tracking. On the other hand, general dense video subtitle generation models for locating and describing temporal events in video sequences often produce coarse-grained comments that lack the substance to build meaningful narratives. Even with the latest capabilities of large language models, fine-grained video description is rarely performed due to the lack of suitable ping-pong data to guide large multimodal models to perform descriptions. Summary of the invention
[0003] In view of the above problems existing in the prior art, the present invention proposes a method and system for constructing multimodal large language model training data, which mainly solves the problem that most existing model-based video commentaries can only generate coarse-grained comments and are difficult to meet the needs of actual event commentary.
[0004] In order to achieve the above purpose and other purposes, the technical solution adopted by the present invention is as follows.
[0005] The present application provides a method for constructing multimodal large language model training data, comprising: obtaining multiple target videos, wherein the target videos have an audio channel; marking the time boundary of each target action in the target video to split the target video into multiple video segments based on the time boundary; associating each of the video segments with the corresponding audio comments in the audio channel to obtain video and audio data pairs to form a training data set, wherein the audio comments are comments related to the game.
[0006] In one embodiment of the present application, the step of marking the time boundary of each target action in the target video includes: identifying the target action in the target video to obtain a target frame corresponding to the action; providing a video marking tool, switching the target video to the target frame for display, and determining the time boundary of the target action through the video marking tool, wherein the video marking tool has the functions of slowing down playback, pausing and rewinding.
[0007] In one embodiment of the present application, after the target action in the target video is identified and the target frame corresponding to the action is obtained, it also includes: constructing a target frame sequence according to the position of the target frame in the target video, so as to switch and display the target frame based on the order of the target frame sequence.
[0008] In one embodiment of the present application, the step of associating each of the video clips with the corresponding audio comments in the audio channel includes: converting the audio comments in the audio channel into subtitle information; filtering out the segments in the subtitle information that are not related to the video content to obtain target subtitles; aligning the target subtitles with the corresponding video clips, and establishing a correspondence between the audio comments corresponding to the target subtitles and the video clips.
[0009] In one embodiment of the present application, after converting the audio comments in the audio channel into subtitle information, it also includes: extracting sensitive words in the subtitle information; replacing the sensitive words with preset characters and filling them into corresponding positions of the subtitle information.
[0010] In an embodiment of the present application, after converting the audio comments in the audio channel into subtitle information, the method further includes: correcting errors in the subtitle information by combining regular expressions with a speech matching algorithm.
[0011] The present application also provides a multimodal large language model training data construction system, including: a video acquisition module, used to acquire multiple target videos, wherein the target videos have an audio channel; a labeling module, used to label the time boundary of each target action in the target video, so as to split the target video into multiple video segments based on the time boundary; a data set construction module, used to associate each of the video segments with the corresponding audio comments in the audio channel, and obtain video and audio data pairs to form a training data set, wherein the audio comments are comments related to the game.
[0012] The present application also provides a method for fine-tuning a large multimodal language model, comprising: obtaining a training data set to fine-tune the large multimodal language model based on the training data set to obtain a target language model.
[0013] The present application also provides an evaluation method for a multimodal large language model, comprising: providing a multimodal large language model, wherein the multimodal large language model is fine-tuned according to the aforementioned fine-tuning method; generating a first comment corresponding to a video clip based on the multimodal large language model; scoring the first comment based on a multidimensional evaluation index to obtain a scoring result, wherein the multidimensional evaluation index includes a relevance index, a consistency index, a fluency index, a logic index, and a comment distribution divergence.
[0014] As described above, the present application proposes a method and system for constructing multimodal large language model training data, which has the following beneficial effects.
[0015] This application combines actions and comments in the video into data pairs, and then constructs a training data set, which can generate a narrative data set of high strength for action-intensive videos, so that the large language model obtained based on the data set can complete more refined video explanation tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 This is a flow chart of a method for constructing multimodal large prediction model training data in one embodiment of the present application.
[0017] Figure 2 This is a module diagram of a multimodal large prediction model training data construction system in one embodiment of the present application. DETAILED DESCRIPTION
[0018] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0019] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0020] The inventors found that although the existing technology can complete tasks such as modeling the trajectory of the ball and identifying the actions of the players, there is still a big gap between this and the completed commentary of table tennis matches. Recently, large language models (LLMs) have demonstrated surprising performance in language generation tasks. Inspired by this success, researchers have explored multimodal large language models (MLLMs) to enhance the ability of large language models (LLMs) to process information from different modalities (such as images, videos, and audio). MM-VID, VisualChat-GPT, HuggingGPT, and ViperGPT are methods that use LLMs such as ChatGPT, GPT4 (vision), and GPT4 and connect them with different expert models to perform various visual tasks. However, coarse-grained comments are often generated, lacking the substantive content to build meaningful narratives. Specifically, ordinary multimodal models tend to provide descriptions from a macro perspective when generating commentary subtitles (for example, two people playing table tennis in a video). In addition, for subtitle generation in sports commentary scenarios, existing evaluation methods cannot objectively and accurately measure the performance of multimodal large models.
[0021] Based on the above problems existing in the prior art, the present application proposes a method and system for constructing multimodal large prediction model training data. The technical solution of the present application is elaborated in detail below in combination with specific embodiments.
[0022] See also Figure 1 , Figure 1 The flowchart of the method for constructing training data of a multimodal large prediction model in one embodiment of the present application is shown in FIG. The training data construction method includes the following steps:
[0023] Step S100: Acquire multiple target videos, wherein the target videos have audio channels.
[0024] In one embodiment, complete, original broadcast match videos of top tournaments and competitions can be collected, including the Olympic Games, World Table Tennis Championships, World Table Tennis Series, Table Tennis World Cup, Table Tennis Diamond League, Asian Table Tennis Championships, Asian Summer Games, and National Games. All videos are high quality and have audio channels. The collected videos serve as target videos for further annotation.
[0025] Step S110, marking the time boundary of each target action in the target video, so as to split the target video into multiple video segments based on the time boundary;
[0026] In one embodiment, the target video is an action-intensive video, such as a table tennis match video. The player actions (i.e., hitting the ball) in the video are very short and intensive, which makes it challenging to locate the actions and, more importantly, to link the voice-over narration to specific interactions. Therefore, the embodiment of the present application divides all comments into four categories when annotating: introduction, match, playback, and summary narration, and annotates the time boundary of each target action in the target video, thereby corresponding the target action to the audio commentary.
[0027] In one embodiment, the step of marking the time boundary of each target action in the target video includes: identifying the target action in the target video to obtain a target frame corresponding to the action; providing a video marking tool, switching the target video to the target frame for display, and determining the time boundary of the target action through the video marking tool, wherein the video marking tool has the functions of slowing down playback, pausing and rewinding.
[0028] Specifically, taking table tennis as an example, unlike sports such as football and basketball, these sports can use recognition models to propose anchor points, and manual annotators can find the specific anchor points described in the game scene and mark the start and end time. Since the ball and players in the table tennis video move very quickly, the recognition model can be used to perform action recognition first to determine the approximate frame position corresponding to the action, and then the video annotation tool can be used to perform refined target action time boundary calibration. The recognition model automatically identifies the approximate position of the target action for rough positioning, and one or more recognized video frames can be selected as the target frame. When annotating through the video annotation tool, the video can be switched according to the position of the target frame, and then jump to the corresponding position of the target frame, reducing the time of manual annotation query. The video annotation tool can slow down the playback, pause, rewind, etc. The operator can operate according to the needs to determine the time boundary of the target action.
[0029] In one embodiment, after the target action in the target video is identified and the target frame corresponding to the action is obtained, it also includes: constructing a target frame sequence according to the position of the target frame in the target video, so as to switch and display the target frame based on the order of the target frame sequence.
[0030] Specifically, after the target action is identified through the recognition model, the target frame corresponding to the target action is located, and the target frames of each target action are grouped into a target frame sequence according to the order in which the target frames appear in the corresponding target video. When the video annotation tool is used for fine annotation later, the video can be switched according to the order of the target frame sequence, which is convenient for automatically locating the target action in the video, assisting manual annotation, and improving efficiency. The recognition model can adopt an existing image recognition model, and the specific recognition algorithm can be set and adjusted according to the actual application requirements, which is not limited here.
[0031] In one embodiment, after the time boundary marking is completed, the target video may be split into multiple video segments according to the time boundary, and each video segment has a corresponding audio commentary segment.
[0032] Step S120 , associating each of the video clips with the corresponding audio comments in the audio channel to obtain video and audio data pairs to form a training data set, wherein the audio comments are comments related to the game.
[0033] In one embodiment, the step of associating each of the video clips with the corresponding audio comments in the audio channel includes: converting the audio comments in the audio channel into subtitle information; filtering out the segments in the subtitle information that are not related to the video content to obtain target subtitles; aligning the target subtitles with the corresponding video clips, and establishing a correspondence between the audio comments corresponding to the target subtitles and the video clips. Specifically, subtitle extraction can be performed based on the audio commentary paragraphs corresponding to each video clip. The specific audio recognition is a prior art and will not be repeated here. The audio comments are converted into subtitle information, and the comments that are not related to the video game are manually filtered, and only the information directly related to the ongoing game is focused on to obtain the target subtitles. After the screening is completed, the target subtitles are aligned with the audio commentary paragraphs, and the audio comments are trimmed accordingly. Selective annotation is essential for maintaining a high-quality, focused dataset. Since the edited video clips are aligned only with the voice narration, the data obtained can solve the problem of delays in the ground truth narration after the subtitle action occurs in the table tennis game.
[0034] In one embodiment, after converting the audio comments in the audio channel into subtitle information, it also includes: extracting sensitive words in the subtitle information; replacing the sensitive words with preset characters, and filling them into the corresponding positions of the subtitle information. Specifically, sensitive information such as player names can be hidden, and the corresponding sensitive words can be extracted from the letters and replaced with characters. Similarly, the corresponding audio comments can also be replaced with specific musical notes to maintain the consistency of subtitles and audio comments. The replacement of sensitive words only involves scenarios with special needs, and the original data can also be selected according to needs to facilitate the model to accurately identify information such as player names.
[0035] In one embodiment, after converting the audio comments in the audio channel into subtitle information, the method further includes: correcting errors in the subtitle information by combining regular expressions with a voice matching algorithm. Specifically, since professional table tennis game commentary usually involves unique table tennis terms and player names, there are usually some spelling errors or misunderstandings in the automatic speech recognition output. Therefore, the misspellings or misunderstandings in the player names and specific terms can be corrected by combining regular expressions with voice matching. When irrelevant fields such as off-topic comments or background noise are identified, the corresponding elements can be deleted. This ensures that the data set obtained after annotation is technically accurate and linguistically coherent.
[0036] In one embodiment, the resulting dataset may include JSON files for individual matches, detailing match metadata, video attributes, and annotated segments with commentary. Segment types (openings, matches, replays, and summaries) are carefully labeled, and commentary text is enhanced with standardization and multi-language translation to facilitate accurate model training and analysis.
[0037] See also Figure 2 , Figure 2 This is a module diagram of a multimodal large prediction model training data construction system in an embodiment of the present application. The system includes: a video acquisition module 20, used to acquire multiple target videos, wherein the target videos have audio channels; a labeling module 21, used to label the time boundary of each target action in the target video, so as to split the target video into multiple video segments based on the time boundary; a data set construction module 22, used to associate each of the video segments with the corresponding audio comments in the audio channel, and obtain video and audio data pairs to form a training data set, wherein the audio comments are comments related to the game.
[0038] In one embodiment, the present application also provides a method for fine-tuning a multimodal large language model, the method comprising: obtaining a training data set constructed in the aforementioned steps, and fine-tuning the multimodal large language model based on the training data set to obtain a target language model. The multimodal large language model includes at least a video encoder, an audio encoder, and a language model embedding space, and the video clips in the training data set are input into the video encoder. The audio comments are input into the audio encoder, and the outputs of the video encoder and the audio encoder are respectively connected to the language model embedding space for alignment, so that the model can generate meaningful responses (such as commentary subtitles, etc.) based on the target actions presented in the video and the information in the audio comments. Among them, the language model embedding space can adopt a more mature language model such as LLM (Vicuna / LLaMA). The specific fine-tuning process is a prior art and will not be repeated here.
[0039] In one embodiment, the present application also provides an evaluation method for a multimodal large language model, the method comprising: providing a multimodal large language model, the multimodal large language model being fine-tuned by the aforementioned fine-tuning method; generating a first comment corresponding to a video clip based on the multimodal large language model; scoring the first comment based on a multidimensional evaluation index to obtain a scoring result, wherein the multidimensional evaluation index includes a relevance index, a consistency index, a fluency index, a logic index, and a comment distribution divergence. Specifically, in order to more accurately benchmark the model on sports narratives, four indicators are designed: relevance (R), consistency (C), fluency (Fluency), and logic (Logicality, L). In order to quantify the difference between comments generated by the model and comments written by humans, the KL divergence indicator (i.e., comment distribution divergence) based on the distribution of model-generated comments and human-written comments is introduced.
[0040] These metrics are designed to more comprehensively evaluate the performance of sports video caption generation models, including their ability to capture details of sports actions, maintain narrative coherence, language fluency, and logical rationality. The KL divergence metric is particularly helpful in understanding the difference between the model output and the output of human professional commentators, which is crucial for further optimizing the model and improving the quality of caption generation.
[0041] Relevance is measured by evaluating the consistency between the Ping-Pong terms and review content presented in the model-generated reviews and the reference reviews. A higher score indicates a stronger connection and relevance between the terms and the context.
[0042] The accuracy and consistency of Ping-Pong terminology in the reviews generated by the model are assessed and compared with the reference reviews. A higher score on this metric indicates not only that the terminology is used precisely but also that it is applied consistently throughout the report. This is essential for presenting information in a coherent and reliable manner.
[0043] Fluency focuses on the comments generated by the model, evaluating the correctness of grammatical structure and overall language fluency. A higher score in this category indicates that the narrative is well-structured and flows naturally, which helps improve the overall readability and comprehension of the report.
[0044] Logic measures the coherence of content in model-generated and reference reports. It assesses the overall logic of the introduction and the order in which information is presented. A higher logic score indicates that the report is more logically organized and structured.
[0045] The richness of professional vocabulary in reviews does not always indicate better quality. Excessive use of professional terms may confuse listeners. However, there is currently no precise metric to measure the suitability of model-generated reviews for human listeners. Therefore, this application evaluates the quality of model-generated reviews by comparing the distribution of professional terms in model-generated reviews with the distribution of professional terms in human-written reviews. This is achieved by calculating the KL divergence to assess the similarity of their distributions.
[0046]
[0047] Among them, M() represents the frequency distribution of professional terms in generated comments within a certain length range, and H() represents the frequency distribution of professional terms in reference comments within a certain length range. The smaller the value of K(M,H), the closer the consistency of the distribution of professional terms in the comments generated by the model and the reference comments within the specified length range. This consistency indicates higher quality, meaning that the model can accurately generate comments suitable for human audiences.
[0048] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A method for constructing multimodal large language model training data, characterized in that: include: Acquire a plurality of target videos, wherein the target videos have an audio channel; Marking the time boundary of each target action in the target video to split the target video into multiple video segments based on the time boundary; Each of the video clips is associated with a corresponding audio comment in the audio channel to obtain video and audio data pairs to form a training data set, wherein the audio comments are comments related to the game.
2. The method for constructing multimodal large language model training data according to claim 1, characterized in that: The step of marking the time boundary of each target action in the target video includes: Recognizing a target action in the target video to obtain a target frame corresponding to the action; A video annotation tool is provided to switch the target video to the target frame for display, and the time boundary of the target action is determined by the video annotation tool, wherein the video annotation tool has the functions of slowing down playback, pausing and rewinding.
3. The method for constructing multimodal large language model training data according to claim 2, characterized in that: After identifying the target action in the target video and obtaining the target frame corresponding to the action, the method further includes: A target frame sequence is constructed according to the position of the target frame in the target video, so as to switch and display the target frame based on the order of the target frame sequence.
4. The method for constructing multimodal large language model training data according to claim 1, characterized in that: The step of associating each of the video clips with a corresponding audio comment in the audio channel comprises: Converting audio comments in the audio channel into subtitle information; Screening out the segments irrelevant to the video content in the subtitle information to obtain target subtitles; The target subtitle is aligned with the corresponding video segment, and a corresponding relationship between the audio commentary corresponding to the target subtitle and the video segment is established.
5. The method for constructing multimodal large language model training data according to claim 4, characterized in that: After converting the audio comments in the audio channel into subtitle information, the method further includes: Extracting sensitive words from the subtitle information; The sensitive words are replaced by preset characters and filled into corresponding positions of the subtitle information.
6. The method for constructing multimodal large language model training data according to claim 4, characterized in that: After converting the audio comments in the audio channel into subtitle information, the method further includes: correcting errors in the subtitle information by combining regular expressions with a speech matching algorithm.
7. A multimodal large language model training data construction system, characterized in that: include: A video acquisition module, used for acquiring a plurality of target videos, wherein the target videos have an audio channel; A marking module, used for marking the time boundary of each target action in the target video, so as to split the target video into multiple video segments based on the time boundary; The data set construction module is used to associate each of the video clips with the corresponding audio comments in the audio channel to obtain video and audio data pairs to form a training data set, wherein the audio comments are comments related to the game.
8. A method for fine-tuning a multimodal large language model, characterized in that: include: Obtain a training data set as described in any one of claims 1-6, and fine-tune the multimodal large language model based on the training data set to obtain a target language model.
9. A method for evaluating a multimodal large language model, characterized in that: include: Providing a multimodal large language model, wherein the multimodal large language model is fine-tuned according to the fine-tuning method according to claim 8; generating a first comment corresponding to the video clip based on the multimodal large language model; The first comment is scored based on multidimensional evaluation indicators to obtain a scoring result, wherein the multidimensional evaluation indicators include a relevance index, a consistency index, a fluency index, a logic index, and a comment distribution divergence.