Fine-grained gymnastics video understanding method and system based on multi-modal large model

Abstract

The invention discloses a fine-grained gymnastics video understanding method and system based on a multi-modal large model, and the method comprises the steps: firstly constructing a professional question and answer data set for the field of gymnastics through combining a decision tree, a question bank template and a mixed labeling process automatically described by the large model; then, a self-adaptive frame sampling module based on inter-frame difference analysis is introduced into a model processing flow to intelligently capture key actions in gymnastics, and a double-flow visual encoder and a subsequent visual token compression module are adopted to efficiently extract and refine visual features after sampling; on the basis, a low-rank adaptation technology is utilized to carry out efficient parameter fine tuning on the model. According to the method, professional knowledge in the gymnastics field can be systematically injected into the large model, the perception ability of the large model to complex dynamics is optimized, and an accurate and efficient technical scheme is provided for fine-grained understanding and analysis of professional gymnastics videos.

Description

Technical Field

[0001] This invention belongs to the field of multimodal large-scale video understanding, specifically involving fine-grained video understanding methods and systems for gymnastics. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large models have achieved remarkable success in tasks combining vision and language understanding. These models can process and understand various media information such as images and videos, demonstrating enormous application potential in many fields. However, directly applying general multimodal large models to vertical domain video understanding (such as football, basketball, gymnastics, etc.) is often limited by domain knowledge. The general multimodal large model structure is not specifically optimized for the characteristics of vertical domains. Therefore, many scholars have conducted research on multimodal large models for different vertical domains.

[0003] Existing methods for constructing large-scale multimodal models in vertical domains include: 1) a method based on multimodal knowledge retrieval enhancement, which first embeds professional domain knowledge into a vector database. During multimodal inference, domain knowledge injection is achieved by searching the vector database for the spatially nearest relevant memories and injecting them into the model's prompts. This retrieval enhancement method does not require updating the large model parameters and is easily adaptable to multimodal models. However, this method is difficult to construct a multimodal domain knowledge database, and current mainstream multimodal models suffer from severe illusions regarding multimodal injection of prompts. Therefore, 2) a method based on efficient parameter fine-tuning of a base multimodal model has been more widely used. Low-rank adaptation techniques can inject a small number of new parameters without modifying the large language model itself, injecting new domain knowledge into the large language model while maintaining its original capabilities. Due to this characteristic, low-rank adaptation fine-tuning techniques have been widely used in many scenarios of post-training of large language models. This invention also belongs to this category of methods.

[0004] The first challenge in constructing a multimodal large-scale model for gymnastics videos (for each vertical domain) using low-rank adaptive fine-tuning methods is building the fine-tuning dataset. This requires manual annotation, which often necessitates extensive expertise and complex processes. Therefore, a semi-automated workflow is needed to assist this process. Many researchers employ large language models and annotation templates to aid in this process. In the field of gymnastics, how to design templates tailored to the required knowledge, how to introduce AI automation tools to optimize this process, and how to refine the granularity down to the Element (the technical movement unit in gymnastics) are all pressing issues that need to be addressed.

[0005] The general multimodal large-scale model paradigm has not been optimized for specific domains. Therefore, many scholars have introduced or designed various parts on the basis of the paradigm for targeted optimization. For example, increasing the context window to input more details, but this will bring greater overhead. Existing work often adopts the equal sampling paradigm. This frame sampling mode is not reasonable in the gymnastics scenario because the frame information density varies significantly in different time periods of a video, which can easily lead to the loss of details. Some methods focus on visual mapping layer processing, aiming to map visual features to the text space more efficiently, such as feature concatenation and clustering. These methods can often be directly improved on the base model, thus avoiding re-pre-training. Targeted optimization of the feature mapping module for different scenarios can not only optimize feature representation, but also reduce a lot of computational overhead. Therefore, how to design a suitable feature mapping and optimization module is a key issue in the design of large-scale models for the gymnastics domain. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a fine-grained method and system for understanding gymnastics videos based on a multimodal large model.

[0007] This invention constructs a professional question-and-answer dataset for the gymnastics field (including eight gymnastics types: floor exercise, horizontal bar, parallel bars, uneven bars, pommel horse, rings, vault, and balance beam) by proposing an annotation method. It also designs and introduces a dual-visual encoder framework, a video frame dynamic sampling module, a visual token compression module, and low-rank adaptive fine-tuning technology, and proposes a method for accurate and efficient fine-grained question-and-answer understanding of gymnastics videos.

[0008] This invention, taking into account the characteristics of gymnastics videos—strong technical patterns, fast movement speed, and uneven information density—proposes a method based on existing multimodal large model approaches. This method includes: 1) a templated process for constructing training datasets; 2) multimodal large model design and optimization based on typical paradigms; and 3) model fine-tuning and inference based on low-rank adaptation. This approach not only endows the multimodal large model with multimodal knowledge from the gymnastics field but also optimizes the model's ability to understand complex movements.

[0009] The first aspect of this invention relates to a fine-grained method for understanding gymnastics videos based on a multimodal large model, comprising the following steps:

[0010] Step 1. Perform hybrid annotation based on decision tree templates, question bank templates, and large model automatic descriptions;

[0011] Step 2. Adaptive frame sampling of video under inter-frame difference analysis: Using the inter-frame difference distribution among all frames of the video, the sampling quota is initially allocated to several equally divided sub-segments according to the floor of the difference ratio, and then the allocation is repeated in turn according to the size of the decimal part; after the allocation is completed, the number of frames allocated evenly in each sub-segment is finally output as the sampling sequence.

[0012] Step 3. Extract visual features from the dual-stream encoder; First, copy the keyframe sequence into two identical frame sequences, then scale all images in these two frame sequences to the predetermined size of the two visual encoders, and finally input the two differently scaled frame sequences into the two encoders to obtain two sets of video features.

[0013] Step 4. Video Feature Mapping and Visual Token Compression: The two sets of video features are mapped and aligned using two multilayer perceptron networks with non-shared parameters. The input linear layer dimensions of the two multilayer perceptrons are set as the output dimensions of the two visual encoders, and the output linear layer dimensions of the two multilayer perceptrons are set as the preset dimensions of the large language model. The two multilayer perceptrons output two different sets of visual tokens. The token compression module consists of two average pooling layers with different pooling kernel sizes. The two sets of visual tokens are then average pooled according to their spatial dimensions and concatenated into a single visual token.

[0014] Step 5. Construct the language module; First, input the text part of the question and answer into the word segmenter corresponding to the large language model to obtain text tokens. Then, concatenate the text tokens with the visual tokens to obtain the complete token sequence. Input the complete tokens into the large language model to obtain the output text token sequence. Use the word segmenter to decode to obtain the text corresponding to the output token.

[0015] Step 6. Supervised fine-tuning and optimization of the model with autoregressive inference; fine-tuning data using instructions, training with a large language model using the low-rank adaptive LoRA fine-tuning method, calculating cross-entropy loss only on the model output and annotation at the position of the answer, and optimizing with the Adam optimizer; in inference mode, the model uses an autoregressive mode for inference, continuously predicting the text token of the next output until the end of the output is marked, and using KV caching technology to optimize repetitive attention calculations.

[0016] Preferably, the hybrid annotation method described in step 1 includes:

[0017] After collection, gymnastics videos were randomly segmented. For video groups containing only one action unit, a classification decision tree annotation method was used, generating a question-answer pair for each decision process. For video groups containing several action units, a question bank template annotation method was used to annotate all relevant questions. The annotation libraries obtained from the first two groups were merged, and the annotations for each video, along with the video itself, were used to generate prompt word templates based on automatic descriptions. These templates were then input into the OpenAI GPT-4o model to obtain refined descriptions. The annotations obtained from the above three methods were combined to obtain all training data.

[0018] Furthermore, the specific steps of the hybrid annotation method described in step 1 are as follows:

[0019] After collection, gymnastics videos are randomly segmented, ensuring that complete technical units are not separated. The video segments are then divided into two groups: group G, which contains only one movement unit. s ={x i |i=1,2,3,…,N s} and video group G containing several action units m ={x i |i=1,2,3,…,N m}; For video group G S Design a complete classification decision tree V = {v i |i=1,2,3,…,N T}, where each node v i Representing one of the questions (i.e., a question-and-answer session with fine-grained understanding), the first two layers of the decision tree represent type classification and technical action unit classification, respectively, while the third layer onwards represents the technical detail classification of action units; for any input x, the decision process of the i-th layer is denoted as g. i (x), each decision-making process generates a set of question-and-answer pairs Y. i The decision-making process is represented as G(x) = (g1(x), g2(x), ..., g L-1 (x)), to obtain the annotation library

[0020] For video group G m For each type of gymnastics, a dedicated question bank template was designed. Each template contains questions and candidate answers for a specific technical unit, and all possible questions (Q) are traversed across all technical units. i Finally, the question bank T is formed. db ={Q i |i=1,2,3,…,N Q}where N QThis represents the total number of template questions; during the annotation process, the technical action-related parts in the question bank are traversed. The annotation process is represented as Q(x)={q|Relative(x,q)=True,q∈T}. db} represents all video-related issues that have been labeled; this process results in a label library.

[0021] The annotations for each individual video obtained earlier, along with the video itself, are used to generate a prompt based on the automatic description. This prompt is then input into the OpenAIGPT-4o multimodal model. The requirement is that the annotation should be as concise as possible, containing descriptive information that covers all descriptive questions, and also describing other details in the video. Finally, the annotation library A is obtained. c This approach leverages the capabilities of a robust, general, multimodal large model to ensure complete accuracy in technical details while simultaneously generating refined descriptions of aspects not addressed in the technical issues. Finally, the three are merged, resulting in A=A. s ∪A m ∪A c After merging, the labelers conduct cross-validation to ensure accuracy.

[0022] Preferably, the video adaptive frame sampling method under inter-frame difference analysis described in step 2 includes:

[0023] A refined sampling strategy combining inter-frame difference analysis is adopted. An inter-frame difference array is calculated to guide sampling. The entire video is first divided into several segments of equal length. Then, the frame sampling problem is transformed into the problem of allocating the target number of samples to these segments. Initial allocation is performed by multiplying the inter-frame difference of the segment corresponding to the proportion of the total difference by the total number of samples (rounded down). Then, the allocation is performed according to the size of the decimal part. This process is repeated until the total number of samples is met. The allocated number is uniformly sampled on these segments. The finally uniformly sampled sequences are spliced ​​together to obtain the sampled frame sequence of the entire video segment.

[0024] Furthermore, the specific steps of the video adaptive frame sampling method under inter-frame difference analysis described in step 2 are as follows:

[0025] Given a video containing T frames, denoted as in This module calculates an inter-frame difference array through the following process. in Calculated in the following way:

[0026]

[0027] Among them, D i G represents the average absolute difference between frame i and frame i+1. i It is the i-th frame F iAfter the calculation is completed following the RGB to L*u*v* color space conversion, copy the last difference value D. T-1 This is then appended to the end of array D, making its total length equal to the number of video frames T. By dividing the entire video into L equal segments, the frame sampling problem is transformed into allocating K target sampled frames to these L segments. The set of frame indices for the i-th segment is denoted as S. i :

[0028]

[0029] Use P i Let P be the number of frames allocated to the i-th segment. i Calculated using the distribution of the difference array D:

[0030]

[0031] when At that time, the remaining Frame, according to The frames are sorted in descending order of size, and one frame is assigned to each segment at a time. This process is repeated until the total number of frames K is met. Then, uniform sampling is performed on each segment. The frame array sampled from the i-th segment is denoted as... Here ({F n |n∈S i},P i ), where U represents the uniform sampling function; finally, all the selected segments are concatenated together to obtain the final sampled frame sequence:

[0032] Preferably, the video feature mapping and token compression method in step 4 includes:

[0033] These raw visual features are mapped into two sets of visual tokens using two multilayer perceptron networks with non-shared parameters. Each multilayer perceptron consists of an input linear layer, a GeLU activation layer, and an output linear layer. The input dimensions of the input linear layer are set to d1 and d2, respectively, aligning with the two visual encoders. The hidden layer dimensions are uniformly set to d1. h The output dimension of the linear layer is d l Align the large language model; use average pooling layers with different pooling kernel sizes to pool the two sets of visual tokens in the spatial dimension, then flatten and stitch them together.

[0034] A second aspect of the present invention relates to a fine-grained gymnastics video understanding system based on a multimodal large model, comprising:

[0035] A data generation module based on a hybrid annotation method combining decision tree templates, question bank templates, and large model automatic descriptions is used to generate high-quality multimodal instruction fine-tuning data for the training phase.

[0036] The video adaptive frame sampling module based on inter-frame difference analysis utilizes the inter-frame difference distribution among all frames of the video. It initially allocates sampling slots to several equally divided sub-segments based on the floor function of the difference percentage, and then repeats the allocation process in turn based on the fractional part. After allocation, the final sampled sequence is output after uniformly sampling the allocated frames from each sub-segment.

[0037] The dual-stream encoder visual feature extraction module copies the sampled sequence and scales it to two preset sizes. It then uses two visual encoders with different purposes to encode the sampled sequence simultaneously, resulting in two different sets of visual features.

[0038] The video feature mapping and token compression module uses two multilayer perceptrons consisting of two linear layers and one activation layer to map visual features to text modalities, called visual tokens. It then uses average pooling layers with different pooling kernel sizes to pool and flatten the two sets of visual tokens in the spatial dimension.

[0039] Language module: Responsible for communicating with humans in the form of natural language. It obtains text tokens using a word segmenter and the embedding layer of a large language model. It concatenates the visual token and text token obtained in the previous step in the length dimension and inputs them into a pre-trained large language model to output the target text token.

[0040] Training and Inference Module: This module uses the instruction-based fine-tuning data constructed in this invention to perform low-rank adaptive fine-tuning on the large model. Cross-entropy loss is used to calculate the loss and perform gradient backpropagation on the response portion, and then an optimizer is used to optimize the target parameters. In inference mode, an autoregressive inference model is used to continuously predict the text token for the next output until the end-of-output marker is reached.

[0041] A third aspect of the present invention relates to a gymnastics motion understanding device based on a multimodal large model, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the gymnastics motion understanding method based on a multimodal large model of Embodiment 1.

[0042] A fourth aspect of the present invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the gymnastic motion understanding method based on a multimodal large model of Embodiment 1.

[0043] The innovation of this invention is:

[0044] 1. This invention is the first to propose a dataset annotation process suitable for fine-grained gymnastics video analysis in natural language. It utilizes decision tree templates, question bank templates, and large model automatic description generation to achieve standardization and semi-automation of the annotation process, which not only ensures fine-grained and high-quality annotation, but also saves human resources.

[0045] 2. This invention innovatively proposes a video adaptive frame sampling method based on inter-frame difference analysis to address the problem of uneven motion density distribution in gymnastics videos and solve the problem of missing details.

[0046] 3. This invention utilizes two different visual encoders for comprehensive feature extraction. A multilayer perceptron is used for feature mapping, and spatial average pooling with a large pooling kernel is innovatively employed to reduce the computational cost and load of the model.

[0047] 4. Based on the paradigm of multimodal large models, this invention constructs a model architecture and training and inference framework suitable for fine-grained gymnastics video analysis, and builds a fine-grained gymnastics video understanding system based on multimodal large models.

[0048] The working principle of this invention is (analyzing the reasons for the advantages of this invention):

[0049] 1. This invention provides fine-grained natural language annotation for technical units in gymnastics. The principle behind this is that gymnastics technical units possess a structured descriptive characteristic; each unit can be categorized by gymnastics class, movement class (i.e., the major movement category to which the unit belongs), and movement detail (e.g., rotation, tumbling, jumping, direction, hand movements, leg movements, etc.). Based on these characteristics, this invention specifically constructs a decision tree annotation method suitable for individual technical movements: classifying from top to bottom to generate annotations; a question bank template annotation method suitable for multiple technical movements: pre-defining movement questions using structured characteristics; and leveraging a powerful, general-purpose model, injecting contextual knowledge through manual annotation to generate corresponding video descriptions.

[0050] 2. This invention utilizes the characteristic that inter-frame differences can represent motion amplitude. Inter-frame difference analysis is used to calculate the difference distribution across all video frames, and this distribution represents the motion amplitude of different parts of the video. Based on this distribution, this invention equally divides the video into sub-segments and samples them evenly to prevent excessively unbalanced frame sampling. Then, it allocates sampling slots based on the proportion of differences in these sub-segments, requiring more samples for areas with larger amplitudes. This approach combines the advantages of dynamic allocation and equal sampling, enabling targeted sampling to address detail loss in unbalanced videos while maintaining good performance in balanced videos.

[0051] 3. Due to differences in pre-training schemes, data, and input methods among different visual encoders, their performance in the feature space varies. Based on this characteristic, this invention utilizes both image and video visual encoders to simultaneously encode features. By combining these features, the visual feature representation is improved, which helps to integrate the static detail bias of images and the dynamic temporal bias of videos, thus enhancing the quality of visual features and improving model performance. This invention uses two multilayer perceptrons with non-shared parameters for feature mapping, ensuring that the two sets of features do not interfere with each other and improving the alignment effect of feature mapping. Furthermore, this invention leverages the characteristic in gymnastics where single static images have low information importance, while temporal information in multiple images has high importance. Spatial pooling is performed while remaining unchanged in the temporal dimension to reduce the number of visual tokens.

[0052] The advantages of this invention are:

[0053] 1. By using decision tree templates, question bank templates, and large model automatic description generation, the annotation process is standardized and semi-automated, which ensures fine-grained and high-quality annotation while saving human resources.

[0054] 2. The video adaptive frame sampling method based on inter-frame difference analysis solves the problem of detail loss caused by equal sampling in videos with uneven motion, and does not affect the performance of balanced videos.

[0055] 3. Improved visual feature quality by utilizing multiple encoders. Two multilayer perceptrons with non-shared parameters were used to improve feature map alignment and reduce the number of visual tokens, thereby reducing model computation. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the annotation method in this invention.

[0057] Figure 2 This is a schematic diagram of the multimodal large model in this invention.

[0058] Figure 3 This is a schematic diagram of the device of the present invention. Detailed Implementation

[0059] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0060] Example 1

[0061] The technical solution of the present invention will be clearly and completely explained and described below with reference to the accompanying drawings.

[0062] This embodiment relates to a fine-grained method for understanding gymnastics videos based on a multimodal large model, such as... Figure 1 It includes the following steps:

[0063] 1. Perform hybrid annotation based on decision tree templates, question bank templates, and large model automatic descriptions.

[0064] In a fine-grained gymnastics video understanding method based on a multimodal large model, the gymnastics videos are randomly segmented after collection (segmentation must ensure that complete technical units are not separated). At this point, the video segments can be divided into two groups: a group G containing only one action unit. s ={x i |i=1,2,3,…,N s} and video group G containing several action units m ={x i |i=1,2,3,…,N m For video group G S Design a complete classification decision tree V = {v i |i=1,2,3,…,N T}, where each node v i Representing one of the questions (i.e., a question-and-answer session with fine-grained understanding), the first two layers of the decision tree represent type classification and technical action unit classification, respectively. From the third layer onwards, they represent the technical detail classification of action units. For any input x, the decision process in the i-th layer is denoted as g. i (x), each decision-making process generates a set of question-and-answer pairs Y. i The decision-making process is represented as G(x) = (g1(x), g2(x), ..., g L-1 (x)), to obtain the annotation library

[0065] For video group G m For each type of gymnastics, a dedicated question bank template is designed. Each template contains a question and candidate answers for a specific technical unit (e.g., the type of jump, number of rolls, number of spins, etc. in this video). All possible questions (Q) for each technical unit are traversed. i Finally, the question bank T is formed. db ={Q i |i=1,2,3,…,N Q}where N Q This represents the total number of template questions. During the annotation process, the technical action-related parts of the question bank are traversed. The annotation process is represented as Q(x)={q|Relative(x,q=True,q∈T} db} represents all video-related issues that have been labeled. This process generates a label library.

[0066] This invention significantly enhances video understanding capabilities through additional video descriptions, thus introducing a large-scale automatic description generation module. The previously obtained single-video annotations, along with the video itself, are used to generate a prompt based on the automatic description, which is then input into the OpenAI GPT-4o multimodal model. The descriptions must be as concise as possible, covering all descriptive issues, and also describing other details in the video. Finally, an annotation library A is obtained. c Leveraging the capabilities of a strong multimodal large model, this approach ensures complete accuracy in technical details while also generating detailed descriptions of aspects not addressed in the technical issues. Finally, the three are merged, and A = A. s ∪A m ∪A c After merging, the labelers conduct cross-validation to ensure accuracy.

[0067] 2. Adaptive frame sampling for video under inter-frame difference analysis.

[0068] This invention overcomes the limitations of traditional uniform sampling methods in capturing subtle differences and key dynamics in gymnastic movements by employing a refined sampling strategy that combines inter-frame difference analysis. Given a video containing T frames, denoted as... in This module calculates an inter-frame difference array through the following process. in It can be calculated in the following ways:

[0069]

[0070] Among them, D i G represents the average absolute difference between frame i and frame i+1. i It is the i-th frame F i After the calculation is completed following the RGB to L*u*v* color space conversion, copy the last difference value D. T-1 This is then appended to the end of array D, making its total length equal to the number of video frames T. By dividing the entire video into L equal segments, the frame sampling problem can be transformed into distributing K target sampled frames into these L segments. We denote the set of frame indices for the i-th segment as S. i :

[0071]

[0072] Use P i Let P be the number of frames allocated to the i-th segment. i It can be calculated using the distribution of the difference array D:

[0073]

[0074] when At that time, the remaining Frame, according to The frames are sorted in descending order of size, and one frame is assigned to each segment at a time. This process is repeated until the total number of frames K is met. Then, uniform sampling is performed on each segment. We denote the frame array sampled from the i-th segment as... Here ({F n |n∈S i},P i ), where U represents the uniform sampling function. Finally, all the selected segments are concatenated together to obtain the final sampled frame sequence:

[0075] 3. Extract visual features from the dual-stream encoder.

[0076] This invention uses a dual-stream visual encoder to enhance visual feature extraction. The two encoders are the SigLIP or CLIP model, suitable for single images, and the InternVideo2 model, suitable for continuous frame video. The keyframe sequence F... S The sequence is copied into two identical frame sequences. Then, all images in these two frame sequences are scaled to predetermined sizes for the two visual encoders: 224×244 and 336×336, respectively. Finally, the two scaled frame sequences are input into the two encoders respectively to obtain two sets of video features. The two sets of features output by this module are represented as follows: and

[0077] 4. Compressed video feature mapping and tokens.

[0078] The raw visual features obtained in step 3 need to undergo additional alignment before being input into the language module. In the visual feature mapping module, two multilayer perceptrons are used as feature mapping layers, denoted as Pi. i and P v A single multilayer perceptron consists of an input linear layer, a GeLU activation layer, and an output linear layer. The input dimensions of the input linear layers are set to d1 and d2, respectively, and the hidden layer dimensions are uniformly d1 and d2. h The output dimension of the linear layer is d l The two sets of features are respectively input into P. i and P v In the process, visual token embedding is obtained. and

[0079] E i =P i (V i(5)

[0080] E v =P v (V v (6)

[0081] In obtaining E i and E v After embedding the two sets of visual tokens, this module, considering the characteristics of gymnastics tasks that emphasize temporal relationships and de-emphasize image details, additionally uses average pooling technology at the spatial scale to compress these two sets of visual token embeddings. Among these, because E i Due to greater visual redundancy, average pooling with a 4×4 spatial pooling kernel size is used, E v We use average pooling with a 2×2 spatial pooling kernel size. After token compression, we obtain:

[0082]

[0083] The final number of tokens obtained through this method is approximately 12% of the original number, which significantly accelerates the inference and training process. After compression... And the growth dimension is flattened in length and width.

[0084] 5. Build language modules.

[0085] First, the linguistic portion of the question and answer is input into the word segmenter corresponding to the large language model to obtain the text token E. text Then, the text tokens are concatenated with the visual tokens to obtain the complete token sequence.

[0086]

[0087] The complete token is input into the Large Language Model (LLM) to obtain the output literal token sequence E. output =LLM(E input The output text token E is obtained using the decoding function of the word segmenter. output The corresponding text. This module is compatible with pre-trained open-source large models such as Llama3, Qwen2.5, and Phi3.

[0088] 6. Training and Reasoning.

[0089] The training module uses the instruction fine-tuning data constructed in step 1, employing the LoRA (Low-Rank Adaptation) large language model fine-tuning method. Cross-entropy loss is calculated only at the position of the response on the model output and annotations, and the Adam optimizer is used for model training optimization. In inference mode, the model uses an autoregressive inference pattern, continuously predicting the text token for the next output until the output ends with a marker, and uses a key-value caching technique to optimize unnecessary computations.

[0090] Example 2

[0091] This embodiment relates to a gymnastics motion understanding system based on a multimodal large model, used to implement the gymnastics motion understanding method based on a multimodal large model in Embodiment 1, including:

[0092] A data generation module based on a hybrid annotation method combining decision tree templates, question bank templates, and large model automatic descriptions is used to generate high-quality multimodal instruction fine-tuning data for the training phase.

[0093] The video adaptive frame sampling module based on inter-frame difference analysis utilizes the inter-frame difference distribution among all frames of the video. It initially allocates sampling slots to several equally divided sub-segments based on the floor function of the difference percentage, and then repeats the allocation process in turn based on the fractional part. After allocation, the final sampled sequence is output after uniformly sampling the allocated frames from each sub-segment.

[0094] The dual-stream encoder visual feature extraction module copies the sampled sequence and scales it to two preset sizes. It then uses two visual encoders with different purposes to encode the sampled sequence simultaneously, resulting in two different sets of visual features.

[0095] The video feature mapping and token compression module uses two multilayer perceptrons consisting of two linear layers and one activation layer to map visual features to text modalities, called visual tokens. It then uses average pooling layers with different pooling kernel sizes to pool and flatten the two sets of visual tokens in the spatial dimension.

[0096] Language module: Responsible for communicating with humans in the form of natural language. It obtains text tokens using a word segmenter and the embedding layer of a large language model. It concatenates the visual token and text token obtained in the previous step in the length dimension and inputs them into a pre-trained large language model to output the target text token.

[0097] Training and Inference Module: This module uses the instruction-based fine-tuning data constructed in this invention to perform low-rank adaptive fine-tuning on the large model. Cross-entropy loss is used to calculate the loss and perform gradient backpropagation on the response portion, and then an optimizer is used to optimize the target parameters. In inference mode, an autoregressive inference model is used to continuously predict the text token for the next output until the end-of-output marker is reached.

[0098] Example 3

[0099] This embodiment relates to a gymnastic motion understanding device based on a multimodal large model, such as... Figure 3 As shown, it includes a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the gymnastic motion understanding method based on a multimodal large model of Embodiment 1.

[0100] Example 4

[0101] A fourth aspect of the present invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the gymnastic motion understanding method based on a multimodal large model of Embodiment 1.

[0102] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A fine-grained method for understanding gymnastics videos based on a multimodal large model, comprising the following steps: Step 1. Perform hybrid annotation based on decision tree templates, question bank templates, and large model automatic descriptions; Step 2. Adaptive frame sampling of video under inter-frame difference analysis: Using the inter-frame difference distribution among all frames of the video, the sampling quota is initially allocated to several equally divided sub-segments according to the floor of the difference ratio, and then the allocation is repeated in turn according to the size of the decimal part; after the allocation is completed, the number of frames allocated evenly in each sub-segment is finally output as the sampling sequence. Step 3. Extract visual features from the dual-stream encoder; First, copy the keyframe sequence into two identical frame sequences, then scale all images in these two frame sequences to the predetermined size of the two visual encoders, and finally input the two differently scaled frame sequences into the two encoders to obtain two sets of video features. Step 4. Video Feature Mapping and Visual Token Compression: The two sets of video features are mapped and aligned using two multilayer perceptron networks with non-shared parameters. The input linear layer dimensions of the two multilayer perceptrons are set as the output dimensions of the two visual encoders, and the output linear layer dimensions of the two multilayer perceptrons are set as the preset dimensions of the large language model. The two multilayer perceptrons output two different sets of visual tokens. The token compression module consists of two average pooling layers with different pooling kernel sizes. The two sets of visual tokens are then average pooled according to their spatial dimensions and concatenated into a single visual token. Step 5. Construct the language module; First, input the text part of the question and answer into the word segmenter corresponding to the large language model to obtain text tokens. Then, concatenate the text tokens with the visual tokens to obtain the complete token sequence. Input the complete tokens into the large language model to obtain the output text token sequence. Use the word segmenter to decode to obtain the text corresponding to the output token. Step 6. Supervised fine-tuning and optimization of the model with autoregressive inference; fine-tuning data using instructions, training with a large language model using the low-rank adaptive LoRA fine-tuning method, calculating cross-entropy loss only on the model output and annotation at the position of the answer, and optimizing with the Adam optimizer; in inference mode, the model uses an autoregressive mode for inference, continuously predicting the text token of the next output until the end of the output is marked, and using KV caching technology to optimize repetitive attention calculations.

2. The fine-grained gymnastics video understanding method based on a multimodal large model as described in claim 1, characterized in that: The hybrid annotation method described in step 1 includes: After collection, gymnastics videos were randomly segmented. For video groups containing only one action unit, a classification decision tree annotation method was used, generating a question-answer pair for each decision process. For video groups containing several action units, a question bank template annotation method was used to annotate all relevant questions. The annotation libraries obtained from the first two groups were merged, and the annotations of individual videos, along with the videos themselves, were used to generate prompt word templates based on automatic descriptions and input into the OpenAI GPT-4o model to obtain refined descriptions. The annotations obtained from the above three methods were combined to obtain all training data.

3. The fine-grained gymnastics video understanding method based on a multimodal large model as described in claim 2, characterized in that: The specific steps of the hybrid annotation method described in step 1 are as follows: After collection, gymnastics videos are randomly segmented, ensuring that complete technical units are not separated. The video segments are then divided into two groups: group G, which contains only one movement unit. s ={x i |i=1,2,3,…,N s } and video group G containing several action units m ={x i |i=1,2,3,…,N m }; For video group G S Design a complete classification decision tree V = {v i |i=1,2,3,…,N T }, where each node v i Representing one of the questions (i.e., a question-and-answer session with fine-grained understanding), the first two layers of the decision tree represent type classification and technical action unit classification, respectively, while the third layer onwards represents the technical detail classification of action units; for any input x, the decision process of the i-th layer is denoted as g. i (x), each decision-making process generates a set of question-and-answer pairs Y. i The decision-making process is represented as G(x) = (g1(x), g2(x), ..., g L-1 (x)), to obtain the annotation library For video group G m For each type of gymnastics, a dedicated question bank template was designed. Each template contains questions and candidate answers for a specific technical unit, and all possible questions (Q) are traversed across all technical units. i Finally, the question bank T is formed. db ={Q i |i=1,2,3,…,N Q }where N Q Represents the total number of template questions; During the annotation process, the technical action-related parts of the question bank are traversed. The annotation process is represented as Q(x)={q|Relative(x,q=True,q∈T} db } indicates that all video-related issues have been marked; This process yields the annotation library. The annotations for each individual video obtained earlier, along with the video itself, are used to generate a prompt based on the automatic description. This prompt is then input into the OpenAIGPT-4o multimodal model. The requirement is that the annotation should be as concise as possible, containing descriptive information that covers all descriptive questions, and also describing other details in the video. Finally, the annotation library A is obtained. c This approach leverages the capabilities of a robust, general, multimodal large model to ensure complete accuracy in technical details while simultaneously generating refined descriptions of aspects not addressed in the technical issues. Finally, the three are merged, resulting in A=A. s ∪A m ∪A c After merging, the labelers conduct cross-validation to ensure accuracy.

4. The fine-grained gymnastics video understanding method based on a multimodal large model as described in claim 1, characterized in that: The video adaptive frame sampling method under inter-frame difference analysis described in step 2 includes: A refined sampling strategy combining inter-frame difference analysis is adopted; To guide sampling, an inter-frame difference array is calculated. The entire video is first divided into several segments of equal length. Then, the frame sampling problem is transformed into the problem of allocating the target number of samples to these segments. Initial allocation is performed by multiplying the inter-frame difference of the segment corresponding to the proportion of the total difference by the total number of samples (rounded down). Then, allocation is performed sequentially according to the size of the decimal part. This process is repeated until the total number of samples is met. The allocated number of samples is uniformly sampled on these segments. The final uniformly sampled sequences are then concatenated to obtain the sampled frame sequence of the entire video segment.

5. The fine-grained gymnastics video understanding method based on a multimodal large model as described in claim 4, characterized in that: The specific steps of the video adaptive frame sampling method under inter-frame difference analysis described in step 2 are as follows: Given a video containing T frames, denoted as in This module calculates an inter-frame difference array through the following process. in Calculated in the following way: G i =Rgb2luv(F i ), (2) Among them, D i G represents the average absolute difference between frame i and frame i+1. i It is the i-th frame F i After the calculation is completed following the RGB to L*u*v* color space conversion, copy the last difference value D. T-1 This is then appended to the end of array D, making its total length equal to the number of video frames T. By dividing the entire video into L equal segments, the frame sampling problem is transformed into allocating K target sampled frames to these L segments. The set of frame indices for the i-th segment is denoted as S. i : Use P i Let P be the number of frames allocated to the i-th segment. i Calculated using the distribution of the difference array D: when At that time, the remaining Frame, according to P i f The frames are sorted in descending order of size, and one frame is assigned to each segment at a time. This process is repeated until the total number of frames K is met. Then, uniform sampling is performed on each segment. The frame array sampled from the i-th segment is denoted as... Here Where U represents the uniform sampling function; finally, all the selected segments are concatenated together to obtain the final sampled frame sequence:

6. The fine-grained gymnastics video understanding method based on a multimodal large model as described in claim 1, characterized in that: The video feature mapping and token compression method described in step 4 includes: These raw visual features are mapped into two sets of visual tokens using two multilayer perceptron networks with non-shared parameters. Each multilayer perceptron consists of an input linear layer, a GeLU activation layer, and an output linear layer. The input dimensions of the input linear layer are set to d1 and d2, respectively, aligning with the two visual encoders. The hidden layer dimensions are uniformly set to d1. h The output dimension of the linear layer is d l Align the large language model; use average pooling layers with different pooling kernel sizes to pool the two sets of visual tokens in the spatial dimension, then flatten and stitch them together.

7. A system for implementing the fine-grained gymnastics video understanding method based on a multimodal large model as described in claim 1, characterized in that, include: A data generation module based on a hybrid annotation method of decision tree template + question bank template + large model automatic description is used to generate high-quality multimodal instruction fine-tuning data for the training phase. The video adaptive frame sampling module under inter-frame difference analysis utilizes the inter-frame difference distribution among all frames of the video to initially allocate sampling slots to several equally divided sub-segments according to the floor of the difference ratio, and then repeatedly allocates them in turn according to the size of the decimal part; after the allocation is completed, the number of frames allocated evenly in each sub-segment is finally output as the sampling sequence. The dual-stream encoder visual feature extraction module copies the sampled sequence and scales it to two preset sizes. It then uses two visual encoders with different purposes to encode the sampled sequence simultaneously, resulting in two different sets of visual features. The video feature mapping and token compression module uses two multilayer perceptrons consisting of two linear layers and one activation layer to map visual features to text modalities, called visual tokens. It uses average pooling layers with different pooling kernel sizes to pool the two sets of visual tokens in the spatial dimension, flatten them, and then stitch them together. The language module is used to communicate with humans in the form of natural language. It uses a word segmenter and the embedding layer of a large language model to obtain text tokens. The visual token obtained in the previous step is concatenated with the text token in the length dimension and then input into a pre-trained large language model to output the target text token. The training and inference module uses the aforementioned instructions to fine-tune the data, performs low-rank adaptive fine-tuning on the large model, uses cross-entropy loss to calculate the loss and backpropagate the gradient in the response part, and then uses an optimizer to optimize the target parameters; in inference mode, an autoregressive inference mode is used to continuously predict the text token of the next output until the output end marker is reached.

8. A gymnastics motion understanding device based on a multimodal large model, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the gymnastic motion understanding method based on a multimodal large model as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the gymnastic motion understanding method based on a multimodal large model as described in any one of claims 1-6.

Images