Motion video understanding method and device based on multi-modal large model

Abstract

The invention discloses a motion video understanding method and device based on a multi-modal large model, and the method comprises the steps: (1) carrying out the uniform sampling of an inputted motion video, extracting K video frames, and extracting the visual features of each video frame through a visual encoder; (2) mapping the visual features to hidden space dimensions of a large language model through a multi-layer perceptron, and generating mapped visual features; (3) splicing the mapped visual features and the user query into a comprehensive prompt, and inputting a large language model to generate a text response; and (4) adopting a multi-stage training strategy to train the multi-modal large model, wherein the multi-stage training strategy comprises visual language alignment pre-training, video instruction following fine-tuning training and sports knowledge fine-tuning training. According to the method, the key visual information in the video can be effectively extracted, so that the ability of understanding a dynamic complex scene in the motion video is improved, and the time dependency relationship between the video frames is fully utilized, so that the accuracy and efficiency of understanding the motion video are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically a method and apparatus for understanding motion videos based on a multimodal large model. Background Technology

[0002] Sports video understanding is a crucial research topic in computer vision and natural language processing, aiming to deeply analyze and interpret the actions, events, and scenes in sports event videos. With the rapid growth and diversification of video content, the ability to accurately understand and describe sports videos is particularly critical in applications such as event analysis, automated commentary generation, and athlete performance evaluation. However, existing sports video understanding methods are typically trained only on small datasets, resulting in models that perform well in specific scenarios but lack generalization ability on new datasets.

[0003] Traditional motion video understanding techniques primarily rely on visual feature extraction and temporal series modeling. Specifically, these methods typically employ convolutional neural networks (CNNs) to process video frames and extract key features such as athlete posture, movement trajectories, and field layout. While 3D CNNs can capture local temporal features, their performance in handling long-term dependencies remains limited. Furthermore, although CNNs are highly effective at extracting features from static images, they often fail to adequately capture the temporal information of motion in dynamic scenes. In sports videos, athlete movements and events are continuous, with high temporal dependencies between frames. Traditional methods typically model these dependencies by inputting the video frame sequence into a time series model (such as a Long Short-Term Memory network, LSTM). However, LSTMs may encounter vanishing or exploding gradients when processing long-term sequences, leading to a decline in their ability to capture long-term dependencies. In addition, LSTMs also have limitations in text understanding and generation, being sensitive to the order of the input sequence; discontinuous information between frames or the presence of noise can significantly reduce the accuracy of understanding.

[0004] With the rapid development of large-scale pre-trained models, such as CLIP (Contrastive Language-Image Pretraining) and Large Language Models (LLM), researchers have begun to apply these advanced models to multimodal video understanding tasks. In particular, the emergence of Large Multimodal Language Models (MLLM) has significantly improved the information fusion and logical reasoning capabilities between visual and textual data. However, current applications of MLLM are mainly concentrated on broad, general-purpose datasets, and have not yet been deeply explored in the field of motion videos. Therefore, how to effectively utilize the fusion and reasoning capabilities of large multimodal models to improve the accuracy and efficiency of motion video understanding remains a pressing challenge. Summary of the Invention

[0005] This embodiment provides a motion video understanding method, system, electronic device, and storage medium based on a multimodal large model to solve the problems of low accuracy and efficiency in motion video understanding caused by the inability to effectively utilize the fusion and reasoning capabilities of multimodal large models in related technologies.

[0006] In a first aspect, embodiments of the present invention provide a motion video understanding method based on a multimodal large model, the motion video understanding method based on a multimodal large model comprising: (1) The input motion video is uniformly sampled to extract K video frames, and the visual features of each video frame are extracted using a visual encoder; (2) Visual features are mapped to the latent space dimension of the large language model through a multilayer perceptron to generate mapped visual features; (3) The mapped visual features are combined with the user query to form a comprehensive prompt, which is then input into the large language model to generate a text response; (4) A multi-stage training strategy is adopted to train the multimodal large model, including visual language alignment pre-training, video instruction follow-up fine-tuning training and sports knowledge fine-tuning training.

[0007] In an optional embodiment, the number of frames for uniform sampling in step (1) is N / K, where N is the total number of frames in the input video, K is the number of sampling frames, and K is 16.

[0008] In an optional embodiment, the visual encoder in step (1) is the CLIP ViT-L / 14 model, whose input image resolution is 224×224, divided into 14×14 pixel regions, and extracts 1024-dimensional visual features through 24 Transformer encoding layers.

[0009] In an optional embodiment, the multilayer perceptron in step (2) comprises L linear layers, the preceding... The layers are followed by ReLU activation functions, and the output of the last layer is mapped to a 4096-dimensional latent space feature, where L=2 and the intermediate dimension is 2048.

[0010] In an optional embodiment, the large language model in step (3) is a Vicuna 3B parameter model, whose input prompt text contains image token placeholders, each placeholder is replaced with 196 visual features, and the relative position is encoded by rotating the embedding vector.

[0011] In an optional embodiment, the rotation embedding vector is implemented using the following formula: in, This represents the embedding vector generated by rotating the embedding vector; Discrete tokens for text. This indicates the absolute index of the current token in the sequence; The initial embedding vector for the input token; This represents the rotation matrix applied to the i-th pair of adjacent dimensions (2i and 2i+1), generated by trigonometric functions, used to calculate the attention score in the self-attention mechanism.

[0012] In an optional embodiment, the rotation matrix is ​​defined as: ; in, Indicates the current token's position index. This indicates the embedded dimension index.

[0013] Compared with existing technologies, the beneficial effects of the motion video understanding method based on a multimodal large model of the present invention are as follows: The multimodal large model of this invention exhibits significant technical advantages in several aspects. First, its pre-training with large-scale and diverse data significantly enhances the model's generalization ability. This design enables the model to flexibly adapt to different sports scenarios and task requirements, effectively overcoming the performance degradation problem caused by insufficient training data, thus demonstrating stronger adaptability and reliability in practical applications.

[0014] Secondly, the model of this invention excels in temporal understanding. Through self-attention mechanisms and sequence modeling techniques, the model can efficiently capture the relationships between frames in a video and deeply analyze time-series information. This enables the model to identify and predict key moments in motion, improving its understanding of dynamic scenes and providing more accurate support for motion video analysis.

[0015] Furthermore, this invention employs a multi-level feature extraction mechanism, enhancing the understanding of fine-grained visual features and multimodal fusion capabilities. This mechanism allows the model to meticulously capture key details and contextual information in videos, and improves the understanding of complex events through cross-modal learning. This design effectively compensates for the shortcomings of traditional methods in detail capture, ensuring higher recognition accuracy and description quality in complex scenes, thereby improving the overall understanding effect.

[0016] Secondly, embodiments of the present invention provide a motion video understanding device based on a multimodal large model, comprising: The visual feature extraction module is used to uniformly sample the input motion video, extract K video frames, and use a visual encoder to extract the visual features of each video frame. The visual feature mapping module is used to map visual features to the latent space dimension of a large language model through a multilayer perceptron, generating mapped visual features. The text generation module is used to combine the mapped visual features with the user query to form a comprehensive prompt, which is then input into a large language model to generate a text response. The multi-stage training module is used to train multimodal large models using a multi-stage training strategy, including visual language alignment pre-training, video instruction follow-up fine-tuning training, and sports knowledge fine-tuning training.

[0017] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.

[0018] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the motion video understanding method based on a multimodal large model as described in the first aspect.

[0019] Compared with the prior art, the beneficial effects of the motion video understanding device, electronic device and storage medium based on multimodal large model of the present invention are the same as those of the motion video understanding method based on multimodal large model described in the first aspect, so they will not be repeated here. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the motion video understanding method based on a multimodal large model in an embodiment of the present invention; Figure 2 This is a structural diagram of the Transformer in an embodiment of the present invention; Figure 3 This is a flowchart of the multi-stage training strategy in an embodiment of the present invention; Figure 4 This is a structural block diagram of the motion video understanding device based on a multimodal large model in an embodiment of the present invention; Figure 5 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation

[0022] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0023] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0024] This invention provides a motion video understanding method based on a multimodal large model. Figure 1 This is a flowchart of the motion video understanding method based on a multimodal large model according to the present invention, such as... Figure 1 As shown, the process includes the following steps: S100. The input motion video is uniformly sampled to extract K video frames, and the visual features of each video frame are extracted using a visual encoder. In this embodiment, the number of frames sampled uniformly in step S100 is N / K, where N is the total number of frames in the input video, K is the number of sampled frames, and K is 16. In this embodiment, the visual encoder is CLIP's ViT-L / 14 model, whose input image resolution is 224×224, divided into 14×14 pixel regions, and extracts 1024-dimensional visual features through 24 Transformer coding layers.

[0025] For example, assuming the input video is a continuous sequence of N frames, and the number of frames to be sampled is K, then the interval is [number of frames]. In this embodiment, 16 video frames are uniformly sampled. Next, the Vit-L / 14 visual encoder in CLIP is used to extract features from each sampled frame, which is based on the ViT architecture. The input image is first scaled to a resolution of 224×224, and then spatially divided into 196 regions of size 14×14 pixels. Each region is mapped to a 768-dimensional vector through a linear projection layer, and trigonometric function absolute position encoding is added to preserve spatial information. Then, it passes through 24 Transformer encoding layers, and each region obtains 1024-dimensional visual features. The specific architecture of the Transformer module is introduced in the text generation module. In this embodiment, 16 sampled video frames are input into the Vit-L / 14 visual encoder in CLIP to obtain 16×196×1024 visual features.

[0026] S200: Visual features are mapped to the latent space dimension of the large language model through a multilayer perceptron to generate mapped visual features. First, it should be noted that the multilayer perceptron in step S200 includes L linear layers, the preceding... The layers are followed by ReLU activation functions, and the output of the last layer is mapped to a 4096-dimensional latent space feature, where L=2 and the intermediate dimension is 2048.

[0027] Specifically, this involves transforming the image features obtained by the CLIP visual encoder into the latent space dimension of a large language model. Specifically, a multilayer perceptron (MLP) is used... The structure consists of linear layers, front Each layer is followed by a ReLU activation function, and the visual features mapped by the output of the last layer are: In this embodiment, the MLP has two linear layers, the input CLIP feature dimension is 1024, the output large model latent space feature dimension is 4096, and the intermediate dimension is 2048. This mapping process aims to ensure that visual information is compatible with the feature space of the large language model, thereby supporting subsequent multimodal analysis.

[0028] S300: Combine the mapped visual features with the user query to form a comprehensive prompt, and input it into the large language model to generate a text response; It should be noted that the large language model in step S300 is the Vicuna 3B parameter model, whose input prompt text contains image token placeholders. Each placeholder is replaced with 196 visual features and is encoded in relative position by rotating the embedding vector.

[0029] Specifically, the mapped visual features are concatenated with the user-input query information to form a comprehensive prompt, providing contextual information for the model. The large language model generates a corresponding text response based on this prompt, aiming to ensure the relevance and accuracy of the generated content. In this embodiment, each video contains 16 video frames, and each frame contains 196 visual features. In the text prompt input to the large language model, image token placeholders are first used. Instead of specific image features, each image token placeholder is placed on a separate line and then concatenated with the user-input query to form the prompt text. The large language model accepts the prompt text and processes each image token placeholder. The response is further output by replacing the 196 specific visual features. In this embodiment, the Vicuna 3B parameter large language model is used, which is composed of several stacked transformer modules.

[0030] Combination Figure 2 As shown, the transformer module will now be explained in detail: First, input embedding: This involves embedding discrete text tokens. Convert to continuous vector representation , in The dimension of the embedded vector is 4096.

[0031] Secondly, relative position encoding based on rotation embedding vectors is used. Specifically, since the input text accepted by the Large Language Model (LLM) language model is very long, this step uses relative position encoding by introducing a relative position matrix. Its elements The formula used to represent the relative distance between tokens at index i and index j is: ; Rotated Embedding Vector (ROPE) is an improved relative position encoding method that incorporates positional information by rotating the embedding vector. This method effectively handles long sequences while maintaining low computational complexity. The rotated embedding vector is implemented using the following formula: ; in, This represents the embedding vector generated by rotating the embedding vector; Discrete tokens for text. This indicates the absolute index of the current token in the sequence; The initial embedding vector for the input token; This represents the rotation matrix acting on the i-th pair of adjacent dimensions (2i and 2i+1), generated by trigonometric functions, used to calculate the attention score in the self-attention mechanism.

[0032] Furthermore, the rotation matrix is ​​defined as: ; in, Indicates the current token's position index. This indicates the embedded dimension index.

[0033] In the self-attention mechanism, all input embeddings The input embedding can be obtained by passing it through three different learnable linear layers. The vector can be directly used to calculate the attention score after rotation: ; Then the residuals are normalized, which is used in this step to stabilize the training process. The formula is: ; in Representing the input before the attention layer, the LayerNorm normalization formula is further as follows: , yes The mean and variance, and These are learnable parameters.

[0034] In the feedforward network stage, the calculation formula for this step is: ; in For the output of the self-attention layer, This is a learnable weight matrix used for the linear transformation of the feedforward network. As a bias term, it is added to the linear transformation of the feedforward network. This is the activation function.

[0035] Finally, the probability distribution prediction step transforms the final output obtained after passing through several transformer blocks into a probability distribution: ; in, This is the weight matrix of the output layer, used to map the final hidden states to the dimensions of the vocabulary. For the bias term of the output layer Indicates the preceding words In this case, the current word The probability distribution.

[0036] The S400 employs a multi-stage training strategy to train multimodal large models, including visual language alignment pre-training, video instruction follow-up fine-tuning training, and sports knowledge fine-tuning training.

[0037] Combination Figure 3 As shown, in this embodiment, the visual-language alignment pre-training stage utilizes image-text pair data to train the model, achieving effective alignment between visual features and the latent space features of the large language model. The model is optimized using a cross-entropy loss function to ensure it accurately understands and generates text information related to the input image. This embodiment employs wide-area image-text pair data. ,in Represents an image. This represents the corresponding text, used to train a multimodal large language model. The network model is trained using the cross-entropy loss function, as shown in the formula: ; in This indicates the current output response token of the LLM. Indicates the previously output response token, This indicates the length of the output response. The specific parameter configuration for this training step is: 1 training round, learning rate of 0.001, and batch size of 108.

[0038] Video commands follow a fine-tuning phase, using wide-area video question-and-answer data. ,in Indicates video, This indicates a user question. This represents the preset correct answer. During training, videos and user questions are input into the multimodal large language model to train the network to generate preset answers. The model is trained using the cross-entropy loss function, similar to the pre-training stage of visual language alignment mentioned above. The specific parameter configuration for this training step is: 1 training epoch, learning rate of 0.0001, and batch size of 108.

[0039] The sports knowledge fine-tuning utilizes wide-area video question-and-answer data. ,in Video clips representing sporting events, This refers to questions regarding sports events. This indicates the answer to the question. The training process follows the fine-tuning phase, as shown in the video instructions. The specific parameter configuration for this step is: 2 training epochs, learning rate of 0.00001, and batch size of 32.

[0040] The multimodal large model of this invention exhibits significant technical advantages in several aspects. First, its pre-training on large-scale and diverse data significantly enhances the model's generalization ability. This design enables the model to flexibly adapt to different sports scenarios and task requirements. Compared to the limitations of traditional methods trained on specific datasets, this model performs better on new datasets, effectively overcoming the performance degradation problem caused by insufficient training data, thus demonstrating stronger adaptability and reliability in practical applications.

[0041] Secondly, the model of this invention excels in temporal understanding. Through self-attention mechanisms and sequence modeling techniques, the model can efficiently capture the relationships between frames in a video and deeply analyze time-series information. This enables the model to identify and predict key moments in motion, improving its understanding of dynamic scenes and providing more accurate support for motion video analysis.

[0042] Furthermore, this invention employs a multi-level feature extraction mechanism, enhancing the understanding of fine-grained visual features and multimodal fusion capabilities. This mechanism allows the model to meticulously capture key details and contextual information in videos, and improves the understanding of complex events through cross-modal learning. This design effectively compensates for the shortcomings of traditional methods in detail capture, ensuring higher recognition accuracy and description quality in complex scenes, thereby improving the overall understanding effect.

[0043] In summary, the multimodal large model of this invention has significant value in terms of technological innovation and practical application effects. Its design concept and implementation method have shown broad application prospects in sports video analysis and other related fields, providing an effective solution for future multimodal understanding tasks.

[0044] To verify the effectiveness of this invention, experiments were conducted on the real Sports-QA dataset, which covers various sports such as basketball, football, tennis, and gymnastics, and contains 18,718 question-answer pairs for testing. The answer categories cover 191 preset options. Question types involve multiple dimensions, including descriptive, counting, temporal understanding, and causal understanding, to comprehensively evaluate the model's performance. The evaluation metric is question accuracy, designed to quantify the model's performance across different question types.

[0045] Unlike traditional methods that treat this task as a closed-domain classification task, this invention transforms it into a more challenging open-ended generative response task, thereby more comprehensively evaluating the capabilities of large language models in language understanding and generation. In experiments, the method of this invention was compared with four control methods: BlindQA, CoMem, HME, and MASN, as shown in Table 1.

[0046] Table 1 Results of the motion video question-and-answer experiment

[0047] Experimental results show that the SportLLM (i.e., a multimodal large model trained through multiple stages) proposed in this invention significantly outperforms traditional classification methods in open-ended question answering capabilities, fully demonstrating its superior sports knowledge understanding and text generation capabilities. This result indicates that SportLLM can not only accurately answer user questions but also exhibit good adaptability and flexibility in complex sports scenarios, further validating its practical value in dynamic and diverse environments.

[0048] This invention aims to effectively integrate sports-related expertise into a multimodal large language model. This strategy has a wide range of applications, covering multiple fields such as event analysis, automatic commentary generation, and athlete performance evaluation, demonstrating significant practical value and broad application prospects.

[0049] In match analysis, this invention can deeply mine match data, providing accurate tactical analysis and match predictions to help coaches and analysts develop more effective strategies. By combining multimodal data, the model can integrate video, text, and statistical data to generate comprehensive match reports, enhancing the depth and accuracy of the analysis. Specifically, the model can analyze key moments in a match, identifying subtle differences in tactical changes and player performance, thereby providing data-driven decision support for coaching teams. Furthermore, the model can predict possible outcomes of future matches through comparison and trend analysis of historical data, providing a scientific basis for team strategic deployment.

[0050] In terms of automatic commentary generation, the model of this invention can interpret the progress of the game in real time and generate fluent and emotionally resonant commentary. This not only enhances the viewing experience for the audience but also provides a more efficient solution for live sports broadcasts, especially in large-scale events, reducing the workload of commentators. Through natural language processing technology, the model can automatically generate commentary text that matches the pace of the game based on its real-time dynamics, and adjust the tone and style according to different game situations to ensure the vividness and appeal of the commentary. Furthermore, the model can also adjust the commentary content in real time based on audience feedback and interaction, making it more tailored to the needs and emotions of the audience.

[0051] For athlete performance evaluation, this invention provides comprehensive performance analysis based on multiple data sources, such as video playback and physiological data. Through detailed analysis of athletes' technical movements, the model can identify athletes' strengths and weaknesses, helping coaches develop personalized training plans to further improve athletes' competitive levels. Furthermore, by comparing the performance of other athletes, the model can identify areas for improvement in athletes' techniques and tactics, thereby promoting their overall development.

[0052] In conclusion, the multi-stage training strategy of this invention has significant practical application value and broad prospects in the sports field, and can effectively promote the development and innovation of related industries. This strategy not only provides scientific decision-making support for teams and athletes, but also brings a richer and more immersive viewing experience to audiences, possessing broad market application potential.

[0053] This invention also provides a motion video understanding device based on a multimodal large model, which is used to implement the above-described method embodiments; details already described will not be repeated. The terms "module," "unit," and "subunit," etc., used below refer to combinations of software and / or hardware that perform predetermined functions. Although the device described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.

[0054] like Figure 4 As shown, Figure 4 This is a structural block diagram of a motion video understanding device based on a multimodal large model according to an embodiment of the present invention. The device includes: The visual feature extraction module 101 is used to uniformly sample the input motion video, extract K video frames, and use a visual encoder to extract the visual features of each video frame. The visual feature mapping module 102 is used to map visual features to the latent space dimension of a large language model through a multilayer perceptron, thereby generating mapped visual features. The text generation module 103 is used to combine the mapped visual features with the user query to form a comprehensive prompt, which is then input into a large language model to generate a text response. The multi-stage training module 104 is used to train multimodal large models using a multi-stage training strategy, including visual language alignment pre-training, video instruction follow-up fine-tuning training, and sports knowledge fine-tuning training.

[0055] The beneficial effects of the motion video understanding device based on multimodal large model in this invention are the same as those of the motion video understanding method based on multimodal large model described above, so they will not be repeated here.

[0056] Figure 5 A structural block diagram of the electronic device provided in the embodiments of the present invention, such as... Figure 5 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following methods: The input motion video is uniformly sampled to extract K video frames, and the visual features of each video frame are extracted using a visual encoder. Visual features are mapped to the latent space dimension of a large language model using a multilayer perceptron, generating mapped visual features. The mapped visual features are combined with the user query to form a comprehensive prompt, which is then input into a large language model to generate a text response. A multi-stage training strategy is employed to train the multimodal large model, including visual language alignment pre-training, video instruction follow-up fine-tuning training, and sports knowledge fine-tuning training.

[0057] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motion video understanding method based on a multimodal large model, characterized in that, The motion video understanding method based on a multimodal large model includes: (1) The input motion video is uniformly sampled to extract K video frames, and the visual features of each video frame are extracted using a visual encoder; (2) Visual features are mapped to the latent space dimension of the large language model through a multilayer perceptron to generate mapped visual features; (3) The mapped visual features are combined with the user query to form a comprehensive prompt, which is then input into the large language model to generate a text response; (4) A multi-stage training strategy is adopted to train the multimodal large model, including visual language alignment pre-training, video instruction follow-up fine-tuning training and sports knowledge fine-tuning training.

2. The motion video understanding method based on a multimodal large model according to claim 1, characterized in that, The number of frames for uniform sampling in step (1) is N / K, where N is the total number of frames in the input video, K is the number of sampling frames, and K is 16.

3. The motion video understanding method based on a multimodal large model according to claim 1, characterized in that, The visual encoder mentioned in step (1) is the CLIP ViT-L / 14 model, whose input image resolution is 224×224, divided into 14×14 pixel regions, and extracts 1024-dimensional visual features through 24 Transformer encoding layers.

4. The motion video understanding method based on a multimodal large model according to claim 1, characterized in that, The multilayer perceptron described in step (2) comprises L linear layers, the previous one being... The layers are followed by ReLU activation functions, and the output of the last layer is mapped to a 4096-dimensional latent space feature, where L=2 and the intermediate dimension is 2048.

5. The motion video understanding method based on a multimodal large model according to claim 1, characterized in that, The large language model mentioned in step (3) is the Vicuna 3B parameter model, whose input prompt text contains image token placeholders. Each placeholder is replaced with 196 visual features and is encoded in relative position by rotating the embedding vector.

6. The motion video understanding method based on a multimodal large model according to claim 5, characterized in that, The rotation embedding vector is implemented using the following formula: ； in, This represents the embedding vector generated by rotating the embedding vector; Discrete tokens for text. This indicates the absolute index of the current token in the sequence; The initial embedding vector for the input token; This represents the rotation matrix applied to the i-th pair of adjacent dimensions, generated by trigonometric functions, and used to calculate the attention score in the self-attention mechanism.

7. The motion video understanding method based on a multimodal large model according to claim 6, characterized in that, The rotation matrix is ​​defined as follows: ； in, Indicates the current token's position index. This indicates the embedded dimension index.

8. A motion video understanding device based on a multimodal large model, characterized in that, include: The visual feature extraction module is used to uniformly sample the input motion video, extract K video frames, and use a visual encoder to extract the visual features of each video frame. The visual feature mapping module is used to map visual features to the latent space dimension of a large language model through a multilayer perceptron, generating mapped visual features. The text generation module is used to combine the mapped visual features with the user query to form a comprehensive prompt, which is then input into a large language model to generate a text response. The multi-stage training module is used to train multimodal large models using a multi-stage training strategy, including visual language alignment pre-training, video instruction follow-up fine-tuning training, and sports knowledge fine-tuning training.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the motion video understanding method based on a multimodal large model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the motion video understanding method based on a multimodal large model as described in any one of claims 1 to 7.

Images