Long-term action prediction method based on multi-modal large language model and related device

Abstract

The invention provides a long-term action prediction method based on a multi-modal large language model and a related device, and the method comprises the steps: carrying out the frame extraction of a video according to a fixed sampling rate, and predicting text action labels corresponding to all video frames; inputting the text action tag into a multi-modal large language model to obtain a text feature, a visual feature and a preset action query feature; constructing a cross-modal interaction network architecture, and taking the text features, the visual features and preset action query features as input to obtain feature expressions after interaction of different modals; and obtaining an output result of action tuning based on the feature expression after interaction of different modalities and an action tuning module, then training a multi-modal large language model, and performing optimization solution on the network by using a loss function to obtain a prediction result.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer vision, and more specifically to a long-term action prediction method based on a multimodal large language model and related devices. Background Art

[0002] Long-term action prediction plays a vital role in promoting technological progress in many fields such as augmented reality, virtual reality, intelligent monitoring systems, and human-computer interaction. However, this technology faces significant challenges because it requires the model to understand and predict action sequences over a long period of time, that is, to capture and remember the long-term dependencies between long-distance actions, which undoubtedly increases the complexity of technical implementation. Large Language Models (LLMs), with their excellent sequence modeling capabilities and rich common sense knowledge, provide a solid foundation for building models that can understand and predict long-distance action relationships and deeply understand the intrinsic semantic connections of actions, and have attracted widespread attention in the field of action prediction.

[0003] In recent years, long short-term memory networks and recurrent neural networks have made some progress in long-term action prediction tasks. However, these methods still face challenges in dealing with long-term dependencies, and their performance is unsatisfactory in terms of computational efficiency and generalization ability in new scenarios. To overcome these limitations, researchers began to adopt the Transformer architecture to improve long-term action prediction, an innovation that significantly improved prediction performance. However, as the action sequence continues to lengthen, the Transformer-based method faces the problem of a sharp increase in the number of parameters, which greatly affects the efficiency of the model. In addition, the high sensitivity of the Transformer-based method to the input data may also lead to unstable performance. Summary of the invention

[0004] The present invention provides a long-term action prediction method and related devices based on a multimodal large language model to solve the above problems.

[0005] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a long-term action prediction method based on a multimodal large language model, comprising: The video is processed by sampling frames at a fixed sampling rate, and the text action labels corresponding to all video frames are predicted; Input the text action label into the multimodal large language model to obtain text features, visual features and preset action query features; Construct a cross-modal interactive network architecture, take text features, visual features, and preset action query features as input, and obtain feature expressions after different modal interactions; Based on the feature expression after the interaction of different modalities, the output result of action tuning is obtained based on the action tuning module, and then the multimodal large language model is trained, and the loss function is used to optimize the network to obtain the prediction result.

[0006] Furthermore, the video is processed by sampling frames at a fixed sampling rate to predict the text action labels corresponding to all video frames, including: Each video is sampled at a fixed sampling rate, and the image classification model ViT-L and the image classification model ResNet-50 are fine-tuned using the video frame data to obtain two frame classification models. All frames are then used as inputs to the classification model to predict the text labels corresponding to all video frames.

[0007] Furthermore, the text action label is input into the multimodal large language model to obtain text features, visual features and preset action query features, including: The obtained text action labels are sent to the Tokenizer of the large language model for processing to generate text tokens; the action text information is converted to a high-dimensional vector space through the frozen Token Embedding layer; since a single action text label may correspond to multiple tokens, these embedding vectors are averaged to form a single text aggregate embedding vector that matches the number of visual features; the video is analyzed using the I3D feature extractor to obtain visual features; the visual features and predefined action queries are mapped to the same vector space through the feature adaptation module; and the multimodal downward mapping layer is used to obtain , and Adapt it to the input requirements of the cross-modal interaction network architecture.

[0008] Furthermore, a cross-modal interaction network architecture is constructed, which takes text features, visual features, and preset action query features as input to obtain feature expressions after different modal interactions, including: A cross-modal interaction network architecture is constructed, and the text, vision and preset action query features are used as the input of the architecture to obtain the feature expression after different modal interactions; the cross-modal interaction attention module explores the unique feature expression of different modalities by introducing the self-attention mechanism, and promotes feature fusion by using the cross-self-attention mechanism; the output of the cross-modal interaction attention module is normalized using RMS Norm, and then further feature extraction is performed through the feedforward neural network layer; the processed text, vision and action query features are obtained through the multimodal upward mapping layer , , and To adapt to the input requirements of large language models.

[0009] Furthermore, based on the feature expression after interaction of different modalities, the output results of action tuning are obtained based on the action tuning module, including: Will , , and Splice in order ;according to The length of the rotation position encoding ROPE is calculated to obtain relative position information and introduce a mask at the same time; the action tuning module is inserted into the large language model transformer structure to maintain a trainable state; input After being processed by RMSNorm, it is transposed and first enters the one-dimensional convolution layer of the motion tuning module. Then, it is reduced by the Dropout layer to reduce overfitting. Then, it is processed by another convolution layer and a residual connection is added. Finally, it is transposed again to restore the original dimensional order to obtain the output result of the motion tuning.

[0010] Furthermore, the loss function is: Constructing a segmentation loss consisting of two actions and , action classification loss and duration loss The long-term action prediction loss .

[0011] Further, specific: Construct a five-fold cross-validation 50 Salads dataset and a four-fold cross-validation Breakfast dataset, and report the average results of all folds of each dataset; construct a segmentation loss consisting of two actions and , action classification loss and duration loss The long-term action prediction loss ; Set the training rounds to 20; the learning rate of the Breakfast dataset is set to 1e-4, and the learning rate of the 50 Salads dataset is set to 1e-3; observation ratio Adjusted to 20%, 30%, and 50% in the breakfast dataset, and adjusted to 10%, 20%, 30%, 40%, and 50% in the 50 Salads dataset; predicted ratio It is set to 50%, and different prediction ratios are obtained from 50% of the prediction results during testing; the sampling rate is set to 6 in the Breakfast dataset and to 8 in the 50 Salads dataset; the batch size is set to 1, and the Adam optimizer is used to update the network parameters and save the model weights; during the test, the selected model weights are loaded, the video I3D visual features and text features are input into the network, and the future action categories and durations under specific observation ratios and prediction ratios are obtained to complete long-term action prediction.

[0012] In a second aspect, the present invention provides a long-term action prediction system based on a multimodal large language model, comprising: The data acquisition module is used to extract frames from the video according to a fixed sampling rate and predict the text action labels corresponding to all video frames; A feature extraction module is used to input text action labels into a multimodal large language model to obtain text features, visual features, and preset action query features; The feature expression module is used to build a cross-modal interaction network architecture, taking text features, visual features, and preset action query features as input to obtain feature expressions after interaction of different modalities; The training output module is used to obtain the output result of action tuning based on the feature expression after interaction of different modalities based on the action tuning module, and then train the multimodal large language model, and use the loss function to optimize and solve the network to obtain the prediction result.

[0013] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the long-term action prediction method based on a multimodal large language model when executing the computer program.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the long-term action prediction method based on a multimodal large language model are implemented.

[0015] Compared with the prior art, the present invention has the following technical effects: The present invention uses a cross-modal interaction module to deeply explore the unique features of different modalities and effectively capture the interaction between vision and text, thereby enhancing the model's ability to integrate and optimize multimodal information. This cross-modal interaction mechanism not only promotes the deep fusion of visual and text information, but also enables the model to better understand and predict long-term action sequences in videos.

[0016] In addition, the present invention adopts an innovative ActionLLM architecture, which treats video sequences as continuous labeled sequences and cleverly uses large language models (LLMs) to predict future actions. In this way, the present invention successfully introduces advanced technologies in the field of natural language processing into video action prediction, achieving a deeper understanding and analysis of video content.

[0017] In the ActionLLM architecture, by setting future markers, integrating the action tuning module, and simplifying the text decoding layer into a linear layer, the present invention achieves the use of LLM to solve action prediction tasks without relying on complex instructions or redundant descriptions. This design not only improves the prediction efficiency of the model, but also significantly improves the prediction accuracy. At the same time, by introducing rotational position encoding (ROPE) and masking mechanisms, the present invention further enhances the model's ability to process temporal information, allowing the model to more accurately capture the action changes and temporal relationships in the video.

[0018] In terms of experimental verification, the present invention constructs a 50 Salads dataset with five-fold cross-validation and a Breakfast dataset with four-fold cross-validation, and comprehensively reports the average results of all folds of each dataset. The experimental results show that the method of the present invention has achieved excellent performance on multiple datasets, fully verifying its effectiveness and generalization ability.

[0019] In summary, the present invention not only improves the accuracy and efficiency of video action prediction through the innovative design of the cross-modal interaction module and the ActionLLM architecture, but also provides a new idea and method for the field of video understanding and analysis. The proposal of this method is of great significance for promoting the development and application of video processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the present invention; Figure 2 It is a logic block diagram of the present invention; Figure 3 Schematic diagram of fine-tuning strategy. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with the accompanying drawings: Example 1, please refer to Figure 1 , a long-term action prediction method based on a multimodal large language model, including: The video is processed by sampling frames at a fixed sampling rate, and the text action labels corresponding to all video frames are predicted; Input the text action label into the multimodal large language model to obtain text features, visual features and preset action query features; Construct a cross-modal interactive network architecture, take text features, visual features, and preset action query features as input, and obtain feature expressions after different modal interactions; Based on the feature expression after the interaction of different modalities, the output result of action tuning is obtained based on the action tuning module, and then the multimodal large language model is trained, and the loss function is used to optimize the network to obtain the prediction result.

[0022] Example 2, please refer to Figure 2 In order to alleviate the problems faced by existing long-term action prediction technologies in dealing with complex scenes and long sequences of actions, such as low accuracy, low computational efficiency, and high sensitivity to input data, the present invention proposes a long-term action prediction method based on a multimodal large language model. This method regards the video sequence as a series of tags and integrates visual and textual information, using a large language model to capture dynamic changes in time and deeply analyze the intrinsic meaning of the action. By innovatively introducing the Cross-Modality Interaction Block, the present invention optimizes the fusion process of different modal information and improves the ability of the large language model in action prediction. In addition, the present invention uses an action adjustment module to fine-tune the LLM specifically for the long-term action prediction task while maintaining the common sense knowledge of the model. The combination of these technologies constitutes an efficient network architecture that can use multimodal LLM for long-term action prediction, significantly improving the accuracy of long-term action prediction.

[0023] A long-term action prediction method based on a multimodal large language model, the steps are as follows: Step 1: Build input: extract frames for each video at a fixed sampling rate; use video frame data to fine-tune VIT and ResNet 50 to obtain two frame classification models, and then use all frames (training + testing) as input to the classification model to predict the text labels corresponding to all video frames. This process will introduce some noise into the training action labels to improve the robustness of the model; Step 2: The obtained text action labels are sent to the Tokenizer of the large language model for processing to generate text tokens; the action text information is converted to a high-dimensional vector space through the frozen Token Embedding layer; since a single action text label may correspond to multiple tokens, these embedding vectors are averaged to form a single text aggregate embedding vector that matches the number of visual features; the video is analyzed using the I3D feature extractor to obtain visual features; to ensure the consistency of features throughout the analysis process, the visual features and the predefined action queries will be mapped to the same vector space through the feature adaptation module; at the same time, the multimodal downward mapping layer is used to obtain , and Adapt it to the input requirements of the cross-modal interaction network architecture; Step 3: Construct a cross-modal interaction network architecture, and use the text, vision, and preset action query features processed by step 2 as the input of the architecture to obtain the feature expression after different modal interactions; the cross-modal interaction attention module is the core component of the cross-modal interaction network architecture. It explores the unique feature expression of different modalities by introducing the self-attention mechanism, and promotes feature fusion by using the cross-self-attention mechanism; the output of the cross-modal interaction attention module is normalized using RMS Norm, and then further feature extraction is performed through the feedforward neural network layer; the processed text, vision, and action query features are obtained through the multi-modal upward mapping layer , , and To adapt to the input requirements of large language models; Step 4: , , and Splice in order ;according to The length of the rotation position encoding ROPE is calculated to obtain the relative position information and introduce the mask at the same time; the action tuning module is inserted into the appropriate position of the large language model transformer structure to maintain the trainable state; input After being processed by RMS Norm, it is transposed and first enters the one-dimensional convolution layer of the motion tuning module. Then, it is processed by the Dropout layer to reduce overfitting. Then, it is processed by another convolution layer and residual connections are added. Finally, it is transposed again to restore the original dimensional order to obtain the output result of motion tuning. Step 5: Remove the complex regression-based text decoding layer in the large language model and replace it with a simple linear layer. Extract the textual and visual corresponding position features from the output of the frozen large language model, and then guide these features through the text action segmentation layer and the visual action segmentation layer respectively. As for the features corresponding to the action query, they will pass through the action classification layer and the duration regression layer respectively to predict the future action category and its corresponding duration.

[0024] Step 6: Construct a five-fold cross-validation 50 Salads dataset and a four-fold cross-validation Breakfast dataset, and report the average results of all folds of each dataset; construct a segmentation loss consisting of two actions and , action classification loss and duration loss The long-term action prediction loss ; Set the training rounds to 20; the learning rate of the Breakfast dataset is set to 1e-4, and the learning rate of the 50 Salads dataset is set to 1e-3; observation ratio In the breakfast dataset, it is adjusted to 20%, 30%, and 50%, and in the 50 Salads dataset, it is adjusted to 10%, 20%, 30%, 40%, and 50%. The sampling rate is set to 6 in the Breakfast dataset and 8 in the 50 Salads dataset. The batch size is set to 1, and the Adam optimizer is used to update the network parameters and save the model weights. During the test, the selected model weights are loaded, and the video I3D visual features and text features are input into the network to obtain the future action categories and durations under specific observation ratios and prediction ratios, completing the long-term action prediction task.

[0025] Embodiment 3, the present invention provides a long-term motion prediction method based on a multimodal large model, such as Figure 2 As shown, the specific steps of the method include: Step 1, obtain a long-term action prediction video dataset; for the 50 salads dataset, there are 50 salad-making related videos in total, and in each fold of data division, the training includes 40 videos and the test includes 10 videos; for the Breakfast dataset, there are 1712 breakfast-making related videos in total, and a four-fold validation estimation is used, the first fold training includes 1460 videos and the test includes 252 videos; the second fold training includes 1261 videos and the test includes 451 videos; the third fold training includes 1279 videos and the test includes 433 videos; the fourth fold training includes 1136 videos and the test includes 576 videos; Step 2: For each fold of video data in each data set in step 1, after extracting frames, a vit classification model or a resnet50 classification model needs to be trained. The classification model is selected according to the accuracy of the final classification result on each fold of the test set. A total of 9 classification models are obtained to predict the text action of the past observation part of the video; Step 3: Feature processing. First, the sampling rate Sampling past text and visual signals to obtain and . It is based on the preset action query length N and the dimension of the visual feature Sure. and This is processed using the following formula: (1) (2) In the formula , are the weights of the linear layers involved in the feature adapter, and is their bias term. , , Represent the dimension sizes of the input, intermediate mapping, and output of the feature adapter respectively. Represents the multimodal down-projection layer.

[0026] At the same time, the text action tag It is processed by LLM's default tokenizer and decomposed into tokens. Then it is converted into text features through token embedding. .

[0027] Step 4: Build the network architecture. First, design the cross-modal interaction module. Figure 2 (b) The internal structure of the cross-modal attention in the cross-modal interaction module is described in detail. In order to enable the cross-modal interaction attention to focus on the representative features inherent in each modality, the present invention applies the self-attention mechanism on each modal dimension. , and , the processing steps are as follows: (3) (4) (5) in, Indicates , As the input attention mechanism, Indicates heads. H is the number of heads, represents the softmax operation, and D represents the input dimension. At the same time, the present invention uses the cross attention mechanism to promote feature fusion, and the processing process is as follows: (6) (7) (8) In the formula, , , Represent the output of CMIA’s text, visual, and query modalities respectively. In the cross-modal interaction module, the input feature , , These fused vectors are then normalized using RMSNorm to produce the output , , ,These outputs are further refined through residual connections and additional RMSNorm applications. Finally, a feed-forward network followed by another residual connection generates the results of the cross-modal interaction module.

[0028] Step 5: The result obtained in step 4 is obtained by passing the multimodal upward mapping layer , , and , and then concatenate and residually connect them to obtain the input of the action tuning module . The processing is as follows: (9) (10) (11) In the formula , are the weights of the convolutional layers involved in the feature adapter. and are their corresponding bias terms. Represents the intermediate dimension of action tuning. The output of the action tuning module will be used as the input of the frozen large language model layer structure, and the action tuning module will be interspersed in the entire frozen large language model, allowing the large model to adapt to long-term action prediction while retaining inherent common sense knowledge. Finally, the output of different positions is passed through different classifiers and predictors to obtain the final prediction result.

[0029] Step 6: After the network is built in step 5, train the long-term action prediction model based on the multimodal large language model: use the loss function composed of four loss components to optimize the entire network, set the hyperparameters: training rounds are 20, the initial learning rate is 0.0001, and the batch size is 1. Use the Adam optimizer to update the network parameters so that the loss function converges to the minimum and complete the model training.

[0030] Step 5: Use the test data set in step 1 to select the model weight with the best prediction performance in step 6 and save it, and report the average prediction results of all folds.

[0031] The effects of the present invention are further described below in conjunction with simulation experiments.

[0032] 1. Simulation experiment conditions: The hardware platform of the invented simulation experiment: the graphics card used is a VGA compatible controller produced by NVIDIA Corporation, specifically model 3090 Ti, connected to the PCI Express x16 interface, the video memory size of the graphics card is 24GB, and the CUDA version is 11.6.

[0033] Software platform: Win10 operating system, Pytorch 1.12.1 The training and test sample sets used in the simulation experiment of the present invention are from the Breakfast dataset and the 50 Salads dataset. Breakfast consists of 1712 videos captured at a frame rate of 15 frames per second and downsampled to a resolution of 240×320 pixels. A four-fold validation estimation method is used, and the number of samples for training and testing in each fold is different. 50Salads consists of 50 overhead videos, with a total of more than 4 hours of video material, recorded in RGB-D format, with a resolution of 640×480 pixels and a frame rate of 30 frames per second. A five-fold validation estimation method is used, with 10 videos used for training and 40 videos used for testing in each fold.

[0034] 2. Simulation content and results analysis: The simulation experiment of the present invention adopts the network trained by the present invention and eight existing technologies (Tem-Agg method, Cycle Cons method, A-ACT method, J-AAN method, ObjectPrompt method, FUTR method, Con-Alig method, CNN method), and performs long-term action prediction on the input test sample set to obtain action prediction results under different observation ratios and prediction ratios.

[0035] In order to verify the simulation experiment effect of the present invention, the present invention selected two data sets and comprehensively compared the present invention with eight existing technologies. The present invention uses Mean over Classes Accuracy (MoC) as the evaluation index, which measures the average accuracy of all future time points within a specified prediction duration. A larger value of the MoC index indicates that the method has higher accuracy in long-term video action prediction. The simulation experiment used eight methods of the present invention and its existing advanced methods respectively: The Tem-Agg method of the prior art refers to the long-term action prediction method based on temporal aggregate representation adopted in (Sener F, Singhania D, Yao A. Temporal aggregate representations for long-range video understanding[C] / / Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16. Springer International Publishing, 2020:154-171.) published by Sener et al.

[0036] The Cycle Cons method in the prior art refers to the long-term action prediction method based on cycle consistency adopted in Farha et al. (Abu Farha Y, Ke Q, Schiele B, et al. Long-term anticipation of activities with cycle consistency[C] / / Pattern Recognition: 42nd DAGM German Conference, DAGM GCPR 2020, Tübingen, Germany, September 28–October 1, 2020, Proceedings 42. SpringerInternational Publishing, 2021: 159-173.).

[0037] The A-ACT method in the prior art refers to the long-term action prediction method based on cycle transformation adopted in (Gupta A, Liu J, Bo L, et al. A-act: Action anticipation through cycle transformations[J]. arXiv preprintarXiv:2204.00942, 2022.) published by Gupta et al.

[0038] The J-AAN method of the prior art refers to the long-term action prediction method based on self-knowledge distillation and cycle consistency adopted in (Moniruzzaman M, Yin Z, He Z, et al. Jointly-Learnt Networks for Future Action Anticipation via Self-Knowledge Distillation and Cycle Consistency[J]. IEEE Transactions onCircuits and Systems for Video Technology, 2022, 33(7): 3243-3256.) published by Moniruzzaman et al.

[0039] The ObjectPrompt method of the prior art refers to the long-term action prediction method based on object-centric video representation adopted in (Zhang C, Fu C, Wang S,et al. Object-centric video representation for long-term action anticipation[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2024: 6751-6761.) published by Zhang et al.

[0040] The FUTR method of the prior art refers to the long-term action prediction method based on the future transformer adopted in (Gong D, Lee J, Kim M, et al. Future transformer for long-term action anticipation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 3052-3061.) published by Gong et al.

[0041] The Con-Alig method of the prior art refers to the long-term action prediction method based on contextual alignment adopted in (Patsch C, Zhang J, Wu Y,et al. Long-Term Action Anticipation Based on Contextual Alignment[C] / / ICASSP2024-2024 IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP). IEEE, 2024: 5920-5924.) published by Patsch et al.

[0042] The CNN method of the prior art refers to the long-term action prediction method based on CNN and RNN adopted in Farha et al. (Abu Farha Y, Richard A, Gall J. When will you do what? -anticipating temporal occurrences of activities [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 5343-5352.).

[0043] The performance of the present invention is compared with that of eight existing technologies on the 50 Salads dataset. The comparison results based on the evaluation index MoC are shown in Table 1:

[0044] The performance of the present invention is compared with that of three prior arts on the Breakfast dataset. The comparison results based on the evaluation index MoC are shown in Table 2:

[0045] As can be seen from Table 1, the average value of the motion prediction results of the present invention on the 50 Salads dataset is higher than that of the eight methods in the prior art. As can be seen from Table 2, the average value of the motion prediction results of the present invention on the Breakfast dataset is higher than that of the three methods in the prior art. This shows that the present invention is more accurate in predicting long-term motion in videos.

[0046] In yet another embodiment of the present invention, a long-term action prediction system based on a multimodal large language model is provided, which can be used to implement the above-mentioned long-term action prediction method based on a multimodal large language model. Specifically, the system includes: The data acquisition module is used to extract frames from the video according to a fixed sampling rate and predict the text action labels corresponding to all video frames; A feature extraction module is used to input text action labels into a multimodal large language model to obtain text features, visual features, and preset action query features; The feature expression module is used to build a cross-modal interaction network architecture, taking text features, visual features, and preset action query features as input to obtain feature expressions after interaction of different modalities; The training output module is used to obtain the output result of action tuning based on the feature expression after interaction of different modalities based on the action tuning module, and then train the multimodal large language model, and use the loss function to optimize and solve the network to obtain the prediction result.

[0047] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0048] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the long-term action prediction method based on the multimodal large language model.

[0049] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the long-term action prediction method based on a multimodal large language model in the above embodiment.

[0050] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0052] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0053] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A long-term action prediction method based on a multimodal large language model, characterized in that: include: The video is processed by sampling frames at a fixed sampling rate, and the text action labels corresponding to all video frames are predicted; Input the text action label into the multimodal large language model to obtain text features, visual features and preset action query features; Construct a cross-modal interactive network architecture, take text features, visual features, and preset action query features as input, and obtain feature expressions after different modal interactions; Based on the feature expression after the interaction of different modalities, the output result of action tuning is obtained based on the action tuning module, and then the multimodal large language model is trained, and the loss function is used to optimize the network to obtain the prediction result.

2. The long-term action prediction method based on a multimodal large language model according to claim 1, characterized in that: The video is processed by sampling frames at a fixed sampling rate to predict the text action labels corresponding to all video frames, including: Each video is sampled at a fixed sampling rate, and the image classification model ViT-L and the image classification model ResNet-50 are fine-tuned using the video frame data to obtain two frame classification models. All frames are then used as inputs to the classification model to predict the text labels corresponding to all video frames.

3. The long-term action prediction method based on a multimodal large language model according to claim 1, characterized in that: The text action label is input into the multimodal large language model to obtain text features, visual features and preset action query features, including: The obtained text action labels are sent to the Tokenizer of the large language model for processing to generate text tokens; the action text information is converted to a high-dimensional vector space through the frozen Token Embedding layer; since a single action text label may correspond to multiple tokens, these embedding vectors are averaged to form a single text aggregate embedding vector that matches the number of visual features; the video is analyzed using the I3D feature extractor to obtain visual features; the visual features and predefined action queries are mapped to the same vector space through the feature adaptation module; and the multimodal downward mapping layer is used to obtain , and Adapt it to the input requirements of the cross-modal interaction network architecture.

4. The long-term action prediction method based on a multimodal large language model according to claim 1, characterized in that: Construct a cross-modal interaction network architecture, take text features, visual features, and preset action query features as input, and obtain feature expressions after different modal interactions, including: A cross-modal interaction network architecture is constructed, and the text, vision and preset action query features are used as the input of the architecture to obtain the feature expression after different modal interactions; the cross-modal interaction attention module explores the unique feature expression of different modalities by introducing the self-attention mechanism, and promotes feature fusion by using the cross-self-attention mechanism; the output of the cross-modal interaction attention module is normalized using RMS Norm, and then further feature extraction is performed through the feedforward neural network layer; the processed text, vision and action query features are obtained through the multimodal upward mapping layer , , and To adapt to the input requirements of large language models.

5. The long-term action prediction method based on a multimodal large language model according to claim 4, characterized in that: Based on the feature expression after interaction of different modalities, the output results of action tuning are obtained based on the action tuning module, including: Will , , and Splice in order ;according to The length of the rotation position encoding ROPE is calculated to obtain relative position information and introduce a mask at the same time; the action tuning module is inserted into the large language model transformer structure to maintain a trainable state; input After being processed by RMSNorm, it is transposed and first enters the one-dimensional convolution layer of the motion tuning module. Then, it is reduced by the Dropout layer to reduce overfitting. Then, it is processed by another convolution layer and a residual connection is added. Finally, it is transposed again to restore the original dimensional order to obtain the output result of the motion tuning.

6. The long-term action prediction method based on a multimodal large language model according to claim 1, characterized in that: The loss function is: Constructing a segmentation loss consisting of two actions and , action classification loss and duration loss The long-term action prediction loss .

7. The long-term action prediction method based on a multimodal large language model according to claim 6, characterized in that: Specific: Construct a five-fold cross-validation 50 Salads dataset and a four-fold cross-validation Breakfast dataset, and report the average results of all folds of each dataset; construct a segmentation loss consisting of two actions and , action classification loss and duration loss The long-term action prediction loss ; Set the training rounds to 20; the learning rate of the Breakfast dataset is set to 1e-4, and the learning rate of the 50 Salads dataset is set to 1e-3; observation ratio Adjusted to 20%, 30%, and 50% in the breakfast dataset, and adjusted to 10%, 20%, 30%, 40%, and 50% in the 50 Salads dataset; predicted ratio It is set to 50%, and different prediction ratios are obtained from 50% of the prediction results during testing; the sampling rate is set to 6 in the Breakfast dataset and to 8 in the 50 Salads dataset; the batch size is set to 1, and the Adam optimizer is used to update the network parameters and save the model weights; during the test, the selected model weights are loaded, the video I3D visual features and text features are input into the network, and the future action categories and durations under specific observation ratios and prediction ratios are obtained to complete long-term action prediction.

8. A long-term action prediction system based on a multimodal large language model, characterized in that: include: The data acquisition module is used to extract frames from the video according to a fixed sampling rate and predict the text action labels corresponding to all video frames; A feature extraction module is used to input text action labels into a multimodal large language model to obtain text features, visual features, and preset action query features; The feature expression module is used to build a cross-modal interaction network architecture, taking text features, visual features, and preset action query features as input to obtain feature expressions after different modal interactions; The training output module is used to obtain the output result of action tuning based on the feature expression after interaction of different modalities based on the action tuning module, and then train the multimodal large language model, and use the loss function to optimize and solve the network to obtain the prediction result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the long-term action prediction method based on a multimodal large language model are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the long-term action prediction method based on a multimodal large language model as claimed in any one of claims 1 to 7 are implemented.

Images