Long-term behavior prediction method based on multi-modal large model

Abstract

This invention provides a long-term behavior prediction method based on a multimodal large model, comprising: extracting and mapping features from observed videos to obtain visual alignment features; combining text embedding representations obtained from text instruction conversion with visual alignment features for autoregressive decoding to predict at least one temporally representative future behavior within the prediction range; generating a future visual latent representation corresponding to each predicted temporally representative future behavior; aggregating multiple visual latent vectors contained in each generated future visual latent representation to obtain a visual summary representation; constructing a visually guided attention bias mechanism based on the correlation between the current decoded hidden state and each visual summary representation, thereby constraining the subsequently generated future behaviors, and finally outputting a complete sequence of future behaviors. This invention effectively improves the accuracy of long-term behavior prediction.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a long-term behavior prediction method based on a multimodal large model. Background Technology

[0002] With the development of computer vision and artificial intelligence technologies, long-term behavior prediction based on first-person perspective video has received widespread attention in fields such as robot collaboration, human-computer interaction, and autonomous driving. Existing technologies can be broadly classified into two categories: end-to-end temporal modeling methods and two-stage methods based on fundamental models.

[0003] End-to-end temporal modeling methods directly extract visual features from input videos and predict future behaviors through temporal modeling. Typical methods include recurrent neural network-based structures (such as LSTM and RNN) and Transformer-based architectures. These methods can effectively capture temporal dependencies and model the transition patterns between behaviors and object-action interactions. However, such methods typically rely solely on visual feature learning and temporal pattern fitting, lacking the ability to model high-level semantic information and common-sense knowledge. This makes it difficult to guarantee the semantic consistency of prediction results and can easily produce predictions that do not conform to actual logic in complex scenarios.

[0004] With the development of large language models, some studies have introduced base models (such as large language models and multimodal large models) to leverage their powerful semantic understanding and reasoning capabilities. These methods typically convert videos into intermediate representations (such as behavior labels or text descriptions of the video) through external modules (such as behavior recognition models or visual language models), and then input these intermediate representations into the base model for future behavior prediction. Although these methods can utilize the prior knowledge of the base model, they still suffer from the following problems: First, reliance on external modules complicates the overall system structure and introduces additional computational overhead; second, intermediate representations (such as discrete behavior labels or text descriptions) inevitably suffer information loss during video information compression, making it difficult to retain fine-grained visual details; third, if an external module makes a recognition error, this error will be passed as input to subsequent reasoning stages, leading to error accumulation and affecting the prediction results; finally, during the prediction process, the base model only reasons based on symbolic input or prior knowledge, lacking continuous utilization of original visual evidence, resulting in insufficient visual constraints and limited visual consistency of the prediction results.

[0005] Therefore, there is an urgent need for a behavior prediction method that can leverage the powerful reasoning capabilities of the base model within an end-to-end framework, directly infer from videos, and continuously introduce visual information constraints during the prediction process. Summary of the Invention

[0006] This invention provides a long-term behavior prediction method based on a multimodal large model to address the shortcomings of existing technologies that emphasize semantics while neglecting visual aspects and lack sufficient visual constraints, resulting in inaccurate long-term behavior prediction.

[0007] A long-term behavior prediction method based on a multimodal large model includes: Acquire observation videos and text instructions; the text instructions are used to define the model's task objective, prediction range, and output format. Feature extraction is performed on the observed video to obtain an initial visual latent representation; The initial visual latent representation is then subjected to feature mapping to obtain visual alignment features; The text instructions are converted into text embedding representations, and the visual alignment features and the text embedding representations are combined into a multimodal input sequence; Autoregressive decoding is performed based on the multimodal input sequence to generate text inference fragments; the text inference fragments include scene summaries, descriptions of observed behaviors, and inferences of behavioral intentions from the observed video. Based on the text reasoning fragment, at least one temporally representative future behavior within the prediction range is predicted; Whenever a representative future behavior is predicted, a future visual latent representation corresponding to that representative future behavior is generated based on the current decoded hidden state, the visual alignment features, and the current historical visual latent representation; the future visual latent representation includes multiple visual latent vectors. For each of the generated future visual latent representations, the multiple visual latent vectors contained therein are aggregated to obtain the corresponding visual summary representation; A visual guidance attention bias mechanism is constructed based on the current visual summary representation, and the self-attention score in the subsequent autoregressive decoding process is modulated accordingly to constrain the generated future behavior and finally output a complete sequence of future behavior.

[0008] Furthermore, according to the long-term behavior prediction method based on a multimodal large model as described above, predicting at least one temporally representative future behavior within the prediction range based on the text reasoning fragment includes: Based on the prediction range, determine the future behavior sequence to be predicted; The future behavior sequence is divided into several consecutive behavior groups according to the time order; The first behavior in each behavior group is selected as the temporally representative future behavior of that group.

[0009] Furthermore, according to the long-term behavior prediction method based on a multimodal large model as described above, the step of generating a future visual latent representation corresponding to the representative future behavior of the time series based on the current decoded hidden state, the visual alignment features, and the current historical visual latent representation includes: A global visual context is constructed based on the visual alignment features and the current historical visual latent representation, and the global visual context is conditionally processed according to the current decoded hidden state. In the process of progressively generating the future visual latent representation, all the visual latent vectors contained in the currently generated future visual latent representation are used as local visual context; Under the constraints of the current decoded hidden state, causal cross-attention computation and causal self-attention computation are performed based on the global visual context and the local visual context to gradually generate the future visual latent representation.

[0010] Furthermore, according to the long-term behavior prediction method based on a multimodal large model as described above, the step of aggregating the multiple visual latent vectors contained in each generated future visual latent representation to obtain the corresponding visual summary representation includes: The multiple visual latent vectors contained in each of the aforementioned future visual latent representations are aggregated to obtain aggregated features; The aggregated features are then subjected to average pooling to obtain the overall visual features; The overall visual features are linearly mapped and then normalized to obtain the visual summary representation.

[0011] Furthermore, according to the long-term behavior prediction method based on a multimodal large model as described above, the step of constructing a visually guided attention bias mechanism based on each current visual summary representation, and modulating the self-attention score in the subsequent autoregressive decoding process accordingly, thereby constraining the subsequently generated future behavior, and finally outputting a complete future behavior sequence includes: Calculate the similarity between the query vector of the current location to be generated and the visual summary representation corresponding to each temporally representative future behavior, and use it as the visual guidance attention bias;

[0012] in, Indicates the first position of the current position i Visual guidance to shift attentional bias This represents the query vector at the current position i. Indicates the first A visual summary representation obtained by aggregating potential future visual representations; Set current position Historical position The original attention score is: in, For historical position The key vector, For the feature dimension, if the first... The set of locations of the future visual latent representations in the decoding context is: Then the modulated attention score for:

[0013] The weighting coefficients for visual attention bias. For indicator functions; The modulated attention score Perform softmax normalization to obtain the current position. For each historical location Attention weights; Using the attention weights for historical positions value vector Perform weighted aggregation to obtain the updated hidden state at the current position; The hidden state is updated based on the current position to constrain the subsequent generated future behaviors, and finally outputs a complete sequence of future behaviors.

[0014] Furthermore, according to the long-term behavior prediction method based on a multimodal large model as described above, the multimodal large model includes: a visual encoder, a connector, and a base model; The visual encoder is used to extract features from the observed video to obtain the initial visual latent representation; The connector is used to map the initial visual latent representation to the embedding space of the autoregressive generated backbone to obtain the visual alignment feature; The base model is used to perform the autoregressive decoding process and generate the text reasoning fragment, temporally representative future behavior, future visual latent representation, and complete future behavior sequence.

[0015] Furthermore, according to the long-term behavior prediction method based on a multimodal large model as described above, the training of the multimodal large model includes: Based on regression language modeling loss function The multimodal large model is initially trained to obtain the initially trained multimodal large model;

[0016] in, The initial visual latent representation and text instructions are provided for the labeled observation videos. This represents the t-th target word generated using the labeled observation video; This represents the target words preceding the t-th target word generated using the labeled observation video; N represents the total number of target word sequences. This indicates that in the input condition X and target word Under the given conditions, the model generates the t-th target word. The probability of; Based on the visual latent representation regression loss function visual semantic alignment loss function The initially trained multimodal large model is then optimized to obtain the final trained multimodal large model.

[0017] Furthermore, according to the long-term behavior prediction method based on a multimodal large model as described above, the visual latent representation regression loss function... for:

[0018] in, For potential representation of future vision, Indicates the number of potential future visual representations. To extract visual latent representations from labeled observation videos.

[0019] Furthermore, based on the long-term behavior prediction method based on a multimodal large model as described above, the visual semantic alignment loss function... for:

[0020] in, The number of potential representations for future vision. For the first A visual summary representation. For the first The future visual potential represents the hidden state output from the current decoding position.

[0021] The long-term behavior prediction method based on a multimodal large model provided in this application generates a future visual latent representation corresponding to each time a representative future behavior is predicted. Multiple visual latent vectors contained in the generated future visual latent are aggregated to obtain a corresponding visual summary representation. Finally, a visual-guided attention bias mechanism is constructed using each current visual summary representation, which directly affects the generation probability of subsequent future behavior sequences. This approach breaks the existing pipeline mode of visual recognition followed by text reasoning, constructing a closed loop of prediction, imagination, and correction. This solves the fundamental defects of existing technologies, such as emphasizing semantics while neglecting visual aspects and insufficient visual constraints. Thus, it achieves direct prediction from raw video to future behavior sequences and effectively improves the accuracy of future behavior sequence prediction.

[0022] In addition, this application utilizes global and local visual contexts, and through causal cross attention and causal self attention, gradually generates a future visual latent representation. This design ensures that the generated future visual latent representation can take into account both global and local features, which is much more refined than directly generating a global vector feature, thereby improving the accuracy and semantic consistency of long-term behavior prediction.

[0023] Furthermore, the method provided in this application calculates the similarity between the visual summary representation corresponding to each temporally representative future behavior and the future visual latent representation when calculating the attention score, and uses this similarity to calculate the modulated attention score. This allows the subsequent generated future behavior to be constrained simply by adjusting the attention weights without modifying the main parameters of the large model, thereby improving the efficiency of long-term behavior prediction.

[0024] In addition, this application does not rely on external behavior recognition models or video description models. It directly inputs the original video frames into the model, converts them into visual alignment features through a visual encoder and connector, and concatenates them with text embedding representations to form a unified multimodal input sequence. This solves the problems of complex system structure and error accumulation in existing technologies, and further improves the prediction accuracy and efficiency of long-term behavior. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 One of the flowcharts for a long-term behavior prediction method based on a multimodal large model is provided for this application; Figure 2 The second schematic diagram of the long-term behavior prediction method based on a multimodal large model provided in this application; Figure 3 This is a flowchart of the method for generating a potential future visual representation using a context-aware visual generation module according to the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The long-term behavior prediction method provided by this invention is based on a multimodal large model. Figure 1 This application provides one of the flowcharts illustrating a long-term behavior prediction method based on a multimodal large model, such as... Figure 1 As shown, the multimodal large model provided in this application includes: a visual encoder, a connector, and a base model.

[0029] This invention provides an end-to-end long-term behavior prediction method based on visual alignment. This method achieves direct prediction of future behavior sequences from raw videos by integrating visual understanding, temporal modeling, and future behavior reasoning within a unified multimodal framework. Specifically, the method uses raw video as input and, under a unified multimodal modeling framework and multimodal thinking chain strategy, simultaneously achieves visual understanding of historical behavior and temporal reasoning of future behavior. This avoids reliance on external modules such as behavior recognition or video description, effectively reducing semantic loss and error propagation problems caused by information compression. Furthermore, by introducing latent visual features during the prediction process, this invention continuously visually constrains the model's reasoning process, ensuring that the prediction results not only rely on high-level semantic reasoning but also maintain consistency with low-level visual evidence. This overcomes the shortcomings of existing base model-based methods that emphasize semantics over visual understanding. Moreover, this invention simplifies the traditional two-stage processing flow through an end-to-end integrated design, reducing system complexity and improving overall reasoning efficiency while ensuring improved prediction performance.

[0030] In summary, this invention improves prediction accuracy, semantic consistency, and system usability in long-term behavior prediction tasks by achieving deep integration of video understanding and future behavior prediction within a unified framework.

[0031] The method provided by this invention takes as input an observation video and text instructions for defining the model's task objectives, prediction range, and output format, outputs an intermediate result multimodal reasoning thought chain, and finally outputs a complete future behavior sequence over a future period based on the intermediate result.

[0032] The multimodal reasoning thought chain serves as an intermediate basis in the future behavior sequence generation process. The future behavior sequence is the final prediction result generated autoregressively under the semantic context provided by the multimodal reasoning thought chain and the visual constraints provided by the future visual latent representation. This multimodal reasoning thought chain consists of textual reasoning fragments, temporally representative future behaviors, and future visual latent representations. Specifically, the textual reasoning fragments include scene summaries of the observed video, descriptions of observed behaviors, and inferences about behavioral intentions; the temporally representative future behaviors are key behavioral nodes predicted from a future timeframe; and the future visual latent representation is a sequence of visual feature vectors corresponding to the temporally representative future behaviors, used to characterize the possible visual states corresponding to those future behaviors. The future behavior sequence is the final output sequence of behavior labels from the model, represented in the form of verb-noun pairs.

[0033] In the unified decoding process, the multimodal large model does not independently complete multimodal inference, then independently generate future visual latent representations, and finally independently predict future behavior sequences. Instead, it organizes these processes into a continuous generation process within the same autoregressive decoding sequence. Specifically, the multimodal large model first generates text inference fragments such as scene summaries, descriptions of observed behaviors, and inferences about behavioral intentions based on observed videos and text instructions. Subsequently, based on the context formed by these text inference fragments, it predicts several temporally representative future behaviors. Whenever the model predicts a temporally representative future behavior, it triggers the future visual state generation process, generating a future visual latent representation corresponding to that future behavior. The basic model includes a visual encoder, a visual feature mapping / connection structure, and an autoregressive generation backbone. The visual encoder is used to extract visual features from observed videos or sampled video frames to obtain visual embedding representations. The visual feature mapping / connection structure is used to convert the visual embedding representations into visual feature representations that can be processed by the autoregressive generation backbone, and combines them with text features to form a unified multimodal input sequence. The autoregressive generation backbone comprises several Transformer decoding layers and an output prediction layer, used to progressively generate subsequent future visual latent representations based on the generated context information. In this process, the generated future visual latent representations are first written as visual feature nodes into the current autoregressive decoding context. Subsequently, the visual latent representation summarization module aggregates, maps, and normalizes these representations to obtain a visual summary representation. As the base model continues decoding subsequent content, a visually guided attention bias mechanism modulates the attention score at the corresponding position of the future visual latent representation in the autoregressive generation backbone based on the similarity between the current decoded hidden state and the visual summary representation. The modulated attention score participates in subsequent hidden state updates, and the updated hidden state is further input into the output prediction layer to generate subsequent future behavior sequences. Finally, under the constraints of a multimodal reasoning thought chain composed of text inference fragments, temporally representative future behaviors, and future visual latent representations, the model generates a complete future behavior sequence.

[0034] The aforementioned video understanding, temporally representative future behavior prediction, future visual latent representation generation, visual constraint injection, and complete future behavior sequence prediction are all completed in the autoregressive decoding process of the same basic model. There is no need to rely on behavior recognition models or video description models to pre-convert the observed video into discrete behavior labels or natural language descriptions, thereby reducing information loss and error propagation caused by intermediate representations and improving the semantic consistency, visual consistency, and temporal coherence of long-term behavior prediction.

[0035] Figure 2 The second schematic diagram of the long-term behavior prediction method based on a multimodal large model provided in this application is shown below. Figure 2 The method specifically includes the following steps: Step 101: Obtain the observation video and text instructions; the text instructions are used to define the model's task objective, prediction range, and output format.

[0036] Specifically, the observed video provides visual information that has occurred up to the current moment, including scene content, object states, hand movements, and interactions between people and objects; text instructions define the model's task objective, prediction range, and output format. For example, in a kitchen operation scenario, the task objective could be to predict the user's possible subsequent actions based on actions already performed, such as kneading dough, taking a bowl, and organizing ingredients; the prediction range could be set to a number of actions within a future period, such as 20 future actions; and the output format could be a sequence of actions consisting of verb-noun pairs, such as taking dough, kneading dough, putting down a bowl, and taking a knife.

[0037] Step 102: Extract features from the observed video to obtain an initial visual latent representation.

[0038] Specifically, after the observed video is input into the visual encoder, the visual encoder extracts features from the video segments to obtain a visual embedding representation of the observed video. This visual embedding representation is mapped to the embedding space of the base model via a connector, forming visual alignment features. Text instructions are processed by a text embedding layer to obtain a text embedding representation.

[0039] In the data processing stage, this application samples the observed video into a fixed-length frame sequence in chronological order and inputs it into the visual encoder built into the multimodal large model to obtain the initial visual latent representation corresponding to the observed video, with the shape being [ , , ],in The number of video frames obtained from sampling. The number of visual lexical units corresponding to each frame. The initial visual latent representation is then mapped to an embedding space that the base model can process via a connector. Text instructions, after text embedding, are combined with the visual alignment features corresponding to the observed video to form a multimodal input sequence, which is then input into the base model. During the autoregressive decoding process, the base model generates a multimodal reasoning thought chain and a future behavior sequence.

[0040] Step 103: Perform feature mapping on the initial visual latent representation to obtain visual alignment features.

[0041] Step 104: Convert the text instruction into a text embedding representation, and combine the visual alignment feature and the text embedding representation into a multimodal input sequence.

[0042] Step 105: Perform autoregressive decoding based on the multimodal input sequence to generate text inference fragments; the text inference fragments include scene summaries, descriptions of observed behaviors, and inferences of behavioral intentions from the observed video.

[0043] Specifically, in the autoregressive decoding process, the base model first generates a text inference fragment based on the multimodal input sequence. This text inference fragment includes a scene summary of the observed video, a description of the observed behavior, and an inference of the behavioral intent. For example, for a kitchen operation video, the model can generate inferences such as the user being in front of the kitchen counter, processing dough, and possibly continuing to knead or place the dough. This text inference fragment is used to provide semantic context for subsequent predictions of future behavior.

[0044] Step 106: Based on the text reasoning fragment, predict at least one temporally representative future behavior within the prediction range.

[0045] Specifically, after obtaining the text reasoning fragment, the base model further predicts several temporally representative future behaviors. These temporally representative future behaviors are not arbitrarily selected from future behaviors, but rather key behavioral nodes determined according to a preset prediction length and grouping rules. Specifically, when the model needs to predict a complete sequence of behaviors over a future period, such as predicting 20 future behaviors, it divides the future behavior sequence into several consecutive behavior groups according to chronological order, and selects the first behavior in each group as the temporally representative future behavior for that group. These temporally representative future behaviors are used to characterize the behavioral state changes at the beginning of each time period and serve as intermediate anchor points triggering the generation of subsequent future visual states, rather than representing the entire content of the final complete behavior sequence.

[0046] Step 107: Whenever a representative future behavior is predicted, a future visual latent representation corresponding to that representative future behavior is generated based on the current decoded hidden state, the visual alignment features, and the current historical visual latent representation; the future visual latent representation includes multiple visual latent vectors.

[0047] Specifically, whenever the base model predicts a temporally representative future action, the context-aware visual generation module is triggered. This module generates a future visual latent representation corresponding to that temporally representative future action based on the current decoded hidden state, the visual alignment features corresponding to the observed video, and the current historical visual latent representation. Specifically, the future visual latent representation corresponding to the (k)th temporally representative future action can be represented as... Each of them Given a continuous real-valued vector, This indicates the number of visual latent vectors contained in the future visual latent representation. Multiple visual latent vectors can respectively represent different visual regions or different visual lexical information in the future scene, such as the hand region, the target object region, the object position change region, and the human-object interaction region. Therefore, the continuous vector sequence used in this invention can retain richer future visual state information than a single vector. This continuous vector sequence is generated stepwise by the context-aware visual generation module in an autoregressive manner: when generating the current latent vector, the module uses the latent vectors generated in the previous step as the local visual context, and combines them with the initial visual latent representation corresponding to the observed video and the historical future visual latent representation to gradually obtain the complete future visual latent representation.

[0048] Specifically, Figure 3 This is a flowchart of the method for generating future visual latent representations using a context-aware visual generation module according to the present invention, such as... Figure 3 As shown below, the detailed process of generating a future visual latent representation based on the current decoding hidden state, the visual alignment features, and the current historical visual latent representation is described, specifically including the following steps: A global visual context is constructed based on the visual alignment features and the current historical visual latent representation, and the global visual context is conditionally processed according to the current decoded hidden state. In the process of progressively generating the future visual latent representation, all the visual latent vectors contained in the currently generated future visual latent representation are used as local visual context; Under the constraints of the current decoded hidden state, causal cross-attention computation and causal self-attention computation are performed based on the global visual context and the local visual context to gradually generate the future visual latent representation.

[0049] Specifically, to introduce future visual constraints in the long-term behavior prediction process, this invention sets up a context-aware visual generation module. This module is used to generate a future visual latent representation corresponding to a temporally representative future behavior after the base model predicts that behavior. Specifically, when the base model predicts a certain temporally representative future behavior, the context-aware visual generation module is triggered. This module receives the hidden state at the current decoding position of the base model, the visual alignment features corresponding to the observed video, the previously generated future visual latent representation, and the historical visual latent representations already generated in the current future visual state. The hidden state at the current decoding position of the base model is output by the autoregressive generation backbone during the autoregressive decoding process; specifically, it is the hidden state generated by the last layer at the current decoding position. It should be noted that this hidden state does not directly store the original video or text data, but rather is an intermediate feature representation obtained by processing the current decoding sequence through self-attention calculation, feedforward transformation, and normalization. The current decoding sequence includes visual information corresponding to the observed video, text information corresponding to the text instruction, generated text inference fragments, and the predicted temporally representative future behavior. Therefore, this hidden state integrates visual information from the observed video, semantics of text instructions, generated inference content, and predicted future behavior information, serving as a semantic condition for the context-aware visual generation module to generate future visual latent representations. Previously generated future visual latent representations refer to the visual feature nodes generated by the context-aware visual generation module and written into the autoregressive decoding sequence after the representative future behaviors of the preceding time series have been predicted. When generating new future visual latent representations, the model reads the corresponding historical future visual latent representations from the current decoding cache or visual latent representation cache based on the position indices of these visual feature nodes in the decoding sequence, and combines them with the initial visual latent representation corresponding to the observed video as the global visual context. The context-aware visual generation module further combines some latent vectors already generated in the current future visual state to progressively generate fixed-length future visual latent representations.

[0050] When generating future visual latent representations, the context-aware visual generation module constructs a global visual context and a local visual context. The global visual context consists of the initial visual latent representation corresponding to the observed video and previously generated future visual latent representations, providing long-term visual memory from the observed video to the predicted future state. The local visual context consists of latent vectors already generated in the current future visual state, maintaining consistency within the current visual state. In each step of latent vector generation, the context-aware visual generation module first performs causal cross-attention calculation on the local and global visual contexts, enabling the currently generated latent vector to reference observed visual information and historical future visual states. Subsequently, it performs causal self-attention calculation on the latent vectors already generated within the current future visual state, ensuring consistency among latent vectors within the same future visual state. Finally, a linear mapping layer maps the attention calculation results to the current step's future visual latent vector, progressively generating a complete future visual latent representation. The generated future visual latent representation is written into the current autoregressive decoding context as a visual feature node in the multimodal reasoning thought chain, and further input into the visual latent representation summarization module for visual constraints in subsequent behavior prediction.

[0051] Step 108: For each of the generated future visual latent representations, aggregate the multiple visual latent vectors contained therein to obtain the corresponding visual summary representation.

[0052] Specifically, the future visual latent representation generated by the context-aware visual generation module is then input into the visual latent representation summarization module. This future visual latent representation consists of multiple visual latent vectors. The visual latent representation summarization module aggregates, linearly maps, and normalizes all of these visual latent vectors to obtain a compact visual summary representation. This visual summary representation is used to summarize the core information of the future visual state and serves as a visual semantic anchor in the subsequent decoding process.

[0053] The following describes how to aggregate the multiple visual latent vectors contained in each generated future visual latent representation to obtain the corresponding visual summary representation, specifically including the following steps: The multiple visual latent vectors contained in each of the future visual latent representations are aggregated to obtain aggregated features; the aggregated features are averaged and pooled to obtain overall visual features; the overall visual features are linearly mapped and then normalized to obtain the visual summary representation.

[0054] Specifically, since a future visual latent representation typically consists of multiple visual latent vectors, directly using all latent vectors in subsequent decoding processes would increase computational overhead. Therefore, this invention includes a visual latent representation summarization module to compress the future visual latent representation output by the context-aware visual generation module into a compact visual summary representation. Specifically, the visual latent representation summarization module receives the future visual latent representation and aggregates its visual latent vectors, using average pooling to obtain the overall visual features. These overall visual features are then projected onto the decoding space of the base model through a linear mapping layer; finally, a normalization layer is used to obtain the visual summary representation. This visual summary representation summarizes the core visual semantics of the future visual state and serves as a visual semantic anchor point in the subsequent autoregressive decoding process. Through this visual summary representation, the base model can obtain visual constraint information related to future behavior without directly focusing on all visual latent vectors at each decoding step.

[0055] Step 109: Construct a visual guidance attention bias mechanism based on the current visual summary representations, and modulate the self-attention score in the subsequent autoregressive decoding process accordingly, thereby constraining the generated future behaviors and finally outputting a complete sequence of future behaviors.

[0056] Specifically, as the base model continues to generate subsequent content, the future visual latent representation generated by the context-aware visual generation module is first input into the visual latent representation summarization module to obtain a compact visual summary representation. Subsequently, the visual-guided attention bias mechanism modulates the self-attention score in the subsequent autoregressive decoding process of the base model based on the correlation between the current decoded hidden state and the visual summary representation, enabling the model to refer more to future visual information relevant to the current prediction when continuing to predict future behavior.

[0057] It should be noted that the temporal representative future behavior is different from the complete future behavior sequence. The former is the first behavior of each group after the future behavior sequence is grouped by time according to the prediction range, which is used to trigger the generation of future visual state; the latter is the complete prediction result of the final output of the model.

[0058] The following describes the implementation scheme for constructing a visually guided attention bias mechanism based on the correlation between the current decoded hidden state and each visual summary representation, and finally outputting a complete sequence of future behaviors. The specific steps include: First, the similarity between the query vector of the current location to be generated and the visual summary representation corresponding to each temporally representative future behavior is calculated, and this similarity is used as the visual guidance attention bias.

[0059] in, Represents the first position of historical position i Visual guidance to shift attentional bias This represents the query vector indicating the current location to be generated. Indicates the first A visual summary representation obtained by aggregating several future visual latent representations; let the current position be... Historical position The original attention score is: in, For position The key vector, For the feature dimension, if the first... The set of locations of the future visual latent representations in the decoding context is: The modulated attention score is:

[0060] This represents the attention score after incorporating visually guided attention bias. The weighting coefficients for visual attention bias. This is an indicator function.

[0061] Secondly, the modulated attention score is normalized using softmax to obtain the current position. For each historical location The attention weights are then used to adjust the value vectors at the corresponding historical positions. Weighted aggregation is performed to obtain the hidden state after the current position is updated. Finally, based on the hidden state after the current position is updated, the generated future behaviors are constrained, and the complete sequence of future behaviors is finally output.

[0062] Specifically, to ensure that the generated future visual latent representation can actually influence subsequent behavior predictions, this invention introduces a visually guided attention bias mechanism during the self-attention calculation process of the base model. This mechanism is not an independent network module, but rather, based on the aforementioned base model structure, it performs lightweight modulation of the attention score during the self-attention calculation stage of the autoregressive generation backbone. Specifically, as the base model continues to generate subsequent content, the current decoded hidden state is obtained by the autoregressive generation backbone based on the current decoded sequence. This current decoded sequence includes text information, visual information corresponding to the observed video, generated text inference fragments, predicted temporally representative future behaviors, and future visual latent representations written into the decoding context. Let the hidden state of the current marker position to be generated be... Historical position The hidden state is The self-attention sublayer performs linear mapping on the above hidden states to obtain the query vector, key vector, and value vector, respectively:

[0063] in, This is the query vector for the current location to be generated, used to represent the query semantics of the current location; For historical position The key vector is used to calculate the degree of attention matching with the query vector; For historical position The value vector is used to represent the content information that actually participated in the weighted aggregation at that historical location.

[0064] The visual latent representation summarization module will... A future visual latent representation is compressed into a visual summary representation. The similarity between the two is calculated as a visually guided attention bias:

[0065] in, This represents the cosine similarity.

[0066] Set current position Historical position The original attention score is: in, For position The key vector, Let be the feature dimension. If the ... The set of locations of the future visual latent representations in the decoding context is: The modulated attention score is: in, This represents the attention score after incorporating visually guided attention bias. The weighting coefficients for visual attention bias. For indicator functions. When the position Belongs to the A set of locations of future visual latent representations in the decoding context When the condition is met, the indicator function takes the value 1; otherwise, it takes the value 0.

[0067] Subsequently, the base model modulated attention scores Perform softmax normalization to obtain the current position. For each historical location The attention weights are then used to adjust the value vectors at the corresponding historical positions. Weighted aggregation is performed to obtain the updated hidden state at the current position. This updated hidden state is then input into the output prediction layer to generate subsequent future behavior sequences. Thus, visual attention bias is achieved. By altering the attention weights at the locations of future visual latent representations, the value vector aggregation results are influenced, thereby affecting the generation of subsequent future behavior sequences.

[0068] This application employs a visually guided attention bias mechanism, enabling future visual latent representations to participate continuously in the subsequent decoding process rather than being generated as intermediate features, thus providing visual constraints on the generation of complete future behavioral sequences. This mechanism requires no changes to the core structure of the basic model, adding only a visually relevant bias during the attention score calculation stage, resulting in low computational overhead.

[0069] The training process of the basic large model provided by this invention is described below: This invention employs a phased training strategy to gradually improve the model's reasoning and visual modeling capabilities.

[0070] Phase 1: Basic Model Training In this stage, only the backbone parameters of the multimodal large model are trained; the context-aware visual generation module and the visual latent representation summarization module are not enabled. The input consists of the initial visual latent representation and text instruction corresponding to the observed video, and the output is the future behavior sequence. The future behavior sequence is the final predicted complete behavior label sequence, represented in verb-noun pair form. This stage uses video training samples with future behavior annotations for training. Each training sample includes the observed video, text instruction, and the corresponding future behavior sequence annotation. During training, the observed video and text instruction are used as the model input, and the output is the complete future behavior sequence.

[0071] Specifically, this stage employs autoregressive language modeling loss to initially optimize the base model, enabling it to learn basic reasoning abilities from understanding observed videos to predicting future behavior. It should be noted that in the first stage, future visual latent representations are not yet involved in generation and supervision; visual constraint-related regression losses for future visual latent representations and visual semantic alignment losses will be introduced in the second stage.

[0072] The first stage uses standard autoregressive language modeling loss for optimization. Let the text inference fragment to be generated by the model and the corresponding behavior sequence be denoted as . The input conditions corresponding to the observed video and text commands are: Then the autoregressive language modeling loss can be expressed as:

[0073] This loss constrains the base model's generation of multimodal reasoning processes and future behavioral sequences, unlike the visual latent representation regression loss in the second stage. The autoregressive language modeling loss applies to textual reasoning and behavioral information; the visual latent representation regression loss applies to the vector difference between the generated future visual latent representation and the true future visual representation.

[0074] Phase Two: Visual Constraint Learning In this phase, the context-aware visual generation module and the visual latent representation summarization module are enabled, and the backbone parameters of the multimodal large model obtained in the first phase through autoregressive language modeling loss are frozen. The training primarily focuses on the context-aware visual generation module, the visual latent representation summarization module, and related mapping parameters. During training, after the base model predicts representative future behaviors, the context-aware visual generation module generates the corresponding future visual latent representation based on the current decoded hidden state, the initial visual latent representation corresponding to the observed video, and historical future visual latent representations. Subsequently, the visual latent representation summarization module compresses this future visual latent representation into a visual summary representation, which then participates in subsequent decoding through a visually guided attention bias mechanism.

[0075] This stage employs visual latent representation regression loss and visual semantic alignment loss for optimization. The visual latent representation regression loss constrains the generated future visual latent representation to maintain consistency with the target visual representation extracted by the pre-trained visual encoder. The target visual representation is extracted from real future video segments or future frames in the training samples that correspond to temporally representative future behaviors, using the pre-trained visual encoder. It should be noted that the initial visual latent representation originates from the observed video and serves as model input and part of the global visual context; the target visual representation originates from real future video content and is only used as a supervision signal during the training phase, not available during inference. In other words, the initial visual latent representation corresponds to the video content already seen, while the target visual representation corresponds to the actual future video content occurring in the training data.

[0076] The visual-semantic alignment loss is used to constrain the visual summary representation to maintain semantic consistency with the current decoded hidden state. This current decoded hidden state is output by the base model during autoregressive decoding; specifically, it's the hidden state generated at the current decoding position of the base model's last layer before generating the corresponding future visual latent representation. This hidden state contains a comprehensive semantic representation of the generated text inference fragment, the predicted temporally representative future behavior, and the observed video visual information, representing the semantic context of the model before generating the future visual state. By constraining the visual summary representation to maintain consistency with this current decoded hidden state, the compressed visual summary representation can match the current inference semantics, thus participating more effectively in subsequent behavior prediction.

[0077] In the second stage of autoregressive decoding, whenever the base model predicts a representative future action, the context-aware visual generation module is triggered to generate a corresponding future visual latent representation. This future visual latent representation consists of a fixed-length sequence of visual latent vectors, used to represent the future visual state corresponding to that representative future action. Specifically, during generation, the context-aware visual generation module uses the initial visual latent representation corresponding to the observed video and previously generated future visual latent representations as the global visual context, and uses a portion of the latent vectors already generated in the current future visual state as the local visual context. In generating the... The first future visual state When generating a potential vector, the module first extracts visual information related to the current local context from the global visual context through causal cross-attention. Then, it models the dependencies between the generated potential vectors within the current future visual state through causal self-attention. Finally, it obtains the potential vector for the current step through linear mapping. This process is repeated until a complete future visual potential representation is obtained.

[0078] It should be noted that the complete future visual state is the model's internal visual description of the future scene corresponding to a representative future behavior at a certain time sequence; the future visual latent representation is the vectorized data form of this future visual state in the model; and the future behavior sequence is the behavior label sequence finally output by the model. The first two belong to intermediate visual features, while the latter belongs to the final prediction result.

[0079] The second stage uses visual latent representation regression loss and visual semantic alignment loss for optimization. Let the context-aware visual generation module generate the first... A potential representation of future vision is The target visual representation (target visual latent representation) is obtained by extracting real future video clips or future frames corresponding to the representative future behavior of this time series from the training samples using a pre-trained visual encoder. Then the visual latent representation regression loss can be expressed as:

[0080] in, This represents the number of future visual states generated. For potential representation of future vision, Indicates the number of potential future visual representations. To obtain a visual latent representation by extracting features from the labeled observation video.

[0081] Simultaneously, the visual latent representation summarization module will generate future visual latent representations. Compression into visual summary representation Current decoding hidden state The output is generated by the base model during the autoregressive decoding process. Specifically, in this embodiment, the base model includes a visual encoder, a visual feature mapping / connection structure, and an autoregressive generation backbone, wherein the autoregressive generation backbone includes multiple Transformer decoding layers and output prediction layers. The observed video is processed by the visual encoder and visual feature mapping / connection structure to form visual tags, and text instructions are processed by text embedding to form text tags. The visual tags, text tags, generated text inference fragments, and predicted temporal representative future behaviors together constitute the current decoding sequence. After this current decoding sequence is processed by the autoregressive generation backbone, it generates the first... The future visual potential represents the hidden state output from the previous current decoding position. This is used to characterize the fused text semantics at the current decoding position, the observed video visual information, the generated inference content, and the predicted temporal representative future behavior information. The visual semantic alignment loss is used to constrain the visual summarization representation. With the current decoding hidden state Maintaining consistency in the semantic space can be represented as:

[0082] in, This represents cosine similarity. Visual latent representation regression loss is used to constrain the generated future visual latent representation to be close to the true future visual representation; visual semantic alignment loss is used to constrain the compressed visual summary representation to be consistent with the current inferred semantics. They operate on different objects and are not the same loss function.

[0083] The long-term behavior prediction method based on a multimodal large model provided by this invention can predict future behavior directly from the original video input through a multimodal large model without relying on external modules such as behavior recognition or video description, thereby avoiding information loss and error propagation problems caused by intermediate representation, while reducing system complexity.

[0084] The long-term behavior prediction method based on a multimodal large model provided by this invention introduces future visual latent features into the autoregressive prediction process, enabling the model to make future predictions based on understanding historical behavior. This achieves continuous and unified modeling of the past and future, thereby improving the coherence, semantic consistency, and visual consistency of predictions over a long period of time, and enhancing the visual constraints and temporal reasoning capabilities in the prediction process.

[0085] The long-term behavior prediction method based on a multimodal large model provided by this invention explicitly aligns the prediction process with visual semantics through a visually guided attention modulation mechanism, reducing the bias caused by relying solely on symbolic reasoning and improving the prediction robustness in complex scenarios. At the same time, the end-to-end unified design reduces system complexity, achieving a more efficient reasoning process while ensuring performance improvement, thereby enhancing the model's prediction stability and practical application efficiency.

[0086] Experimental example: Experiments were conducted on three datasets: Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+, demonstrating that the method provided in this application effectively improves the accuracy and efficiency of predicting future behavior sequences from raw videos.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A long-term behavior prediction method based on a multimodal large model, characterized in that, include: Acquire observation videos and text instructions; the text instructions are used to define the model's task objective, prediction range, and output format. Feature extraction is performed on the observed video to obtain an initial visual latent representation; The initial visual latent representation is then subjected to feature mapping to obtain visual alignment features; The text instructions are converted into text embedding representations, and the visual alignment features and the text embedding representations are combined into a multimodal input sequence; Autoregressive decoding is performed based on the multimodal input sequence to generate text inference fragments; the text inference fragments include scene summaries, descriptions of observed behaviors, and inferences of behavioral intentions from the observed video. Based on the text reasoning fragment, at least one temporally representative future behavior within the prediction range is predicted; Whenever a representative future behavior is predicted, a future visual latent representation corresponding to that representative future behavior is generated based on the current decoded hidden state, the visual alignment features, and the current historical visual latent representation; the future visual latent representation includes multiple visual latent vectors. For each of the generated future visual latent representations, the multiple visual latent vectors contained therein are aggregated to obtain the corresponding visual summary representation; A visual guidance attention bias mechanism is constructed based on the current visual summary representations, and the self-attention score in the subsequent autoregressive decoding process is modulated accordingly to constrain the generated future behaviors and finally output a complete future behavior sequence. The future behavior sequence is a label sequence, which is composed of verb-noun pairs.

2. The long-term behavior prediction method based on a multimodal large model according to claim 1, characterized in that, The step of predicting at least one temporally representative future behavior within the prediction range based on the text reasoning fragment includes: Based on the prediction range, determine the future behavior sequence to be predicted; The future behavior sequence is divided into several consecutive behavior groups according to the time order; The first behavior in each behavior group is selected as the temporally representative future behavior of that group.

3. The long-term behavior prediction method based on a multimodal large model according to claim 1, characterized in that, The step of generating a future visual latent representation corresponding to the representative future behavior of this time series based on the current decoding hidden state, the visual alignment features, and the current historical visual latent representation includes: A global visual context is constructed based on the visual alignment features and the current historical visual latent representation, and the global visual context is conditionally processed according to the current decoded hidden state. In the process of progressively generating the future visual latent representation, all the visual latent vectors contained in the currently generated future visual latent representation are used as local visual context; Under the constraints of the current decoded hidden state, causal cross-attention computation and causal self-attention computation are performed based on the global visual context and the local visual context to gradually generate the future visual latent representation.

4. The long-term behavior prediction method based on a multimodal large model according to claim 1, characterized in that, The step of aggregating the multiple visual latent vectors contained in each of the generated future visual latent representations to obtain the corresponding visual summary representation includes: The multiple visual latent vectors contained in each of the aforementioned future visual latent representations are aggregated to obtain aggregated features; The aggregated features are then subjected to average pooling to obtain the overall visual features; The overall visual features are linearly mapped and then normalized to obtain the visual summary representation.

5. The long-term behavior prediction method based on a multimodal large model according to claim 1, characterized in that, The step of constructing a visually guided attention bias mechanism based on each current visual summary representation, and modulating the self-attention score in the subsequent autoregressive decoding process accordingly, thereby constraining the generated future behaviors and finally outputting a complete sequence of future behaviors, includes: Calculate the similarity between the query vector of the current location to be generated and the visual summary representation corresponding to each temporally representative future behavior, and use it as the visual guidance attention bias; in, Indicates the first position of the current position i Visual guidance to shift attentional bias This represents the query vector at the current position i. Indicates the first A visual summary representation obtained by aggregating potential future visual representations; Set current position Historical position The original attention score is: in, For historical position The key vector, For feature dimension, if the first The set of locations of the future visual latent representations in the decoding context is: Then the modulated attention score for: The weighting coefficients for visual attention bias. For indicator functions; The modulated attention score Perform softmax normalization to obtain the current position. For each historical location Attention weights; Using the attention weights for historical positions value vector Perform weighted aggregation to obtain the updated hidden state at the current position; The hidden state is updated based on the current position to constrain the subsequent generated future behaviors, and finally outputs a complete sequence of future behaviors.

6. The long-term behavior prediction method based on a multimodal large model according to claim 1, characterized in that, The multimodal large model includes: a visual encoder, a connector, and a base model; The visual encoder is used to extract features from the observed video to obtain the initial visual latent representation; The connector is used to map the initial visual latent representation to the embedding space of the autoregressive generated backbone to obtain the visual alignment feature; The base model is used to perform the autoregressive decoding process and generate the text reasoning fragment, temporally representative future behavior, future visual latent representation, and complete future behavior sequence.

7. The long-term behavior prediction method based on a multimodal large model according to claim 6, characterized in that, The training of the multimodal large model includes: Based on regression language modeling loss function The multimodal large model is initially trained to obtain the initially trained multimodal large model; in, The initial visual latent representation and text instructions are provided for the labeled observation videos. This represents the t-th target word generated using the labeled observation video; This represents the target words preceding the t-th target word generated using the labeled observation video; N represents the total number of target word sequences. This indicates that in the input condition X and target word Under the given conditions, the model generates the t-th target word. The probability of; Based on the visual latent representation regression loss function visual semantic alignment loss function The initially trained multimodal large model is then optimized to obtain the final trained multimodal large model.

8. The long-term behavior prediction method based on a multimodal large model according to claim 7, characterized in that, The visual latent representation regression loss function for: in, For potential representation of future vision, Indicates the number of potential future visual representations. To extract visual latent representations from labeled observation videos.

9. The long-term behavior prediction method based on a multimodal large model according to claim 7, characterized in that, The visual semantic alignment loss function for: in, The number of potential representations for future vision. For the first A visual summary representation. For the first The future visual potential represents the hidden state output from the current decoding position.

Images