A video long-time action prediction method and device

Abstract

The application provides a video long-time action prediction method and device, the method comprising: processing an obtained observation video segment sequence by using a first prediction model, extracting visual features and generating a preliminary future action sequence prediction, the preliminary prediction comprising verb and noun probability distributions of each time step in the future; obtaining a corresponding observed noun sequence based on the observation video segment sequence; reasoning the observed noun sequence by using a second prediction model to obtain a noun set irrelevant to a current scene context; performing mask and normalization processing on the noun probability distributions of each time step in the future based on the noun set irrelevant to the current scene context to obtain optimized noun probability distributions; and outputting a predicted future action sequence by combining the optimized noun probability distributions and corresponding verb probability distributions. Based on the method, a video long-time action prediction device is also provided. The application significantly improves the accuracy and semantic rationality of long-time action prediction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a method and device for long-term video motion prediction. Background Technology

[0002] Long-term video motion prediction is the prediction of future activities over a longer time span (e.g., 20 action steps) based on relatively short observed video data (e.g., 4 action steps). Long-term video motion prediction technology is one of the important research topics in the field of computer vision and has broad application prospects. For example, in human-computer interaction scenarios, to achieve efficient collaboration, robots need to predict in advance the actions that humans are about to perform in order to promptly carry out parallel auxiliary tasks; in intelligent monitoring and early warning systems, to ensure the personal safety of children or the elderly, the system needs to predict whether their future behavior poses a potential danger and issue risk warnings in advance. How to fully utilize visual cues and temporal features in observed video data to achieve accurate and stable long-term motion prediction is a technical problem that needs to be solved.

[0003] Currently, methods for long-term action prediction tasks can be broadly categorized into two types. The first type extracts video features through a visual encoder, integrates the information via an aggregation module, and finally generates the predicted future action by a decoder. This type of method has a clear structure, high computational efficiency, and is easy to optimize and deploy. However, its main limitation lies in its insufficient ability to model complex behavioral patterns, especially when facing long-term future actions, making it difficult to capture implicit semantic relationships and high-level action intentions. The second type of method introduces large language models, leveraging their powerful contextual understanding and generation capabilities to model future actions. This type of method can effectively integrate external knowledge, object semantics, and action logic, improving the coherence and semantic rationality of the prediction results. However, due to its high dependence on large-scale pre-trained models, this type of method often incurs high training and inference costs and has limited ability to model key details in visual information. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a method and device for long-duration video action prediction. Through a collaborative mechanism between a visual prediction model and a large language model, it retains the visual prediction model's ability to perceive detailed information while incorporating the common-sense reasoning ability of the large language model, significantly improving the accuracy and semantic rationality of long-duration action prediction.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for predicting long-duration motion in a video includes the following steps: Obtain a sequence of observed video clips; process the sequence of observed video clips using a first prediction model to extract visual features and generate a preliminary prediction of future action sequences, the preliminary prediction including the probability distribution of verbs and the probability distribution of nouns at each future time step; the first prediction model is a visual prediction model; Based on the observed video segment sequence, the corresponding observed gerund sequence is obtained; the second prediction model is used to infer the observed gerund sequence to obtain a set of nouns that are not related to the current scene context; the second prediction model is a large language model. Based on the set of nouns that are not related to the current scene context, the probability distribution of nouns at each future time step is masked and normalized to obtain an optimized noun probability distribution; combined with the optimized noun probability distribution and the corresponding verb probability distribution, the final predicted future action sequence is output.

[0006] The present invention also proposes a video long-duration motion prediction device, including at least one processor and a memory, wherein the memory stores a computer program, and the computer program implements the video long-duration motion prediction method when executed by the at least one processor.

[0007] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects: This invention proposes a method and device for long-duration video motion prediction, the method comprising the following steps: The method involves acquiring a sequence of observed video clips; processing the sequence using a first prediction model to extract visual features and generate preliminary predictions of future action sequences, including verb and noun probability distributions for each future time step; the first prediction model is a visual prediction model; obtaining corresponding observed verb and noun sequences based on the observed video clip sequence; reasoning about the observed verb and noun sequences using a second prediction model to obtain a set of nouns unrelated to the current scene context; the second prediction model is a large language model; masking and normalizing the noun probability distributions for each future time step based on the set of nouns unrelated to the current scene context to obtain optimized noun probability distributions; and combining the optimized noun probability distributions with the corresponding verb probability distributions to output the final predicted future action sequence. Based on this method for long-term video action prediction, a device for long-term video action prediction is also proposed. This invention, through the collaborative mechanism of a visual prediction model and a large language model, retains the visual prediction model's ability to perceive detailed information while incorporating the common-sense reasoning ability of the large language model, significantly improving the accuracy and semantic rationality of long-term action prediction.

[0008] This invention utilizes a large language model to constrain the prediction space, effectively reducing the uncertainty of long sequence predictions and making the prediction results more consistent with the logic of real-world scenarios.

[0009] The noun temporal smoothing loss and noun distribution alignment loss functions proposed in this invention constrain the prediction results from both temporal and semantic dimensions, further enhancing the stability and consistency of the predicted sequence. Attached Figure Description

[0010] Figure 1 This is a flowchart of a video long-duration motion prediction method proposed in Embodiment 1 of the present invention; Figure 2 This is a framework diagram of a video long-duration motion prediction method proposed in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of a video long-duration motion prediction device proposed in Embodiment 2 of the present invention. Detailed Implementation

[0011] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0012] Example 1 Embodiment 1 of this invention proposes a long-term video action prediction method, which combines the efficiency of visual prediction module in visual information extraction with the reasoning capabilities of large language model to construct a collaborative mechanism. On one hand, the visual prediction model is responsible for perceiving object regions in the video and extracting object features; on the other hand, the semantic knowledge contained in the large language model filters unreasonable action nouns, constructs a low-probability candidate noun set, and filters and constrains the action prediction space of the visual prediction model, thereby effectively improving the accuracy and semantic consistency of prediction. The low-probability candidate noun set is a set of nouns that are irrelevant to the current scene context.

[0013] Figure 1 This is a flowchart of a video long-duration motion prediction method proposed in Embodiment 1 of the present invention; In step S110, an observation video segment sequence is obtained; the observation video segment sequence is processed using a first prediction model to extract visual features and generate a preliminary prediction of future action sequences. The preliminary prediction includes the probability distribution of verbs and the probability distribution of nouns at each future time step; the first prediction model is a visual prediction model. Figure 2 This is a framework diagram of a video long-duration action prediction method proposed in Embodiment 1 of the present invention; the goal of the video long-duration action prediction task is to predict actions by including... Using video clips of continuous action steps to predict the future In each action step, the most likely action that the actor will perform is determined. For ease of modeling, for the observation portion of the video, this algorithm uses the labeled information to separate the original, untrimmed video... Divided into A series of consecutive video clips. For each video clip, a 2-second segment is randomly selected and uniformly sampled, denoted as . Each video clip Each action tag corresponds to a specific action.

[0014] The first prediction model in this application is a visual prediction model, which includes a visual encoder, a visual auxiliary information extractor, and an action predictor. A visual encoder is used to extract global visual features from a sequence of observed video clips. The visual encoder uses a Slowfast network as its backbone, which combines the ability to model both spatial and dynamic temporal information of the video. The output obtained from the input encoder is represented as ,in, It is the first Global feature representation of an action segment.

[0015] A visual-assisted information extractor (VAI) is used to extract hand and object features from the observed video segment sequence using a cross-attention mechanism. The VAI extractor also extracts local semantic information, including hand and object features. Feature extraction is achieved using a Transformer suitable for video structure, combined with a cross-attention mechanism. Specifically, for the input video... Uniform sampling Frames, forming a frame set ;Will Visual features are obtained by feeding them into the visual backbone network. Subsequently, two sets of shared query vectors are introduced: the hand query vector. and Query vector of an object , respectively used from Extract features from the hand and key objects. Visual features... The query vector, serving as both key and value, is input to the cross-attention module as the query. Finally, the corresponding output is... and These correspond to the hand features and object features learned by the model, respectively.

[0016] The action predictor generates preliminary predictions of future action sequences based on the global visual features, hand features, and object features. The predictive encoder employs a bidirectional Transformer architecture to perform temporal modeling and multimodal information fusion on the aforementioned features, initializing... Query parameters Each query corresponds to an action prediction for a future time step. , and The query Q, along with the input query vector, is fed into the predictive encoder. Positional and modal encodings are added before encoding to help the model capture temporal relationships and feature sources. Each output query vector corresponds to a gerund probability distribution at a future time step. Finally, the predictive distribution is sampled to obtain the final prediction result.

[0017] In step S120, based on the observed video segment sequence, the corresponding observed gerund sequence is obtained; the observed gerund sequence is inferred using the second prediction model to obtain a set of nouns that are not related to the current scene context; the second prediction model is a large language model; To obtain more reasonable and accurate prediction results, the prediction space of the LLM compression model is introduced. In this invention, the large language model plays an auxiliary role. By combining external knowledge and contextual understanding ability, the noun space is reduced, and unreasonable candidates in the context are eliminated.

[0018] For input video First, using the annotation information provided in the dataset, the corresponding verb and noun sequences are extracted and combined into verb-noun pairs as the semantic representation of the action sequence. Based on this, natural language-formatted prompts are designed to guide the LLM in generating a set of nouns that are unlikely to appear in the current scenario.

[0019] It mainly consists of two core parts: the first is the system prompt words, which describe the structure of the input observation action sequence and explicitly indicate the noun prediction space. The definition, and requires LLM to consider possible scenarios for the current observed action sequence, combining the scenario context from The set of low-relevance nouns that do not fit the current context is identified. The second part provides a formal description of the action sequences in the currently observed video. LLM, based on its common-sense knowledge and contextual understanding, combines the input verbs and inferred scenarios to... The system filters out nouns that do not fit the context and generates a set of low-probability candidate nouns. To improve the reliability of LLM output, this method introduces a post-processing mechanism into the initial LLM results: removing the noun prediction space. Irrelevant nouns, and at the same time, normalize the nouns (convert plurals to singulars). If If the number of nouns in the code is less than 120, a reduction request will be sent to LLM again until the requirement is met.

[0020] In step S130, based on a set of nouns that are not related to the current scene context, the probability distribution of nouns at each future time step is masked and normalized to obtain an optimized noun probability distribution; combined with the optimized noun probability distribution and the corresponding verb probability distribution, the final predicted future action sequence is output.

[0021] Based on a set of nouns that are irrelevant to the current context Construct mask tensor ,in, The size of the noun prediction space, the first of the mask tensors Each component Defined as: ; in, Represents the noun prediction space The first in A noun category; this mask tensor is then applied to the predicted distribution of nouns at each time step to filter out unreasonable candidates.

[0022] The following formula is used for the first Noun probability distribution at each time step By performing masking and normalization, the optimized distribution is obtained. ; ; in, This indicates element-wise multiplication; This indicates a normalization operation.

[0023] Through the above operations, the large language model filters out unreasonable noun choices in the current context for the visual prediction model, thereby helping the visual prediction model focus on more reasonable candidates during the prediction process and reducing prediction uncertainty.

[0024] The present invention also includes: minimizing the total loss function Optimize the parameters of the first prediction model; total loss function Cross-entropy loss Noun distribution alignment loss and noun time sequence smoothing loss The weighted summation is used; the specific expression is: ; in, The preset weighting coefficients represent the cross-entropy loss; The predefined weight coefficients represent the loss of noun distribution alignment; and satisfy the following conditions: ; ; .

[0025] Cross-entropy loss Used to constrain the differences between the probability distributions of verbs and nouns and their corresponding true labels. Constrain the prediction distribution of the visual prediction model using true values; noun distribution alignment loss. Used to constrain the noun probability distribution to maintain consistency with prior co-occurrence patterns based on training data statistics; noun temporal smoothing loss. This is used to constrain the first prediction model so that the predicted noun sequence conforms to the prior knowledge of continuous and stable interactive objects in the real scene.

[0026] Cross-entropy loss The specific expression is: ; in, The number of future steps to be predicted; Indicates the first prediction model in the... The probability distribution of verbs predicted at each time step; Indicates the first prediction model in the... The probability distribution of nouns predicted at each time step; Indicates the first Real tags for verbs at each time step; Indicates the first The true label of the noun at each time step; This represents the cross-entropy function.

[0027] Noun distribution alignment loss The predicted distribution of nouns is constrained using a pre-constructed co-occurrence matrix. This is because objects always appear in specific contexts. For example, kitchen utensils like dishwashers, knives, and forks generally appear in scenarios related to kitchen activities; while objects like bricks and concrete always appear in scenarios related to outdoor construction work. The two scenarios in the above example are quite different, and it's almost impossible for different objects from both scenarios to appear simultaneously in a single video.

[0028] Noun distribution alignment loss The noun distribution predicted using the KL divergence constraint model is made closer to common co-occurrence patterns in the data. The specific expression is as follows: ; in, Indicates KL divergence; This represents the target prior noun distribution constructed based on nouns in the input observed video segment sequence; Distribution of target prior terms Common term: co-occurrence matrix construction; the co-occurrence matrix is ​​obtained through a statistical training set, and its co-occurrence probability... The calculation formula is: ; in, Noun and nouns The number of times they appear together in the same video; The total number of videos in the training set; co-occurrence matrix coordinates The value at the position represents a noun. and nouns The joint probability of them appearing in the same scene reflects their correlation in the semantic scene.

[0029] Noun time sequence smoothing loss The specific expression is: ; in, Indicates the first Time step; Indicates the first Time step; Indicates the first prediction model in the... The probability distribution of nouns predicted by time step; Indicates the first prediction model in the... The probability distribution of nouns predicted by time step.

[0030] Combining the optimized noun probability distribution and the corresponding verb probability distribution, the final predicted future action sequence is output. Specifically, for each future time step, the verb with the highest probability value is selected from the verb probability distribution of that time step as the predicted verb; the noun with the highest probability value is selected from the optimized noun probability distribution of that time step as the predicted noun; the predicted verb and the predicted noun are combined to form the predicted action for that time step; and the predicted actions of all future time steps are arranged in chronological order to form the final predicted future action sequence.

[0031] For each future time step Perform the following operations: From the probability distribution of verbs The verb with the highest probability value is selected as the predicted verb for that time step. ,Right now ; From the optimized noun probability distribution The term with the highest probability value is selected as the predicted term for that time step. ,Right now ; Combine the predicted verb with the predicted noun to form the predicted action for that time step. ; All The predicted actions at each time step are arranged in chronological order to form the final sequence of future actions. And output it.

[0032] The video long-term action prediction method proposed in Embodiment 1 of this invention, through the collaborative mechanism of visual prediction model and large language model, not only retains the visual prediction model's ability to perceive detailed information, but also incorporates the common sense reasoning ability of large language model, which significantly improves the accuracy and semantic rationality of long-term action prediction.

[0033] The video long-time action prediction method proposed in Embodiment 1 of this invention uses a large language model to constrain the prediction space, which effectively reduces the uncertainty of long sequence prediction and makes the prediction results more consistent with the scene logic of the real world.

[0034] The present invention provides a video long-term action prediction method in Embodiment 1. The proposed noun temporal smoothing loss and noun distribution alignment loss function constrain the prediction results from both temporal and semantic dimensions, thereby further enhancing the stability and consistency of the prediction sequence.

[0035] Example 2 The present invention also proposes a device, Figure 3 This is a schematic diagram of a video long-duration motion prediction device according to Embodiment 2 of the present invention, comprising: Memory, used to store computer programs; When a processor executes the computer program, the method steps are as follows: In step S110, an observation video segment sequence is obtained; the observation video segment sequence is processed using a first prediction model to extract visual features and generate a preliminary prediction of future action sequences. The preliminary prediction includes the probability distribution of verbs and the probability distribution of nouns at each future time step; the first prediction model is a visual prediction model. In step S120, based on the observed video segment sequence, the corresponding observed gerund sequence is obtained; the observed gerund sequence is inferred using the second prediction model to obtain a set of nouns that are not related to the current scene context; the second prediction model is a large language model; In step S130, based on a set of nouns that are not related to the current scene context, the probability distribution of nouns at each future time step is masked and normalized to obtain an optimized noun probability distribution; combined with the optimized noun probability distribution and the corresponding verb probability distribution, the final predicted future action sequence is output.

[0036] The video long-duration action prediction device proposed in Embodiment 2 of this invention, through the collaborative mechanism of visual prediction model and large language model, not only retains the visual prediction model's ability to perceive detailed information, but also incorporates the common sense reasoning ability of large language model, which significantly improves the accuracy and semantic rationality of long-duration action prediction.

[0037] The video long-time action prediction device proposed in Embodiment 2 of the present invention uses a large language model to constrain the prediction space, which effectively reduces the uncertainty of long sequence prediction and makes the prediction results more consistent with the scene logic of the real world.

[0038] The present invention provides a video long-duration action prediction device in embodiment 2. The proposed noun temporal smoothing loss and noun distribution alignment loss function constrain the prediction results from both temporal and semantic dimensions, thereby further enhancing the stability and consistency of the prediction sequence.

[0039] It should be noted that the present invention also provides an electronic device, including: a communication interface capable of interacting with other devices such as network devices; and a processor connected to the communication interface to enable information interaction with other devices, used to execute a video long-time motion prediction method provided by one or more of the above technical solutions when running a computer program, wherein the computer program is stored in a memory. In practical applications, the various components of the electronic device are coupled together through a bus system. It is understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The memory in the embodiments of this application is used to store various types of data to support the operation of the electronic device. Examples of this data include any computer program used to operate on the electronic device. It is understood that the memory can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache.By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory. The methods disclosed in the embodiments of this application can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing digital signal processing technology), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method. When the processor executes the program, it implements the corresponding processes in the various methods of the embodiments of this application; for simplicity, these will not be elaborated further here.

[0040] The description of the relevant parts of the video long-duration motion prediction device provided in Embodiment 2 of this application can be found in the detailed description of the corresponding parts of the video long-duration motion prediction method provided in Embodiment 1 of this application, and will not be repeated here.

[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0042] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for predicting long-duration motion in video, characterized in that, Includes the following steps: A sequence of observed video clips is acquired; the sequence of observed video clips is processed using a first prediction model to extract visual features and generate a preliminary prediction of future action sequences. The preliminary prediction includes the probability distribution of verbs and the probability distribution of nouns at each future time step; the first prediction model is a visual prediction model; the first prediction model includes a visual encoder, a visual auxiliary information extractor, and an action predictor; the visual encoder is used to extract global visual features from the sequence of observed video clips. The visual aid information extractor is used to extract hand features and object features from the observed video segment sequence through a cross-attention mechanism; the action predictor generates a preliminary prediction of future action sequences based on the global visual features, hand features, and object features. Based on the observed video segment sequence, the corresponding observed gerund sequence is obtained; the second prediction model is used to infer the observed gerund sequence to obtain a set of nouns that are not related to the current scene context; the second prediction model is a large language model. Based on the set of nouns that are not related to the current scene context, the probability distribution of nouns at each future time step is masked and normalized to obtain the optimized probability distribution of nouns. By combining the optimized noun probability distribution and the corresponding verb probability distribution, the final predicted sequence of future actions is output. The method further includes: minimizing the total loss function. Optimize the parameters of the first prediction model; the total loss function Cross-entropy loss Noun distribution alignment loss and noun time sequence smoothing loss The weighted summation is used; the specific expression is: ； in, The preset weighting coefficients represent the cross-entropy loss; The predefined weight coefficients represent the loss of noun distribution alignment; and satisfy the following conditions: ; ; ; Cross-entropy loss Used to constrain the differences between the verb probability distribution and the noun probability distribution and their corresponding true labels; noun distribution alignment loss. Used to constrain the noun probability distribution to maintain consistency with prior co-occurrence patterns based on training data statistics; noun temporal smoothing loss. This is used to constrain the first prediction model so that the predicted noun sequence conforms to the prior knowledge of continuous and stable interactive objects in the real scene. The noun distribution alignment loss The specific expression is: ； in, Indicates KL divergence; This represents the target prior noun distribution constructed based on nouns in the input observed video segment sequence; The number of future steps to be predicted; Indicates the first prediction model in the... The probability distribution of nouns predicted at each time step; Distribution of target prior terms Common term: co-occurrence matrix construction; the co-occurrence matrix is ​​obtained through a statistical training set, and its co-occurrence probability... The calculation formula is: ； in, Noun and nouns The number of times they appear together in the same video; The total number of videos in the training set; The term time-series smoothing loss The specific expression is: ； in, Indicates the first Time step; Indicates the first Time step; Indicates the first prediction model in the... The probability distribution of nouns predicted by time step; Indicates the first prediction model in the... The probability distribution of nouns predicted by time step.

2. The method according to claim 1, characterized in that, Based on the observed video segment sequence, the corresponding observed gerund sequence is obtained, specifically: Read the observed gerund sequences directly from the dataset annotation information; Alternatively, the observed gerund sequence can be obtained by identifying the observed video segment sequence through a pre-trained action recognition model.

3. The method according to claim 1, characterized in that, Based on the set of nouns that are irrelevant to the current scene context, the probability distribution of nouns at future time steps is masked and normalized, specifically as follows: Based on a set of nouns that are irrelevant to the current context Construct mask tensor ,in, The size of the noun prediction space, the first of the mask tensors Each component Defined as: ； in, Represents the noun prediction space The first in Noun categories; The following formula is used for the first Noun probability distribution at each time step By performing masking and normalization, the optimized distribution is obtained. ; ； in, This indicates element-wise multiplication; This indicates a normalization operation.

4. The method according to claim 1, characterized in that, Combining the optimized noun probability distribution and the corresponding verb probability distribution, the final predicted sequence of future actions is output, as follows: For each future time step, select the verb with the highest probability value from the verb probability distribution of that time step as the predicted verb; Select the noun with the highest probability value from the optimized noun probability distribution at this time step as the predicted noun; The predicted verb is combined with the predicted noun to form the predicted action for that time step; Arrange all the predicted actions for all future time steps in chronological order to form the final predicted sequence of future actions.

5. The method according to claim 1, characterized in that, The cross-entropy loss The specific expression is: ； in, The number of future steps to be predicted; Indicates the first prediction model in the... The probability distribution of verbs predicted at each time step; Indicates the first prediction model in the... The probability distribution of nouns predicted at each time step; Indicates the first Real tags for verbs at each time step; Indicates the first The true label of the noun at each time step; This represents the cross-entropy function.

6. A video long-duration motion prediction device, comprising at least one processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the at least one processor, it implements a video long-duration motion prediction method as described in any one of claims 1 to 5.

Images