A long video action planning method and device based on a hierarchical re-planning architecture

Abstract

The application provides a long video action planning method and device based on a hierarchical re-planning architecture, and belongs to the technical field of computer vision. The method comprises the following steps: based on input starting video observation and target video observation, an intermediate observation and a task category are generated through a first layer diffusion model; based on the starting video observation and the target video observation, and the generated intermediate observation and the task category, an initial action sequence is generated through a second layer diffusion model; the initial action sequence is filtered to reduce the search space of intermediate actions and freeze the actions at both ends of the sequence, and the intermediate actions are re-planned through a third layer diffusion model to generate a final action sequence. Based on the method, a long video action planning device based on the hierarchical re-planning architecture is also provided. Through the observation planning, action planning and re-planning mechanism, the planning accuracy of the middle actions in the long video action sequence is significantly improved, and the balance and robustness of the overall planning are improved.

Description

A method and device for motion planning in long videos based on a hierarchical replanning architecture Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a method and device for long video motion planning based on a hierarchical replanning architecture. Background Technology

[0002] Long-video motion planning is a core research direction in the field of video understanding. Its task is to plan a series of coherent motion steps from unstructured long videos based on a starting state (e.g., an unassembled part) and a target state (e.g., an assembled product). This technology has wide applications in smart manufacturing, embodied intelligent robots, and other fields. In modern smart factories, motion planning typically relies on predefined rules or expert systems. Long-video motion planning technology enables robotic arms to autonomously learn operational processes, such as car assembly, electronic component soldering, and robot inspection, thereby reducing reliance on manual programming and improving production efficiency and flexible manufacturing capabilities. Furthermore, in the context of embodied intelligence, robots (such as home service robots and medical robots) need to learn how to complete tasks, such as tidying a room, preparing food, or caring for a patient, from long videos demonstrated by humans. Through motion planning, robots can understand and decompose task steps and execute them in the actual environment, thereby improving the level of intelligence in human-robot interaction.

[0003] Existing techniques for long-video motion planning can be broadly categorized into two-branch autoregressive planning methods and single-branch non-autoregressive planning methods. Two-branch autoregressive planning uses intermediate action labels and visual states for supervised prediction within an iterative planning mechanism, which often leads to error accumulation. Single-branch non-autoregressive planning uses only action labels for weakly supervised learning while simultaneously planning the entire intermediate action sequence. Although this method has achieved some performance breakthroughs, the lack of intermediate observation compensation makes it difficult to effectively capture the observational dependencies of dynamic action evolution, thus weakening the stability of planning accuracy during long-term inference and leading to accuracy decay in the middle of the action sequence. This is because existing methods prioritize optimizing the overall sequence planning accuracy while neglecting the balance of action prediction accuracy at different positions within the sequence. This causes the model to prioritize improving the accuracy at the two ends of the sequence with real observation information, while ignoring the accuracy of intermediate sequence actions lacking observation information.

[0004] Existing methods typically rely solely on action labels for weakly supervised learning, lacking explicit modeling of the intrinsic relationships (conjugate relationships) between observation information and actions. This exacerbates the uncertainty in intermediate action planning. Therefore, there is an urgent need in this field for a technical solution that can balance the accuracy of the entire action sequence planning, particularly improving the accuracy of mid-stage action planning. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a long video motion planning method and device based on a hierarchical replanning architecture, which achieves more accurate and balanced global motion sequence planning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A long video motion planning method based on a hierarchical replanning architecture includes the following steps:

[0008] Based on the initial and target video observations, intermediate observations and task categories are generated through a first-layer diffusion model.

[0009] Based on the initial video observations and target video observations, as well as the generated intermediate observations and task categories, an initial action sequence is generated through a second-layer diffusion model;

[0010] The initial action sequence is filtered to narrow down the search space for intermediate actions and the actions at both ends of the sequence are frozen. The intermediate actions are then replanned using a third-layer diffusion model to generate the final action sequence.

[0011] The present invention also proposes a long video motion planning device based on a hierarchical replanning architecture, comprising at least one processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the at least one processor, implements the long video motion planning method based on the hierarchical replanning architecture.

[0012] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0013] This invention proposes a long video motion planning method and device based on a hierarchical replanning architecture. The method includes the following steps: based on input initial video observations and target video observations, intermediate observations and task categories are generated through a first-layer diffusion model; based on the initial video observations, target video observations, and the generated intermediate observations and task categories, an initial motion sequence is generated through a second-layer diffusion model; the initial motion sequence is filtered to narrow the search space for intermediate actions and the actions at both ends of the sequence are frozen; the intermediate actions are replanned through a third-layer diffusion model to generate the final motion sequence. Based on this method, a long video motion planning device based on a hierarchical replanning architecture is also proposed. The proposed "observation-action-replanning" hierarchical architecture effectively fills information gaps by generating intermediate observations, providing richer contextual information for motion planning.

[0014] The action observation loss and adaptive replanning loss designed in this invention optimize the model from two dimensions: semantic consistency and dynamic error correction, effectively alleviating the uncertainty and accuracy decay problems of mid-stage actions.

[0015] This invention significantly improves the planning accuracy of mid-segment actions in long video action sequences through hierarchical planning and replanning mechanisms, and improves the overall planning balance and robustness. Experiments on public datasets have verified its superior performance. Attached Figure Description

[0016] Figure 1 is a flowchart of a long video motion planning method based on a hierarchical replanning architecture proposed in Embodiment 1 of the present invention;

[0017] Figure 2 is a framework diagram of a long video motion planning method based on a hierarchical replanning architecture proposed in Embodiment 1 of the present invention;

[0018] Figure 3 is a schematic diagram of a long video motion planning device based on a hierarchical replanning architecture proposed in Embodiment 2 of the present invention. Detailed Implementation

[0019] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0020] Example 1

[0021] Embodiment 1 of this invention proposes a long video motion planning method based on a hierarchical replanning architecture, aiming to solve the technical problem of motion planning accuracy decay in the middle of the sequence in existing non-autoregressive long video motion planning methods.

[0022] This invention fully utilizes the principles of hierarchical planning and replanning, proposing a hierarchical replanning algorithm. On one hand, it supplements intermediate observation information using a hierarchical replanning architecture, reducing the search space for intermediate actions to alleviate the imbalance in planning accuracy across the entire action sequence. On the other hand, it enhances the model's ability to model the conjugate relationship between actions and observations using action-observation loss; and it applies a dynamic penalty strategy to different samples using replanning loss to mitigate the impact of uncertainty in intermediate action planning.

[0023] The application fields of this invention are not limited to intelligent manufacturing, embodied intelligent robots, and other technical fields.

[0024] Figure 1 is a flowchart of a long video motion planning method based on a hierarchical replanning architecture proposed in Embodiment 1 of the present invention;

[0025] In step S110, based on the input initial video observation and target video observation, intermediate observations and task categories are generated through the first-layer diffusion model;

[0026] Based on the initial and target video observations, intermediate observations and task categories are generated through a first-layer diffusion model, specifically:

[0027] Construct a three-dimensional input matrix; the three dimensions include observation dimension, action dimension, and task dimension;

[0028] The task dimension, the observation dimension excluding the initial video observation, and the observation dimension excluding the target video observation are all initialized with Gaussian noise, and the action dimension is set to zero.

[0029] Intermediate observations and task categories are generated through iterative denoising.

[0030] Figure 2 is a framework diagram of a long video motion planning method based on a hierarchical replanning architecture proposed in Embodiment 1 of the present invention;

[0031] The input for a long video motion planning task is the start video observation. and target video observation The output is a sequence of actions that transforms the initial video observation into the target video observation. , Indicates the length of the action sequence.

[0032] Figure 2 shows an input matrix consisting of three dimensions. ,in Indicates the diffusion step size. This represents the observation planning hierarchy. The three dimensions are: observation dimension (including the observation sequence of the initial and target observations), action dimension (including the intermediate action sequence), and task dimension (action-related task categories). The goal of step S110 is to model the sample distribution of the observation dimension and task categories, denoted as... The first-layer diffusion model initializes the task dimension and the observation dimension (excluding the initial and target observations) with Gaussian noise, and then generates intermediate observations through iterative denoising. and task categories During this stage, the action dimension is always set to 0 to prevent interference with the generation process.

[0033] The method also includes constraining the generated intermediate observations and task categories using an observation loss function; the first-layer diffusion model, at this level, aims to generate observation features more relevant to the action sequence, thereby guiding action planning. However, since action segments consist of multiple video frames, some of these frames may contain redundant information unrelated to the action. Therefore, we use cosine similarity to filter video frames by comparing the similarity of each frame to the semantic representation of the action label. A predetermined number of video frames with the highest similarity are selected to represent the key observations of each intermediate action. These key observations and task labels Together, they are used as true values ​​to constrain the model. The expression for the observation loss is:

[0034] ;

[0035] in, Indicates observation loss; Indicates the total length of the action sequence; The first layer diffusion model is represented as the... Intermediate observation features generated by each action; This indicates that the first [item] is from the actual video. Key observational features extracted from each action; The model predicts the first The task category corresponding to each action; Indicates the first Each action represents a real task category.

[0036] In this application, the first-layer diffusion model operates at the observation planning level. Its process is as follows: it receives initial and target video observations, constructs an input matrix containing observation, action, and task dimensions, and initializes the observation and task dimensions (excluding the initial and target observations) with Gaussian noise, while setting the action dimension to zero. Subsequently, the model iteratively denoises to gradually recover a coherent sequence of intermediate observations and task categories from the noise. The core function of this model is to fill the information gap in long video sequences where realistic visual observations are lacking. By generating reasonable intermediate visual states, it provides crucial contextual information for subsequent action planning, thereby transforming the abstract initial and target states into a series of concrete, transitional visual states.

[0037] The first-layer diffusion model is built upon the Transformer architecture. The scope of this invention is not limited to the listed Transformer architectures; those skilled in the art can make appropriate choices based on the specific circumstances.

[0038] In step S120, an initial action sequence is generated based on the initial video observation and the target video observation, as well as the generated intermediate observations and task categories, through a second-layer diffusion model;

[0039] The specific process includes: constructing the input matrix for the action planning hierarchy. The input matrix is ​​obtained by splicing together the initial video observations. Target video observation Intermediate observations generated The generated task categories And an action dimension initialized with Gaussian noise. The specific details obtained are as follows:

[0040] ;

[0041] During the iterative denoising process of the second-layer diffusion model, the initial video observation is maintained. Target video observation Intermediate observation and task categories Fixed and unchanging, only for the action dimension Perform noise reduction;

[0042] Through the iterative denoising process, the second-layer diffusion model learns and generates the initial action sequence. probability distribution ;in, ; The parameter is The conditional probability distribution defined by the second-layer diffusion model; This represents the probability distribution of the initial action sequence generated after iterative denoising.

[0043] Step S120 aims to generate the probability distribution of the action sequence through an iterative denoising process. The model only initializes the action dimension in the output of the previous stage with Gaussian noise and freezes the observation sequence. and task categories Construct the input matrix ,in This represents the action planning level. The training objective of this level is to model the sample distribution of action sequences, denoted as... Through an iterative denoising process, the model generates action sequences. Used for subsequent action replanning levels.

[0044] The generation process of the initial action sequence is constrained by an action planning loss function, specifically as follows:

[0045] ;

[0046] in, Represents the action planning loss function; Represents the action observation loss function; This represents the mean squared error loss; Indicates weight;

[0047] ;

[0048] ;

[0049] in, This indicates the second-layer diffusion model. The characteristic representation of an action; Indicates the first The true value of each action; Indicates the first Each action generates a feature representation of the intermediate observations; Indicates the first The true key observational features of each action; Indicates the length of the action sequence.

[0050] Action observation loss The aim is to further constrain action generation based on the semantic similarity of intermediate observations associated with the planned action. If the generated action... Its corresponding real action in action space If they are highly consistent (i.e., their orientations exhibit a high degree of consistency), then their relational similarity should also be similar to the observation sequence in the representation space. and key observations The relationships between them exhibit consistent structural similarity.

[0051] The action observation loss should constrain the similarity between generated actions and real actions in the action space to approximate the similarity between corresponding observations in the representation space. Therefore, the ratio of the cosine similarity between the two is used as a constraint to encourage the second-layer diffusion model to generate actions that are not only close to the ground truth in the action space, but also semantically consistent with the visual context.

[0052] The second-layer diffusion model operates at the action planning level. Its process involves using the intermediate observation sequence and task category generated in the previous layer, along with the initial and target observations, as frozen conditional information, and separately initializing the action dimension with Gaussian noise. Using this visual and task context as conditions, the model iteratively denoises and generates the probability distribution of the initial action sequence. The core function of this model is to complete the initial conversion from visual information to action instructions, and to explicitly constrain the model's learning of the conjugate relationship between actions and observations using the action observation loss function. This ensures that the generated actions are not only numerically accurate, but their semantic relationships also remain consistent with the evolution of the visual state, thus achieving a more logical initial planning.

[0053] The second-layer diffusion model of this application is built on the Transformer architecture. The scope of protection of this invention is not limited to the listed Transformer architectures. Those skilled in the art can make reasonable choices according to the actual situation.

[0054] In step S130, the initial action sequence is filtered to narrow down the search space for intermediate actions and the actions at both ends of the sequence are frozen. The intermediate actions are then replanned using the third-layer diffusion model to generate the final action sequence.

[0055] While the above steps generate a probability distribution for the action sequence, the evolution of actions in the middle of the sequence is inherently uncertain; that is, the same observation information may lead to different subsequent actions. Furthermore, the potential errors introduced by intermediate observations further exacerbate this uncertainty, resulting in an imbalance in planning accuracy. To mitigate this problem, we introduce an action replanning hierarchy, which freezes the planning results for actions at both ends of the sequence and replans only the middle actions.

[0056] The initial action sequence is filtered to narrow down the search space for intermediate actions and the actions at both ends of the sequence are frozen; specifically:

[0057] For the initial action sequence Each intermediate action in Keep the one with the highest probability. We select 1 candidate actions and set the probability of the remaining candidate actions to zero, thus forming a filtered action probability distribution. ;

[0058] ;

[0059] in, This indicates a Top-K filtering operation;

[0060] Initial action of the sequence and final action Keep the prediction results obtained in the initial action planning phase unchanged, that is: ;

[0061] Use the filtered and frozen action sequence and task categories The input matrix of the action replanning hierarchy is constructed by concatenating the matrices. ;

[0062] ;

[0063] in, Indicates the starting action of the frozen sequence; This indicates the end of the frozen sequence.

[0064] Use an action filter to filter the planning results from the previous stage. This filter only retains the preceding steps of each intermediate action. The most likely action candidates (of which) (These are predefined hyperparameters), and they are masked by setting the probability of remaining low-probability actions to zero. This operation significantly reduces the action search space by eliminating semantically inconsistent or context-inappropriate actions, thereby reducing the uncertainty of intermediate action evolution. Simultaneously, this process freezes the initial predictions of the actions at both ends to ensure the stability of action sequence planning. Then, the input samples for the action replanning stage are constructed using the filtered distribution. ,in This represents the action replanning level. The training objective of this level is to model the sample distribution of actions in the middle of a sequence, denoted as... The model generates action sequences through iterative denoising. This is the final output of the algorithm. It's worth noting that since the selected action distribution essentially follows a Gaussian-like distribution, no additional noise is introduced during the action replanning process.

[0065] To impose targeted constraints on the action replanning process, we introduce a replanning loss function. This function applies adaptive constraints based on the correctness of the initial planning and replanning results relative to the true action. If the replanned action is correct, only the traditional MSE loss is used for constraint, without further adjustment. If the initial planned action is correct but the replanned action is incorrect, the model applies stricter constraints to prevent unnecessary bias. Conversely, if both planned actions are incorrect, additional similarity constraints are added to promote model convergence to the true action. This adaptive penalty mechanism ensures that the model pays more attention to incorrect replanning samples and maintains the stability of the action replanning process, thereby mitigating the impact of action uncertainty. The replanning loss function is expressed as:

[0066] ;

[0067] in, This represents the replanning loss function; This indicates that the action replanning level is the 1st. Individual planning actions; This represents a stability penalty term introduced when the replanning action is incorrect but the initial planning action is correct. When the initial planning is correct, it increases the similarity constraint with the initial plan, emphasizing the necessity of maintaining accurate predictions; and ; This represents a reinforcement guidance penalty introduced when both the replanning action and the initial planning action are incorrect. When both phases fail, it strengthens the constraint that the output of the replanning should be close to the true value.

[0068] The third-layer diffusion model operates at the action replanning level. Its process is as follows: it filters the initial action sequence, retaining the preceding actions of each intermediate action. The model selects high-probability candidates to significantly reduce the search space and freezes the correctly planned actions at both ends of the sequence. Using this processed sequence as input, the model iterates and optimizes intermediate actions within the limited search space. The core function of this model is to specifically alleviate the accuracy decay problem caused by the evolutionary uncertainty of actions in the middle of the sequence. By refining the replanning process and combining it with an adaptive replanning loss function, it achieves local calibration and optimization of intermediate actions, thereby effectively improving the planning accuracy and robustness of the overall action sequence while maintaining the stability of both ends of the sequence.

[0069] The third-layer diffusion model in this application is built based on the Transformer architecture. The scope of protection of this invention is not limited to the listed Transformer architectures; those skilled in the art can make reasonable choices based on the actual situation.

[0070] Embodiment 1 of this invention discloses a long video action planning method based on a hierarchical replanning architecture. The proposed "observation-action-replanning" hierarchical architecture effectively supplements information gaps by generating intermediate observations, providing richer contextual information for action planning.

[0071] Embodiment 1 of this invention discloses a long video motion planning method based on a hierarchical replanning architecture. The designed motion observation loss and adaptive replanning loss optimize the model from two dimensions: semantic consistency and dynamic error correction, effectively alleviating the uncertainty and accuracy decay problems of mid-segment actions.

[0072] Embodiment 1 of this invention discloses a long video motion planning method based on a hierarchical replanning architecture. Through hierarchical planning and replanning mechanisms, it significantly improves the planning accuracy of mid-segment actions in long video motion sequences, and improves the overall planning balance and robustness. Experiments on public datasets have verified its superior performance.

[0073] Example 2

[0074] The present invention also proposes a device. Figure 3 is a schematic diagram of a long video motion planning device based on a hierarchical replanning architecture proposed in Embodiment 2 of the present invention, comprising:

[0075] Memory, used to store computer programs;

[0076] When a processor executes the computer program, the method steps are as follows:

[0077] In step S110, based on the input initial video observation and target video observation, intermediate observations and task categories are generated through the first-layer diffusion model;

[0078] In step S120, an initial action sequence is generated based on the initial video observation and the target video observation, as well as the generated intermediate observations and task categories, through a second-layer diffusion model;

[0079] In step S130, the initial action sequence is filtered to narrow down the search space for intermediate actions and the actions at both ends of the sequence are frozen. The intermediate actions are then replanned using the third-layer diffusion model to generate the final action sequence.

[0080] Embodiment 2 of this invention discloses a long video motion planning device based on a hierarchical replanning architecture. The proposed "observation-action-replanning" hierarchical architecture effectively supplements information gaps by generating intermediate observations, providing richer contextual information for motion planning.

[0081] It should be noted that the present invention also provides an electronic device, including: a communication interface capable of interacting with other devices such as network devices; and a processor connected to the communication interface to enable information interaction with other devices, used to execute a long video motion planning method based on a hierarchical replanning architecture provided by one or more of the above technical solutions when running a computer program, wherein the computer program is stored in a memory. Of course, in practical applications, the various components in the electronic device are coupled together through a bus system. It is understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The memory in the embodiments of this application is used to store various types of data to support the operation of the electronic device. Examples of this data include any computer program used to operate on the electronic device. It is understood that the memory can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache.By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory. The methods disclosed in the embodiments of this application can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing digital signal processing technology), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method. When the processor executes the program, it implements the corresponding processes in the various methods of the embodiments of this application; for simplicity, these will not be elaborated further here.

[0082] The description of the relevant parts of the long video motion planning device based on a hierarchical replanning architecture provided in Embodiment 2 of this application can be found in the detailed description of the corresponding parts of the long video motion planning method based on a hierarchical replanning architecture provided in Embodiment 1 of this application, and will not be repeated here.

[0083] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A long video motion planning method based on a hierarchical replanning architecture, characterized in that, Includes the following steps: Based on the initial and target video observations, intermediate observations and task categories are generated through a first-layer diffusion model. Specifically, the process involves: constructing a three-dimensional input matrix, comprising observation, action, and task dimensions; initializing the task dimension, all observation dimensions except the initial video observation, and all observation dimensions except the target video observation with Gaussian noise, and setting the action dimension to zero; generating intermediate observations and task categories through iterative denoising; and generating an initial action sequence based on the initial video observation, the target video observation, and the generated intermediate observations and task categories using a second-layer diffusion model. Specifically, this involves constructing an input matrix for the action planning layer. The input matrix is ​​obtained by splicing together the initial video observations. Target video observation Intermediate observations generated The generated task categories And an action dimension initialized with Gaussian noise. The specific details obtained are as follows: During the iterative denoising process of the second-layer diffusion model, the initial video observation is maintained. Target video observation Intermediate observation and task categories Fixed and unchanging, only for the action dimension Denoising is performed; through the iterative denoising process, the second-layer diffusion model learns and generates an initial action sequence. probability distribution ;in, ； The parameter is The conditional probability distribution defined by the second-layer diffusion model; This represents the probability distribution of the initial action sequence generated after iterative denoising; the initial action sequence is filtered to narrow down the search space of intermediate actions and freeze the actions at both ends of the sequence; the intermediate actions are then replanned using a third-layer diffusion model to generate the final action sequence; specifically, the initial action sequence is filtered to narrow down the search space of intermediate actions and freeze the actions at both ends of the sequence; Each intermediate action in Keep the one with the highest probability. We select 1 candidate actions and set the probability of the remaining candidate actions to zero, thus forming a filtered action probability distribution. ； ;in, This represents the Top-K filtering operation; the initial action on the sequence. and final action Keep the prediction results obtained in the initial action planning phase unchanged, that is: ; Use filtered and frozen action sequences and task categories The input matrix of the action replanning hierarchy is constructed by concatenating the matrices. ； ;in, Indicates the starting action of the frozen sequence; This indicates the end of the frozen sequence.

2. The method according to claim 1, characterized in that, The method further includes using an observation loss function to constrain the generated intermediate observations and task categories; the expression for the observation loss is: ;in, Indicates observation loss; Indicates the total length of the action sequence; The first layer diffusion model is represented as the... Intermediate observation features generated by each action; This indicates that the first [item] is from the actual video. Key observational features extracted from each action; The model predicts the first The task category corresponding to each action; Indicates the first Each action is a real task category.

3. The method according to claim 1, characterized in that, The method further includes: constraining the generation process of the initial action sequence using an action planning loss function, specifically: ;in, Represents the action planning loss function; Represents the action observation loss function; This represents the mean squared error loss; Indicates weight; ； ;in, This indicates the second-layer diffusion model. The characteristic representation of an action; Indicates the first The true value of each action; Indicates the first Each action generates a feature representation of the intermediate observation; Indicates the first The true key observational features of each action; Indicates the length of the action sequence.

4. The method according to claim 1, characterized in that, The method further includes: constraining the replanning process using a replanning loss function, wherein the replanning loss function applies an adaptive penalty based on the correctness of the initial planning result and the replanning result.

5. The method according to claim 4, characterized in that, The replanning loss function is expressed as: ;in, This represents the replanning loss function; This indicates that the action replanning level is the 1st. Individual planning actions; This represents a stability penalty term introduced when the replanning action is incorrect but the initial planning action is correct; and ； This indicates a reinforcement guidance penalty introduced when both the replanning action and the initial planning action are incorrect.

6. The method according to any one of claims 1 to 5, characterized in that, The first-layer diffusion model, the second-layer diffusion model, and the third-layer diffusion model are all built on the Transformer architecture.

7. A long video motion planning device based on a hierarchical replanning architecture, comprising at least one processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the at least one processor, it implements a long video motion planning method based on a hierarchical replanning architecture as described in any one of claims 1 to 6.

Images