Hamiltonian physics-informed world model method and system for long-term prediction

Abstract

The application provides a Hamiltonian physical prior world model method and system for long-term prediction, and relates to the technical field of artificial intelligence. The method comprises the following steps: encoding an observation image into a world state centered on an object; projecting the world state into a phase space representation through a Hamiltonian physical prior module, defining an evolution direction based on an energy gradient and a Hamiltonian equation, and generating a first candidate world state by using symplectic integration for discrete updating; extracting dynamic features from the time and space dimensions through a space-time adaptive decoupling module to generate a second candidate world state that compensates for non-conservative disturbance; fusing the two candidate states to obtain a predicted world state, and decoding the predicted world state into a predicted image frame; and recursively performing multi-step prediction. The application improves the stability and physical consistency of long-term prediction through Hamiltonian dynamics and symplectic integration, enhances environmental adaptability in combination with an adaptive module, and effectively reduces non-physical phenomena in multi-object interaction scenes.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a Hamiltonian physical prior world model method and system for long-term prediction. Background Technology

[0002] In tasks such as video prediction, robot environmental perception, and embodied intelligence, systems need to make continuous predictions of future states over long time spans based on limited historical observations. Related technologies often employ a world model framework with a latent state space, mapping observations to hidden states and learning the temporal evolution of states in the latent space.

[0003] However, existing state progression methods based on neural network regression are prone to error accumulation in multi-step recursive prediction, causing the predicted trajectory to gradually deviate from the actual state. First-order update methods in discrete time struggle to maintain the geometric structure of the physical system, often leading to energy drift and numerical instability. In multi-object interaction scenarios, non-physical phenomena such as object overlap, collision timing errors, and unreasonable motion trajectories are common. Furthermore, real-world environments contain non-conservative perturbations and observation-dependent changes, making it difficult for a single fixed dynamic model to accurately characterize such complex dynamic processes. Therefore, how to balance stability, physical consistency, and environmental adaptability in long-term forecasting remains a pressing technical challenge that needs to be addressed. Summary of the Invention

[0004] The purpose of this application is to provide a Hamiltonian physics prior world model method and system for long-term prediction, which can solve the problems in related technologies such as trajectory deviation caused by error accumulation during long-term prediction, energy drift and numerical instability, non-physical phenomena in multi-object interaction scenarios, and the difficulty of fixed dynamic models to adapt to non-conservative perturbations and observation-dependent changes in real environments.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: A first aspect of this application discloses a Hamiltonian physical prior world model method for long-term prediction, the method comprising: Each frame of the observation sequence is encoded as a world state centered on an object, the world state including multiple object slots for describing different objects in the scene; The world state at the current moment is projected into a phase space representation containing configuration variables and momentum variables using the Hamiltonian physics prior module. Based on the gradient of the total energy of the system calculated from the phase space representation, the evolution direction is defined by the Hamiltonian equation, and the evolution direction is updated in discrete time using the symplectic integral method to generate the first candidate world state. Herein, the configuration variables represent the structural relationships between objects, and the momentum variables represent the historical evolution trend of the objects. The spatiotemporal adaptive decoupling module extracts temporal dynamic features from the memory structure storing historical world states along the time dimension and extracts spatial dynamic features from the spatial relationships between the multiple object slots along the spatial dimension. Based on the temporal dynamic features and spatial dynamic features, a second candidate world state is generated to compensate for non-conservative perturbations. By fusing the first candidate world state and the second candidate world state, the predicted world state for the next time step is obtained; Decode the predicted world state at the next moment into a predicted image frame; The above steps are executed recursively to generate a sequence of predicted images for future steps.

[0006] Optionally, each frame of the observation sequence is encoded as the world state at the center of the object, including: Spatial feature maps of each frame of image are extracted using a convolutional neural network, and positional encoding is added to the spatial feature maps. Flatten the spatial feature map with added location encoding into multiple image tokens; The multiple image tokens are aggregated into a preset number and dimension of object slots using an object-centric encoder.

[0007] Optionally, the world state at the current moment is projected into a phase space representation including configuration variables and momentum variables, including: The object slots in the current world state are input into the attention network to obtain the configuration variables; The momentum variable is obtained by inputting the object slots in the current world state and the set of historical world states into the cross-attention network.

[0008] Optionally, the evolution direction is defined by the Hamiltonian equation based on the gradient of the total system energy calculated from the phase space representation, including: The configuration variables and momentum variables are concatenated along the feature dimension to obtain the phase space input representation; An object-level self-attention mechanism is applied to the phase space input representation to obtain phase space features containing information about interactions between objects; For each object, the energy contribution of that object is calculated by passing the corresponding part of the phase space features through a learnable scalar mapping function; The total energy of the system is obtained by summing the energy contributions of all objects. Calculate the gradient of the total energy of the system with respect to the configuration variable and the gradient with respect to the momentum variable; According to the Hamiltonian equation, the evolution direction of the configuration variable is defined as the gradient of the total energy of the system with respect to the momentum variable, and the evolution direction of the momentum variable is defined as the negative value of the gradient of the total energy of the system with respect to the configuration variable.

[0009] Optionally, the evolution direction is updated in discrete time using a symplectic integral method to generate a first candidate world state, including: Based on the gradient of the total energy of the system with respect to the current configuration variable, perform a half-step update on the momentum variable; Based on the gradient of the total energy of the system with respect to the momentum variable after the half-step update, a one-step full update is performed on the configuration variable; Based on the gradient of the total energy of the system with respect to the updated configuration variable, another half-step update is performed on the momentum variable to obtain the updated momentum variable; The updated configuration variables and the updated momentum variables are concatenated and mapped back to the object slot space through a projection function to obtain the first candidate world state.

[0010] Optionally, temporal dynamic features are extracted from the memory structure storing historical world states along the time dimension, including: Use the most recently preset number of historical world states as short-term memory; Long-term memory is maintained by a recursive update rule, which weights and fuses the long-term memory of the previous moment with the world state of the previous moment to obtain the long-term memory of the current moment. Using the current world state as a query, short-term and long-term time features are read from the short-term memory and the current long-term memory respectively through a cross-attention mechanism. The short-term time features are fused with the long-term time features to obtain the time dynamic features.

[0011] Optionally, spatial dynamic features are extracted from the spatial relationships between the plurality of object slots along the spatial dimension, including: The object slots in the current world state are input into the attention network to obtain inter-object interaction features used to characterize the interactions between objects; The object slots in the current world state and the image token of the current frame are input into the cross-attention network to obtain observation alignment features that characterize observation conditional dependencies. The spatial dynamics features are obtained by fusing the inter-object interaction features with the observation alignment features.

[0012] Optionally, based on the aforementioned temporal and spatial dynamic characteristics, a second candidate world state is generated to compensate for non-conservative perturbations, including: The current world state, the temporal dynamics feature, and the spatial dynamics feature are spliced ​​together to obtain the fused feature; The fused features are input into a multilayer perceptron, and the object-level update component with the same dimension as the world state is obtained by mapping through the multilayer perceptron, which serves as the second candidate world state.

[0013] Optionally, the method is implemented using a sequence prediction model, which includes at least an object center encoding module, a Hamiltonian physical prior module, and a spatiotemporal adaptive decoupling module; the sequence prediction model is trained according to the following steps: Acquire training data, which includes observed image sequences and corresponding real future image sequences; The observed image sequence is input into the sequence prediction model, and multi-step prediction is performed to obtain the predicted future image sequence and the corresponding predicted world state sequence. Obtain the real-world state sequence obtained by encoding the real future image sequence through the object center encoding module; Calculate the slot-level mean squared error loss between the predicted world state sequence and the real world state sequence; Calculate the image-level mean squared error loss between the predicted future image sequence and the actual future image sequence; The weighted sum of the slot-level mean squared error loss and the image-level mean squared error loss is used as the overall loss to jointly optimize the parameters of the sequence prediction model.

[0014] A second aspect of this application discloses a Hamiltonian physical prior world model system for long-term prediction, the system comprising: The object center encoding module is used to encode each frame of the observation sequence into a world state of the object center, wherein the world state is used to describe multiple object slots of different objects in the scene. The Hamiltonian physics prior module projects the current world state into a phase space representation containing configuration variables and momentum variables. Based on the gradient of the total system energy calculated from the phase space representation, the evolution direction is defined by the Hamiltonian equation, and the evolution direction is updated in discrete time using the symplectic integral method to generate a first candidate world state. Herein, the configuration variables represent the structural relationships between objects, and the momentum variables represent the historical evolution trend of the objects. The spatiotemporal adaptive decoupling module is used to extract temporal dynamic features from the memory structure storing historical world states along the time dimension, extract spatial dynamic features from the spatial relationship between the multiple object slots along the spatial dimension, and generate a second candidate world state to compensate for non-conservative perturbations based on the temporal dynamic features and spatial dynamic features. The state fusion module is used to fuse the first candidate world state and the second candidate world state to obtain the predicted world state at the next moment. The object-centered decoding module is used to decode the predicted world state of the next moment into a predicted image frame; The recursive prediction module is used to recursively call the Hamiltonian physics prior module, the spatiotemporal adaptive decoupling module, the state fusion module, and the object center decoding module to generate a sequence of predicted images for multiple future steps.

[0015] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the Hamiltonian physical prior world model method for long-term prediction described in the first aspect of this application.

[0016] A fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the Hamiltonian physical prior world model method for long-term prediction described in the first aspect of this application.

[0017] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the Hamiltonian physical prior world model method for long-term prediction described in the first aspect of this application.

[0018] The embodiments of this application have the following advantages: In this embodiment, each frame of the observation sequence is encoded as the world state of the object center, and the world state is projected into configuration variables and momentum variables in phase space through the Hamiltonian physics prior module. Then, the gradient of the total energy of the system is calculated, and the evolution direction is defined according to the Hamiltonian equation. This makes the state update process subject to structured dynamic constraints, avoiding the error accumulation generated by traditional neural network regression in multi-step recursion, thereby improving the stability of long-term prediction.

[0019] Meanwhile, using the symplectic integral method to update the evolution direction in discrete time can maintain the geometric structure of the phase space, suppress energy drift and long-term accumulation of numerical errors, thereby enhancing the physical consistency and numerical stability of the prediction process.

[0020] Furthermore, a spatiotemporal adaptive decoupling module extracts historical evolution features from the memory structure along the time dimension and interaction features from the spatial relationships between object slots along the spatial dimension, generating a second candidate world state to compensate for non-conservative perturbations. This allows the model to flexibly adapt to changes in non-conservative perturbations and observation dependencies in the real environment while maintaining the stability of the structured backbone.

[0021] Finally, the first and second candidate world states are fused and decoded to generate predicted image frames. Multi-step prediction is then recursively executed, achieving a unification of structured dynamics and adaptive environmental compensation. In multi-object interaction scenarios, this method can reduce non-physical phenomena such as object overlap, collision timing errors, and unreasonable trajectories, improving the realism and reliability of the prediction results.

[0022] In summary, this application balances stability, physical consistency, and environmental adaptability in long-term prediction tasks. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the steps of a Hamiltonian physical prior world model method for long-term prediction provided in an embodiment of this application. Figure 2 This is an overall architecture diagram of a Hamiltonian physical prior world model method for long-term prediction provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a Hamiltonian physical prior world model system for long-term prediction provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To facilitate understanding of the technical solution of this application, a brief description of the relevant technologies will be given first.

[0027] Existing long-term time series forecasting and world modeling methods are mostly based on a latent state-space framework. The core idea is to map observed data to latent states and model the temporal evolution within the latent space. These methods utilize encoders to map observed sequences into low-dimensional latent representations, and learn the state transition process (typically represented as...) through recurrent neural networks or state-space models. The state transitions are then processed by a decoder to generate future observations. This type of method has achieved some success in reinforcement learning and video prediction, but its state transitions essentially still rely on free-form neural network regression.

[0028] To enhance multi-object scene modeling capabilities, some studies have introduced object-centric representation mechanisms, improving the expressive power of complex scenes through explicit object decomposition. However, their dynamic modeling still primarily relies on neural network regression, lacking constraints on the underlying physical dynamics. On the other hand, to incorporate physical priors, some methods construct energy functions to characterize system evolution and utilize conserved structures to improve physical consistency; others attempt to use structure-preserving numerical integrals to improve long-term prediction stability. However, these methods are mostly applied to low-dimensional physical systems or simplified scenes, failing to effectively integrate with high-dimensional visual perception and object-centric modeling frameworks. Overall, existing methods still primarily rely on discrete-time neural network regression updates, lacking a unified framework that explicitly introduces Hamiltonian structures into state evolution and employs structure-preserving integrals for long-term advancement, resulting in significant shortcomings in long-term prediction stability and physical consistency.

[0029] To overcome the limitations of related technologies, this application proposes a Hamiltonian physical prior world model method for long-term prediction. The core concept is as follows: the observation sequence is encoded as the world state of the object center; the object slot is projected into configuration and momentum variables in phase space through a Hamiltonian physical prior module; the total system energy is calculated based on a learnable energy function; the evolution direction is defined by the energy gradient using the Hamiltonian equation, and discrete-time updates are performed using symplectic integrals to generate a structured first candidate world state; simultaneously, a spatiotemporal adaptive decoupling module extracts dynamic features along the time and space dimensions to generate a second candidate world state that compensates for non-conservative perturbations; the two are then fused and decoded to generate a prediction frame, and multi-step prediction is recursively executed. This approach maintains the stability of the Hamiltonian structure while also considering environmental adaptability, thereby improving the physical consistency and prediction accuracy of long-term predictions.

[0030] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a Hamiltonian physical prior world model method for long-term prediction provided in an embodiment of this application. Figure 1 As shown, this may include steps S110 to S160: Step S110: Encode each frame of the observation sequence as a world state centered on an object, the world state including multiple object slots for describing different objects in the scene.

[0031] In this step, the observation sequence is a sequence of video frames or continuous image frames acquired for a dynamic environment. The world state at the object center is a structured representation used to decompose a complex scene into several independent object units (object slots). Each object slot is a fixed-dimensional latent feature vector, corresponding to an object in the scene (such as a moving object, an obstacle, etc.).

[0032] By encoding each frame of the observation sequence as a world state centered on an object, subsequent dynamic modeling can be performed at the object level, facilitating the characterization of multi-object interactions and local evolution. This step provides a structured state foundation for the subsequent introduction of physical priors and adaptive modeling.

[0033] Step S120: Using the Hamiltonian physics prior module, the current world state is projected into a phase space representation containing configuration variables and momentum variables. Based on the gradient of the total system energy calculated from the phase space representation, the evolution direction is defined by the Hamiltonian equation, and the evolution direction is updated in discrete time using the symplectic integral method to generate a first candidate world state. Herein, the configuration variables represent the structural relationships between objects, and the momentum variables represent the historical evolution trend of the objects.

[0034] In this step, the Hamiltonian physics prior module learns a differentiable energy function and uses the gradient of this energy function with respect to phase space variables to deduce the direction of state evolution, thereby introducing Hamiltonian-form structured dynamic constraints into the latent space.

[0035] Specifically, the world state represented by the object slots is first projected into phase space to obtain configuration variables and momentum variables. Configuration variables primarily capture the spatial structural relationships between objects (e.g., relative position, orientation), while momentum variables capture the historical motion trends of the objects (e.g., velocity, acceleration). Then, based on these two variables, the total system energy is calculated using a learnable energy function. According to the Hamiltonian equations, the evolution direction of the configuration variables is given by the gradient of the total energy with respect to the momentum variables, and the evolution direction of the momentum variables is given by the negative gradient of the total energy with respect to the configuration variables. Finally, the evolution direction is discretized and updated using a symplectic integral method to obtain the phase space state at the next time step, which is then projected back into the object slot space, becoming the first candidate world state. This module effectively constrains the state evolution process by explicitly introducing a Hamiltonian dynamic structure, reducing the error accumulation caused by free regression.

[0036] Step S130: Using the spatiotemporal adaptive decoupling module, extract temporal dynamic features from the memory structure storing historical world states along the time dimension, extract spatial dynamic features from the spatial relationships between the multiple object slots along the spatial dimension, and generate a second candidate world state to compensate for non-conservative perturbations based on the temporal dynamic features and spatial dynamic features.

[0037] In this step, the spatiotemporal adaptive decoupling module is used to compensate for non-conservative factors (such as external disturbances, friction, and changes in observation conditions) that are difficult for the Hamiltonian physics prior module to handle. This module models environment-specific, non-stationary dynamic changes in the latent space at the center of the object, and effectively supplements the structured dynamic backbone without compromising long-term stability by explicitly decoupling time and space factors.

[0038] Specifically, along the time dimension, the module maintains short-term memory (the most recent historical world states) and long-term memory (historical information updated through recursive smoothing), and extracts temporal dynamic features through a cross-attention mechanism using the current world state as the query. Along the spatial dimension, the module captures the interaction relationships between object slots through self-attention, obtains observation-dependent features through cross-attention, and obtains spatial dynamic features after fusion. Finally, the current world state, temporal dynamic features, and spatial dynamic features are input into a multilayer perceptron to generate a second candidate world state, which is used to compensate for complex dynamic changes not covered by structured dynamics.

[0039] Step S140: Merge the first candidate world state and the second candidate world state to obtain the predicted world state for the next moment.

[0040] In this step, the first candidate world state is the structured backbone prediction, and the second candidate world state is the environmental adaptive compensation. The predicted world state for the next moment obtained by fusing the two maintains the long-term stability provided by Hamiltonian dynamics and has the ability to adapt to the dynamic changes of the actual scene.

[0041] The fusion method can be corresponding addition at the object slot level, or more flexible methods such as weighted averaging or gated fusion can be used. For example, the predicted world state at the next moment... It can be represented as:

[0042] in, The first candidate world state. It is the second candidate world state.

[0043] Step S150: Decode the predicted world state of the next moment into a predicted image frame.

[0044] In this step, the object-centric decoder can be used ( The fused predicted world state for the next time step is mapped back to pixel space to generate a visualized predicted image frame. ,Right now The object center decoder can employ a structure symmetrical to the encoder. For example, it can reconstruct the object image from each object slot through spatial broadcasting and deconvolution operations, and then synthesize the complete scene image. This step allows the model's internal predictions to be compared with real observations, facilitating training and evaluation.

[0045] Step S160: Recursively execute the above steps to generate a sequence of predicted images for future steps.

[0046] In this step, the predicted image frame generated in step S150 is used as the input for the next time step. Steps S120 to S150 are repeated to autoregressively generate a sequence of predicted image frames for multiple future time steps. During the recursive process, the Hamiltonian physics prior module and the spatiotemporal adaptive decoupling module share parameters, and the long-term memory is continuously updated, thus supporting long-term predictions of arbitrary length.

[0047] The technical solution of this embodiment achieves a balance between stability, physical consistency, and environmental adaptability in long-term prediction by encoding the observation sequence into the world state centered on the object and fusing the Hamiltonian physics prior module and the spatiotemporal adaptive decoupling module. Specifically, the Hamiltonian physics prior module uses energy gradients and Hamiltonian equations to define the evolution direction, avoiding error accumulation in free-form neural network regression; the symplectic integral method preserves the phase space geometry, suppressing energy drift and numerical errors; the spatiotemporal adaptive decoupling module effectively compensates for non-conservative perturbations and observation-dependent changes in the real environment through dynamic feature extraction in both time and space dimensions. The fusion of these two modules and recursive prediction reduces non-physical phenomena such as object overlap, collision timing errors, and unreasonable trajectories in multi-object interaction scenarios, improving the realism and reliability of the prediction results. Compared to existing technologies, this embodiment has significant advantages in tasks requiring long-term dynamic modeling, such as video prediction, robot environmental perception, and embodied intelligence.

[0048] In an optional embodiment, step S110 above, "encoding each frame of the observation sequence as the world state of the object center," may include steps S120-1 to S120-3: Step S120-1: Extract spatial feature maps of each frame image using a convolutional neural network, and add positional encoding to the spatial feature maps.

[0049] In this step, the convolutional neural network is used to extract spatial features from each frame of the image, outputting a spatial feature map that preserves spatial structure information (such as position, shape, and texture). Since the convolution operation itself does not have the ability to explicitly perceive spatial position, a two-dimensional position encoding needs to be added to the feature map so that the feature vector of each spatial position carries coordinate information, thereby enhancing the model's ability to perceive the spatial layout of objects.

[0050] Step S120-2: Flatten the spatial feature map after adding location encoding into multiple image tokens.

[0051] In this step, the spatial feature map with added location encoding is flattened in the spatial dimension to form a sequence of image tokens, denoted as . Where M is the number of tokens, This is the feature dimension. Each token corresponds to a spatial location in the spatial feature map. The number of tokens after flattening is equal to the height of the feature map multiplied by its width, which is usually greater than the preset number of object slots, providing rich local feature input for the object center encoder.

[0052] Step S120-3: Aggregate the multiple image tokens into a preset number and dimension of object slots using the object center encoder.

[0053] In this step, the object-centric encoder aggregates a large number of image tokens into multiple object slots with a fixed number and clear semantics. One implementation is to use a slot attention mechanism: initialize a set of learnable slot vectors (the number of which is the preset number of object slots), and through iterative attention calculation, make each slot vector aggregate relevant feature information from the image tokens, ultimately forming a latent representation of each object, i.e., the object slot.

[0054] For example, the world state can be represented as:

[0055] in, This is an object-centric encoder. This process does not rely on the prior location or category of objects, and can discover and separate independent objects in a scene in an unsupervised manner, laying the foundation for subsequent object-level dynamic modeling.

[0056] The technical solution adopted in this embodiment extracts spatial features and adds positional encoding through a convolutional neural network, preserving the spatial structure information of the image; it obtains an image token sequence through a flattening operation, providing fine-grained local features for subsequent aggregation; and it achieves the transformation from pixel space to structured object representation through an object-centered encoder. Object-centered representation is naturally suitable for handling multi-object interaction scenarios, providing a favorable representational foundation for reducing non-physical phenomena such as object overlap and collision errors.

[0057] In an optional embodiment, step S120 above, "projecting the world state at the current moment into a phase space representation including configuration variables and momentum variables," may include steps S120-1 to S120-2: Step S120-1: Input the object slots in the current world state into the attention network to obtain the configuration variables.

[0058] In this step, the self-attention network interactively models multiple object slots at the current time. The self-attention mechanism allows each object slot to exchange information with all other object slots, thereby capturing the spatial structural relationships and relative layouts between objects. For example, in a multi-object scene, the positional relationship, occlusion relationship, or support relationship between two adjacent objects can be encoded through self-attention. The output of the self-attention network is a configuration variable, with the same dimensions as the input, and the features of each object slot incorporate information from other objects. The configuration variable physically corresponds to generalized coordinates in a Hamiltonian system, primarily describing the configuration state of the system at a given time.

[0059] For example, configuration variables It can be represented as:

[0060] in, This represents a self-attention operation, which can model the interaction relationships between objects while keeping the feature dimensions unchanged.

[0061] Step S120-2: Input the object slots in the current world state and the set of historical world states into the cross-attention network to obtain the momentum variable.

[0062] In this step, the historical world state set refers to the world state at several past moments. The attention network extracts motion trends relevant to the current moment from historical information. Object slots from the current moment's world state are queried, with keys and values ​​derived from the historical world state set. The output is a momentum variable with the same dimension as the input, physically corresponding to the generalized momentum in a Hamiltonian system, primarily describing the system's motion trend and rate of change.

[0063] For example, momentum variable It can be represented as:

[0064] in, Indicates cross-attention operations; It represents the set of historical world states, which is used to extract dynamic change information in the time dimension.

[0065] By incorporating historical information, momentum variables can capture dynamic characteristics over time, providing a basis for subsequent energy calculations and evolution updates.

[0066] The technical solution in this embodiment generates configuration variables and momentum variables through self-attention and cross-attention respectively, achieving a structured and functional distinction in phase space construction. Self-attention focuses on the spatial structural relationships of the current frame, while cross-attention focuses on historical temporal information. This design conforms to the phase space definition of Hamiltonian mechanics, enhances the physical interpretability of the model, and can flexibly adapt to scenarios with different numbers of objects.

[0067] In an optional embodiment, step S120 above, "defining the evolution direction using the Hamiltonian equation based on the gradient of the total system energy calculated from the phase space representation," may include steps S120-3 to S120-8: Step S120-3: Concatenate the configuration variable and the momentum variable along the feature dimension to obtain the phase space input representation.

[0068] In this step, the configuration variables and momentum variables are concatenated along the feature dimension to obtain a phase space input representation that simultaneously contains spatial configuration information and motion trend information, i.e. The splicing operation maintains the correspondence at the object level, meaning that the splicing vector for each object contains both the object's configuration and momentum information.

[0069] Step S120-4: Apply an object-level self-attention mechanism to the phase space input representation to obtain phase space features containing inter-object interaction information.

[0070] In this step, the phase space input is represented as the input object-level self-attention network. The self-attention mechanism allows different objects to exchange information, enabling the features of each object to incorporate the influence of other objects. Since the phase space input contains both configuration and momentum, after self-attention, the output phase space features not only capture the interaction between objects in spatial structure, but also incorporate the mutual influence in motion trends (such as momentum coupling between two relatively moving objects).

[0071] For example, phase space features It can be represented as:

[0072] Step S120-5: For each object, calculate the energy contribution of the object by passing the corresponding part of the phase space features through a learnable scalar mapping function.

[0073] In this step, the phase space features are split into N segments, each segment corresponding to one object. For each object, its phase space features are mapped to a scalar value, i.e., the object's energy contribution, through a learnable scalar mapping function (such as a multilayer perceptron). Each object can share the same mapping function structure, thus enabling the model to handle any number of objects in a parametrically efficient manner.

[0074] For example, the energy contribution of an object can be represented as:

[0075] in, Indicates the first The characteristic representation of an object in phase space This represents a learnable function that maps vectors to scalars. Indicates the first Energy contribution of each object.

[0076] Step S120-6: Sum the energy contributions of all objects to obtain the total energy of the system.

[0077] In this step, the total system energy reflects the total mechanical energy of the system in the current phase space state, and is additive and differentiable. Since the total system energy is differentiable, the gradient of the total energy with respect to the configuration and momentum variables can be calculated using automatic differentiation techniques.

[0078] For example, the total energy of the system It can be represented as:

[0079] Step S120-7: Calculate the gradient of the total energy of the system with respect to the configuration variable and the gradient with respect to the momentum variable.

[0080] In this step, since the configuration variables and momentum variables are two components of the phase space representation, and the total energy is a differentiable function obtained through neural network parameterization, its gradient can be calculated precisely. Using an automatic differentiation mechanism, the partial derivatives of the system's total energy with respect to the configuration variables are calculated (…). ) and partial derivatives of momentum variables ( The partial derivatives of the total energy with respect to the configuration variables and the momentum variables represent the rates of change (gradients) of the total energy in the configuration space and momentum space, respectively. These gradients will be used in the Hamiltonian equations to define the direction of evolution.

[0081] Step S120-8: According to the Hamiltonian equation, the evolution direction of the configuration variable is defined as the gradient of the total energy of the system with respect to the momentum variable, and the evolution direction of the momentum variable is defined as the negative value of the gradient of the total energy of the system with respect to the configuration variable.

[0082] In this step, the Hamiltonian equation is applied to define the evolution direction of the phase space state. Specifically, the evolution direction of the configuration variable is equal to the gradient of the total energy with respect to the momentum variable, and the evolution direction of the momentum variable is equal to the negative gradient of the total energy with respect to the configuration variable, that is:

[0083] in, It indicates the instantaneous direction of change of the configuration variable at the current moment (i.e., the direction of evolution of the configuration variable). This represents the instantaneous direction of change of the momentum variable at the current moment (i.e., the direction of evolution of the momentum variable). This equation ensures that the evolution process satisfies geometric properties such as energy conservation and invariant phase space volume.

[0084] The technical solution of this embodiment, by concatenating configuration and momentum variables and applying object-level self-attention, enables the energy function to model the complex interactions between objects, thereby more accurately reflecting the overall dynamic characteristics of a multi-object system. Specifically, a learnable scalar mapping function is used to map the phase space features of each object to energy contributions, and the total system energy is obtained by summing, making the energy function additive and differentiable. This retains the physical property that energy is a scalar and decomposable quantity while giving the neural network sufficient expressive power. Based on the Hamiltonian equation, the evolution direction is defined using the total energy gradient, and the physical inductive bias is explicitly incorporated into the state update process, allowing the model to still follow the laws of energy conservation and geometric structure preservation even without explicit physical equations. This provides a physically consistent evolution direction for subsequent discrete-time evolution using symplectic integrals.

[0085] In an optional embodiment, step S120 above, "updating the evolution direction in discrete time using the symplectic integral method to generate the first candidate world state," may include steps S120-9 to S120-12: Step S120-9: Perform a half-step update on the momentum variable based on the gradient of the total energy of the system with respect to the current configuration variable.

[0086] This step is the first stage of the leapfrog symplectic integral. Let the current time be t, and the current configuration variable be... The current momentum variable is Based on the total energy of the system Regarding the gradient of the current configuration variable, update the momentum variable by half a time step along the negative gradient direction. ,Right now:

[0087] in, This refers to the momentum variable after a half-step update (i.e., half-step momentum). A half-step update means that the change in the momentum variable only completes half of the entire time step, providing an intermediate state for the subsequent complete update of the configuration variable.

[0088] Step S120-10: Perform a one-step full update on the configuration variable based on the gradient of the total energy of the system with respect to the momentum variable after the half-step update.

[0089] This step uses the momentum variable updated in half a step to update the configuration variable. First, the total energy of the system is calculated. Regarding the gradient of the momentum variable after the half-step update. Then, perform a full-step update on the configuration variable, i.e.:

[0090] in, These are the updated configuration variables. Because half-step momentum is used, this update can more accurately reflect the dynamic state at intermediate time points, thereby improving integration accuracy.

[0091] Step S120-11: Based on the gradient of the total energy of the system with respect to the updated configuration variable, perform another half-step update on the momentum variable to obtain the updated momentum variable.

[0092] This step completes the final half-update of the frog jump integral. The updated configuration variables are used. Based on this, calculate the total energy of the system. The gradient with respect to the new configuration variable. Then, from the half-step momentum. Starting from there, update the step size again along the negative gradient direction by half a step, i.e.:

[0093] in, This is the updated momentum variable. At this point, the momentum variable has also completed a full time step update.

[0094] Step S120-12: Concatenate the updated configuration variables and the updated momentum variables, and map them back to the object slot space through the projection function to obtain the first candidate world state.

[0095] This step will update the configuration variables. and the updated momentum variable The image is concatenated along the feature dimensions to reconstruct the phase space representation. Then, it is projected using a lightweight projection function. (e.g., linear transformation or single-layer / multilayer perceptron), mapping the spliced ​​phase space representation back to the object slot space, yields the first candidate world state. ,Right now:

[0096] Wherein, projection function It is learnable and can adaptively convert phase space information back to object-centric representation based on training data, while maintaining object-level correspondence.

[0097] The technical solution employed in this embodiment preserves the geometric structure of the Hamiltonian system, suppresses energy drift and numerical error accumulation, thereby improving the long-term stability of predictions. The leapfrog integration splits the momentum variable update into two half-steps, inserting a complete update of the configuration variable in between. This staggered approach improves integration accuracy without increasing computational complexity. The projection function enables seamless integration with subsequent modules. The entire update process is fully differentiable, supporting end-to-end training.

[0098] In an optional embodiment, step S130 above, "extracting temporal dynamic features from the memory structure storing historical world states along the time dimension," may include steps S130-1 to S130-4: Step S130-1: Use the most recent preset number of historical world states as short-term memory.

[0099] In this step, short-term memory is used to store the world state of the most recent K time steps. A sliding window update method is used to provide the model with local temporal information that is sensitive to current dynamic changes.

[0100] For example, short-term memory It can be represented as:

[0101] in, This represents the world state at the K times preceding the current time t.

[0102] Step S130-2: Maintain long-term memory through a recursive update rule, wherein the recursive update rule weighted and merges the long-term memory of the previous moment with the world state of the previous moment to obtain the long-term memory of the current moment.

[0103] In this step, long-term memory is used to store the global evolutionary trend throughout the entire historical process, and its update method employs an exponential moving average. Specifically, the long-term memory at the current moment is obtained by weighted fusion of the long-term memory from the previous moment and the world state from the previous moment, that is:

[0104] in, This is the smoothing coefficient for long-term memory. This recursive rule allows long-term memory to retain all historical information in a decaying manner, with older information having a smaller impact and more recent information having a larger impact. Long-term memory does not need to store the complete historical sequence; it only needs to maintain a memory vector (or a set of object slots) with the same dimension as a single world state. It is memory efficient and can capture evolutionary patterns over long time scales.

[0105] Step S130-3: Using the current world state as a query, read short-term and long-term time features from the short-term memory and the current long-term memory respectively through a cross-attention mechanism.

[0106] In this step, a cross-attention mechanism is used to extract temporal information related to the current world state from two memory structures. Specifically, the current world state is used as the query, and short-term memory is used as the key and value. The short-term temporal features are obtained by weighted aggregation through attention weights. Similarly, the long-term temporal features are obtained by weighted aggregation through attention weights using the current world state as the query and long-term memory as the key and value.

[0107] For example, short-term time characteristics and long-term time characteristics They are represented as follows:

[0108] For short-term memory, since memory contains states from multiple historical moments, cross-attention can focus on the historical moment most relevant to the current state; for example, the current object's direction of motion may resemble the motion pattern of a certain historical moment. For long-term memory, since memory is a single compressed representation, cross-attention essentially aligns the current state with long-term memory, extracting global evolutionary trends.

[0109] Step S130-4: Fuse the short-term time features with the long-term time features to obtain the time dynamic features.

[0110] In this step, short-term and long-term time features are merged into a unified temporal dynamic feature. The fusion can be achieved by adding corresponding elements, or by concatenating the features and then performing a linear transformation or weighted summation.

[0111] For example, time dynamics features It can be represented as:

[0112] By fusing these features, the temporal dynamics features simultaneously incorporate recent high-resolution dynamic details and long-term evolutionary trend information, enabling the model to respond quickly to mutations while maintaining global consistency.

[0113] The technical solution adopted in this embodiment constructs two complementary temporal memory structures, short-term memory and long-term memory, and uses a cross-attention mechanism to extract temporal dynamic features. Short-term memory captures rapid dynamic changes, long-term memory achieves compressed storage of historical information, and the cross-attention mechanism adaptively selects relevant information. The fused features provide rich temporal information for subsequent adaptive updates.

[0114] In an optional embodiment, step S130 above, "extracting spatial dynamic features from the spatial relationship between the plurality of object slots along the spatial dimension," may include steps S130-5 to S130-7: Step S130-5: Input the object slots in the current world state into the attention network to obtain inter-object interaction features used to characterize the interactions between objects.

[0115] In this step, the self-attention network captures the spatial structural relationships and mutual influences between objects, outputs the interaction features between objects, and reflects the dynamic changes determined by the object layout and spatial relationships.

[0116] For example, inter-object interaction features It can be represented as:

[0117] Step S130-6: Input the object slots in the world state at the current moment and the image token of the current frame into the cross-attention network to obtain observation alignment features used to characterize observation conditional dependencies.

[0118] In this step, an alignment relationship between the object slot and the original observation is established using a cross-attention mechanism. The object slot from the world state at the current moment is queried, and the key and value come from the image token of the current frame (i.e., the token sequence obtained by flattening the spatial feature map in step S120-2). The image token preserves the original pixel-level spatial information, including details such as texture, edges, and color.

[0119] Through cross-attention, each object slot can aggregate its associated local visual features from the image token, thereby acquiring environmental information (e.g., illumination variations, surface material, occlusion boundaries, etc.) under the current observation conditions. For example, observation-aligned features... It can be represented as:

[0120] Step S130-7: Fuse the inter-object interaction features with the observation alignment features to obtain the spatial dynamics features.

[0121] In this step, the interaction features between objects and the observation alignment features are merged into a unified spatial dynamics feature. The fusion method can include adding corresponding elements, performing a linear transformation after concatenation, or learning-gated weighting.

[0122] For example, space dynamics characteristics It can be represented as:

[0123] Through fusion, spatial dynamics features encompass both the structured effects of interactions between objects and fine-grained environmental information driven by current observations.

[0124] The technical solution of this embodiment uses self-attention modeling to model complex spatial relationships in multi-object scenes, cross-attention to compensate for local features that may be lost during object slot encoding, and the fused features provide comprehensive spatial information input for generating compensation update components.

[0125] In an optional embodiment, step S130 above, "generating a second candidate world state for compensating non-conservative perturbations based on the temporal and spatial dynamic characteristics," may include steps S130-8 to S130-9: Step S130-8: Combine the current world state, the temporal dynamics feature, and the spatial dynamics feature to obtain the fused feature.

[0126] In this step, the current world state is... Time-dynamic characteristics Space dynamics characteristics These three pieces of information are concatenated to obtain a fused feature. This fused feature simultaneously includes the current state, historical evolution trend, spatial structural relationships, and observation dependency information, providing rich input for the subsequent generation of adaptive update components.

[0127] Step S130-9: Input the fused features into a multilayer perceptron, and use the multilayer perceptron to map the object-level update component with the same dimension as the world state, which is then used as the second candidate world state.

[0128] In this step, the Multilayer Perceptron (MLP) maps the high-dimensional fused features back to the original object slot space, outputting an adaptive update component for each object. This component is used to characterize non-conservative effects that are difficult for the Hamiltonian physics prior module to cover. That is, the second candidate world state is represented as: .

[0129] The technical solution adopted in this embodiment improves the accuracy and rationality of the compensation amount by fusing three complementary types of information: current world state, temporal dynamics characteristics, and spatial dynamics characteristics. The multilayer perceptron, as a general function approximator, eliminates the need for manual design of compensation rules. The output second candidate world state has the same dimension as the first candidate world state, which facilitates subsequent fusion operations and achieves seamless integration of structured backbone and environmental adaptive compensation.

[0130] In an optional embodiment, the method is implemented using a sequence prediction model, which includes at least an object center encoding module, a Hamiltonian physical prior module, and a spatiotemporal adaptive decoupling module; the sequence prediction model is trained according to steps A1 to A6: Step A1: Obtain training data, which includes observed image sequences and corresponding real future image sequences.

[0131] In this step, each training sample contains a continuous image sequence. The first few frames are used as the observed image sequence (input), and the subsequent few frames are used as the real future image sequence (supervised target). For example, given a video clip of length T+K, the first T frames are taken as the observed sequence, and the last K frames are taken as the future sequence. The training data covers various scenes, numbers of objects, and motion patterns.

[0132] Step A2: Input the observed image sequence into the sequence prediction model, perform multi-step prediction, and obtain the predicted future image sequence and the corresponding predicted world state sequence.

[0133] In this step, each frame in the observation sequence is sequentially converted into a world state representation through the object-centered encoding module; starting with the world state of the last frame of the observation sequence, the Hamiltonian physics prior module and the spatiotemporal adaptive decoupling module are recursively called to perform multi-step prediction; the outputs of all prediction steps are collected to obtain the predicted future image sequence and the corresponding predicted world state sequence.

[0134] Step A3: Obtain the real-world state sequence obtained by encoding the real future image sequence through the object center encoding module.

[0135] In this step, the real future image sequence is input frame by frame into the object-centered encoding module to obtain the real-world state sequence. This encoding module is the same as the module used to encode the observed images in step A2, and they share parameters, serving as a supervisory target for predicting the world state sequence.

[0136] Step A4: Calculate the slot-level mean squared error loss between the predicted world state sequence and the real world state sequence.

[0137] In this step, for each prediction step m (m equals 1 to M), the predicted world state is calculated. With the state of the real world The mean squared error between them. Specifically, for each object slot and each feature dimension, the squared difference is calculated, summed, and averaged. The loss from all prediction steps is then summed to obtain the slot-level mean squared error loss. ,Right now:

[0138] This loss forces the model to make accurate evolutionary predictions in the object-centric representation space, thereby learning physically consistent state transition rules.

[0139] Step A5: Calculate the image-level mean squared error loss between the predicted future image sequence and the actual future image sequence.

[0140] In this step, for each prediction step m, the predicted image frame is calculated. With real image frames The mean squared error between each prediction step is calculated by averaging the squared differences of each pixel. The sum of the losses from all prediction steps yields the image-level mean squared error loss. ,Right now:

[0141] This loss ensures that the decoder can correctly reconstruct the predicted world state into a visually realistic image, while also indirectly constraining the prediction quality of the world state.

[0142] Step A6: Use the weighted sum of the slot-level mean squared error loss and the image-level mean squared error loss as the overall loss to jointly optimize the parameters of the sequence prediction model.

[0143] Among them, the overall loss It can be represented as:

[0144] in, These are the weighting coefficients that balance the two losses. Using backpropagation, the gradient of the overall loss with respect to all trainable parameters of the model (including the object center encoding module, the energy and projection functions in the Hamiltonian physics prior module, the spatiotemporal adaptive decoupling module, the decoder, etc.) is calculated, and an optimizer (such as Adam) is used to update the parameters. During training, the model simultaneously learns how to encode images into object slots, how to perform structure-preserving evolution in the latent space, how to compensate for non-conservative perturbations, and how to decode the predicted state into an image. Joint optimization enables the modules to adapt to each other, collectively improving long-term prediction performance.

[0145] The technical solution adopted in this embodiment uses slot-level loss to directly supervise the evolution of world state in the latent space, avoiding representation drift that may be caused by relying solely on pixel space supervision; image-level loss ensures the visual realism of the predicted image; joint optimization enables each module to adapt to each other and jointly improve long-term prediction performance.

[0146] like Figure 2 As shown, Figure 2 This is an overall architecture diagram of a Hamiltonian physical prior world model method for long-term prediction provided in an embodiment of this application. The method mainly includes three stages: encoding stage, evolution stage, and decoding stage.

[0147] Encoding phase: Observation sequence ( Each frame of the image is sequentially input into the object-centered encoder, where spatial feature maps are extracted by a convolutional neural network and positional encoding is added. These are then aggregated into a fixed number of object slots via a slot attention mechanism, forming the world state of the object center. ).

[0148] Evolutionary stage: The world state is simultaneously fed into two parallel paths. The Hamiltonian physics prior module projects it as configuration variables in phase space. ) and momentum variables ( The total energy of the system is calculated using a learnable energy function. And using the Hamiltonian equation, the evolution direction is defined by the energy gradient (i.e., Discrete updates are performed using leapfrog symplectic integrals, i.e., a half-step update is first performed on the momentum variable. Then perform a full update on the configuration variables ( ), Perform another half-step update on the momentum variable ( Finally, the projection function maps the result back to the object slot space, generating the first candidate world state. ).

[0149] The spatiotemporal adaptive decoupling module extracts temporal dynamic features from short-term and long-term memory along the time dimension. ), extracting inter-object interaction features and observation alignment features along the spatial dimension and fusing them to obtain spatial dynamic features ( The current world state is concatenated with the two types of features and then input into a multilayer perceptron to generate a second candidate world state. The two are then combined to obtain the predicted world state for the next moment. ).

[0150] Decoding phase: The predicted world state is fed into the object-centered decoder, which maps the object slots back to pixel space to generate the predicted image frame. The predicted world state is used as the input for the next moment, and the evolution and decoding process is executed recursively to generate a sequence of predicted images for multiple future steps.

[0151] Thus, by encoding the observation sequence into the world state at the object center and integrating the Hamiltonian physical prior module and the spatiotemporal adaptive decoupling module, the unity of stability, physical consistency and environmental adaptability in long-term prediction is achieved.

[0152] The technical solution of this application can be applied to tasks requiring long-term dynamic modeling, such as video prediction, robot environmental perception, embodied intelligence, autonomous driving, physical simulation and digital twins, and multi-object trajectory prediction. The specific form of the observation sequence varies in different application scenarios; for example, it is continuous video frames in video prediction, real-time images captured by cameras in robot perception, and road image sequences in autonomous driving. This solution can generate future frames or motion trajectories that conform to physical laws according to specific scenario requirements, effectively reducing problems such as prediction ambiguity, object deformation, and unreasonable trajectories.

[0153] The above application scenarios are merely illustrative examples and are not intended to limit the scope of this application. This solution can be used for any task that requires long-term prediction of dynamic environments and can encode observation information into object-centered representations. Hyperparameters such as observation preprocessing methods, number of object slots, and memory window length can be adaptively adjusted according to specific needs.

[0154] This application also provides a Hamiltonian physical prior world model system for long-term prediction, referring to... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a Hamiltonian physical prior world model system for long-term prediction provided in an embodiment of this application. The system includes: The object center encoding module 310 is used to encode each frame of the image in the observation sequence into a world state of the object center, wherein the world state is used to describe multiple object slots of different objects in the scene. The Hamiltonian physics prior module 320 is used to project the current world state into a phase space representation containing configuration variables and momentum variables. Based on the gradient of the total energy of the system calculated from the phase space representation, the evolution direction is defined by the Hamiltonian equation, and the evolution direction is updated in discrete time using the symplectic integral method to generate a first candidate world state. Herein, the configuration variables represent the structural relationships between objects, and the momentum variables represent the historical evolution trend of the objects. The spatiotemporal adaptive decoupling module 330 is used to extract temporal dynamic features from the memory structure storing historical world states along the time dimension, extract spatial dynamic features from the spatial relationship between the multiple object slots along the spatial dimension, and generate a second candidate world state to compensate for non-conservative perturbations based on the temporal dynamic features and spatial dynamic features. The state fusion module 340 is used to fuse the first candidate world state and the second candidate world state to obtain the predicted world state at the next moment. The object center decoding module 350 is used to decode the predicted world state of the next moment into a predicted image frame; The recursive prediction module 360 ​​is used to recursively call the Hamiltonian physical prior module, the spatiotemporal adaptive decoupling module, the state fusion module and the object center decoding module to generate a sequence of predicted images for multiple future steps.

[0155] It is understood that the Hamiltonian physics prior world model system for long-term prediction in the embodiments of this application can implement the Hamiltonian physics prior world model method for long-term prediction in the above embodiments. The Hamiltonian physics prior world model system for long-term prediction has the same advantages as the Hamiltonian physics prior world model method for long-term prediction compared with the prior art, and will not be repeated here.

[0156] This application also provides an electronic device, see embodiments thereof. Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device 400 includes a memory 410 and a processor 420. The memory 410 and the processor 420 are connected via a bus for communication. The memory 410 stores a computer program that can run on the processor 420 to implement the steps of the Hamiltonian physical prior world model method for long-term prediction described in the embodiments of this application.

[0157] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the Hamiltonian physical prior world model method for long-term prediction described in this application.

[0158] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the Hamiltonian physical prior world model method for long-term prediction described in this application.

[0159] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0160] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0164] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0165] The foregoing has provided a detailed description of the Hamiltonian physical prior world model method and system for long-term prediction provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A Hamiltonian physical prior world model method for long-term prediction, characterized in that, include: Each frame of the observation sequence is encoded as a world state centered on an object, the world state including multiple object slots for describing different objects in the scene; The world state at the current moment is projected into a phase space representation containing configuration variables and momentum variables using the Hamiltonian physics prior module. Based on the gradient of the total energy of the system calculated from the phase space representation, the evolution direction is defined by the Hamiltonian equation, and the evolution direction is updated in discrete time using the symplectic integral method to generate the first candidate world state. Herein, the configuration variables represent the structural relationships between objects, and the momentum variables represent the historical evolution trend of the objects. The spatiotemporal adaptive decoupling module extracts temporal dynamic features from the memory structure storing historical world states along the time dimension and extracts spatial dynamic features from the spatial relationships between the multiple object slots along the spatial dimension. Based on the temporal dynamic features and spatial dynamic features, a second candidate world state is generated to compensate for non-conservative perturbations. By fusing the first candidate world state and the second candidate world state, the predicted world state for the next time step is obtained; Decode the predicted world state at the next moment into a predicted image frame; The above steps are executed recursively to generate a sequence of predicted images for future steps.

2. The method according to claim 1, characterized in that, Encode each frame of the observation sequence as the world state centered on the object, including: Spatial feature maps of each frame of image are extracted using a convolutional neural network, and positional encoding is added to the spatial feature maps. Flatten the spatial feature map with added location encoding into multiple image tokens; The multiple image tokens are aggregated into a preset number and dimension of object slots using an object-centric encoder.

3. The method according to claim 1, characterized in that, Projecting the current world state into a phase space representation that includes configuration variables and momentum variables includes: The object slots in the current world state are input into the attention network to obtain the configuration variables; The momentum variable is obtained by inputting the object slots in the current world state and the set of historical world states into the cross-attention network.

4. The method according to any one of claims 1-3, characterized in that, The evolution direction is defined by the Hamiltonian equation based on the gradient of the total system energy calculated from the phase space representation, including: The configuration variables and momentum variables are concatenated along the feature dimension to obtain the phase space input representation; An object-level self-attention mechanism is applied to the phase space input representation to obtain phase space features containing information about interactions between objects; For each object, the energy contribution of that object is calculated by passing the corresponding part of the phase space features through a learnable scalar mapping function; The total energy of the system is obtained by summing the energy contributions of all objects. Calculate the gradient of the total energy of the system with respect to the configuration variable and the gradient with respect to the momentum variable; According to the Hamiltonian equation, the evolution direction of the configuration variable is defined as the gradient of the total energy of the system with respect to the momentum variable, and the evolution direction of the momentum variable is defined as the negative value of the gradient of the total energy of the system with respect to the configuration variable.

5. The method according to claim 1, characterized in that, The evolution direction is updated in discrete time using a symplectic integral method to generate a first candidate world state, including: Based on the gradient of the total energy of the system with respect to the current configuration variable, perform a half-step update on the momentum variable; Based on the gradient of the total energy of the system with respect to the momentum variable after the half-step update, a one-step full update is performed on the configuration variable; Based on the gradient of the total energy of the system with respect to the updated configuration variable, another half-step update is performed on the momentum variable to obtain the updated momentum variable; The updated configuration variables and the updated momentum variables are concatenated and mapped back to the object slot space through a projection function to obtain the first candidate world state.

6. The method according to claim 1, characterized in that, Extracting temporal dynamic features from memory structures storing historical world states along the time dimension, including: Use the most recently preset number of historical world states as short-term memory; Long-term memory is maintained by a recursive update rule, which weights and fuses the long-term memory of the previous moment with the world state of the previous moment to obtain the long-term memory of the current moment. Using the current world state as a query, short-term and long-term time features are read from the short-term memory and the current long-term memory respectively through a cross-attention mechanism. The short-term time features are fused with the long-term time features to obtain the time dynamic features.

7. The method according to claim 2, characterized in that, Extracting spatial dynamic features from the spatial relationships between the multiple object slots along the spatial dimension, including: The object slots in the current world state are input into the attention network to obtain inter-object interaction features used to characterize the interactions between objects; The object slots in the current world state and the image token of the current frame are input into the cross-attention network to obtain observation alignment features that characterize observation conditional dependencies. The spatial dynamics features are obtained by fusing the inter-object interaction features with the observation alignment features.

8. The method according to claim 6 or 7, characterized in that, Based on the aforementioned temporal and spatial dynamic characteristics, a second candidate world state is generated to compensate for non-conservative perturbations, including: The current world state, the temporal dynamics feature, and the spatial dynamics feature are spliced ​​together to obtain the fused feature; The fused features are input into a multilayer perceptron, and the object-level update component with the same dimension as the world state is obtained by mapping through the multilayer perceptron, which serves as the second candidate world state.

9. The method according to claim 1, characterized in that, The method is implemented through a sequence prediction model, which includes at least an object center encoding module, a Hamiltonian physical prior module, and a spatiotemporal adaptive decoupling module; the sequence prediction model is trained according to the following steps: Acquire training data, which includes observed image sequences and corresponding real future image sequences; The observed image sequence is input into the sequence prediction model, and multi-step prediction is performed to obtain the predicted future image sequence and the corresponding predicted world state sequence. Obtain the real-world state sequence obtained by encoding the real future image sequence through the object center encoding module; Calculate the slot-level mean squared error loss between the predicted world state sequence and the real world state sequence; Calculate the image-level mean squared error loss between the predicted future image sequence and the actual future image sequence; The weighted sum of the slot-level mean squared error loss and the image-level mean squared error loss is used as the overall loss to jointly optimize the parameters of the sequence prediction model.

10. A Hamiltonian physical prior world model system for long-term prediction, characterized in that, include: The object center encoding module is used to encode each frame of the observation sequence into a world state of the object center, wherein the world state is used to describe multiple object slots of different objects in the scene. The Hamiltonian physics prior module projects the current world state into a phase space representation containing configuration variables and momentum variables. Based on the gradient of the total system energy calculated from the phase space representation, the evolution direction is defined by the Hamiltonian equation, and the evolution direction is updated in discrete time using the symplectic integral method to generate a first candidate world state. Herein, the configuration variables represent the structural relationships between objects, and the momentum variables represent the historical evolution trend of the objects. The spatiotemporal adaptive decoupling module is used to extract temporal dynamic features from the memory structure storing historical world states along the time dimension, extract spatial dynamic features from the spatial relationship between the multiple object slots along the spatial dimension, and generate a second candidate world state to compensate for non-conservative perturbations based on the temporal dynamic features and spatial dynamic features. The state fusion module is used to fuse the first candidate world state and the second candidate world state to obtain the predicted world state at the next moment. The object-centered decoding module is used to decode the predicted world state of the next moment into a predicted image frame; The recursive prediction module is used to recursively call the Hamiltonian physics prior module, the spatiotemporal adaptive decoupling module, the state fusion module, and the object center decoding module to generate a sequence of predicted images for multiple future steps.

Images