An end-to-end unified perception method and device for dynamic three-dimensional scenes

Abstract

The application discloses an end-to-end unified perception method and device for a dynamic three-dimensional scene, and belongs to the fields of computer three-dimensional geometry reconstruction and computer motion estimation. Firstly, a continuous video sequence input image encoder is used to construct a multi-modal feature sequence containing geometry, camera and motion tokens; cross-frame interaction modeling is performed through an alternating attention mechanism to generate intermediate representation fusing geometry structure and motion information; the representation is used to dynamically modulate geometry features based on the motion tokens, and a pixel-level three-dimensional motion field is generated through conditional decoding; finally, camera parameters, a depth map, a point cloud map and the three-dimensional motion field are jointly output through a single forward propagation to realize unified representation of geometry and motion. The application overcomes the limitation of separate geometry and motion estimation in traditional methods, realizes efficient end-to-end perception, and significantly improves the integrity and accuracy of three-dimensional scene perception.

Description

Technical Field

[0001] This invention relates to the fields of computer three-dimensional geometric reconstruction and computer motion estimation, and in particular to an end-to-end unified perception method, apparatus, device and storage medium for dynamic three-dimensional scenes. Background Technology

[0002] 3D reconstruction technology is a core research direction in the field of computer vision, and it has significant value for applications such as augmented reality, autonomous driving, robot navigation, and digital twins. One key aspect is the direct extraction of 3D geometry and motion from videos. Traditional 3D reconstruction methods primarily rely on Structured Motion (SfM) and Multi-View Synthesis (MVS) techniques. These methods typically require precise camera calibration parameters and complex optimization processes, resulting in low reconstruction efficiency and strong constraints on the input image.

[0003] In recent years, deep learning-based 3D reconstruction methods have made significant progress. A series of methods have achieved substantial improvements in robustness and efficiency, with DUSt3R being a representative example. DUSt3R can regress dense point clouds from uncalibrated image pairs lacking pose information and recover accurate scene geometry in a single forward propagation. Subsequent work has extended the DUSt3R framework to multi-view scenes, achieving more efficient batch reconstruction and streaming reconstruction.

[0004] However, the aforementioned methods are primarily designed for static scenes and face significant challenges when dealing with dynamic scenes. Dynamic scene reconstruction requires capturing both geometric and motion information simultaneously, but existing technologies suffer from the following key issues: First, there is a lack of a unified framework capable of handling both geometric and motion information simultaneously; second, most methods require additional post-optimization steps, leading to low computational efficiency; third, the motion and geometry of objects in dynamic scenes interact, and existing methods struggle to effectively model this complex relationship; fourth, existing methods impose strong constraints on the input image, making it difficult to handle dynamic scenes with significant viewpoint changes and numerous occlusions.

[0005] Therefore, there is an urgent need for a framework capable of efficiently and accurately reconstructing dense 3D dynamic scenes from dynamic video sequences. This framework should be able to directly output a unified representation containing geometric structure and temporal motion information, avoiding complex post-optimization processes, while supporting the reconstruction of dynamic scenes. This invention proposes an innovative solution to address the above problems, achieving end-to-end unified perception of dense 3D geometry and motion. Summary of the Invention

[0006] The present invention aims to at least partially solve one of the technical problems in the related art.

[0007] To address this, this invention proposes an end-to-end unified perception method for dynamic 3D scenes. It utilizes continuous video sequences to construct a multimodal feature sequence incorporating geometry, camera data, and motion tokens. Through cross-frame interactive modeling, it generates an intermediate representation fusing geometric structure and motion information. Based on the motion tokens, it dynamically modulates the geometric features and finally generates a pixel-level 3D motion field through conditional decoding. This method achieves efficient end-to-end perception by jointly outputting camera parameters, depth maps, point cloud maps, and the 3D motion field in a single forward propagation, significantly improving the completeness and accuracy of 3D scene understanding.

[0008] Another objective of this invention is to provide an end-to-end unified sensing device for dynamic three-dimensional scenes.

[0009] The third objective of this invention is to provide a computer device.

[0010] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0011] To achieve the above objectives, this invention proposes an end-to-end unified perception method for dynamic 3D scenes, comprising:

[0012] S1, Input a continuous video sequence to the image encoder to construct a multimodal feature sequence containing geometric tokens, camera tokens and motion tokens; S2, cross-frame interactive modeling of the multimodal feature sequence is performed through alternating intra-frame attention layers and global attention layers to generate an intermediate representation that integrates geometric structure and motion information; S3, based on the intermediate representation, the motion token is used as a conditional signal to dynamically modulate the geometric features through adaptive layer normalization, and then the pixel-level three-dimensional motion field is generated by decoding through a conditional dense prediction head. S4, through a single forward propagation, jointly outputs camera parameters, depth maps, point cloud maps, and 3D motion fields of a continuous video sequence, achieving a unified representation of geometry and motion.

[0013] An end-to-end unified perception method for dynamic 3D scenes according to an embodiment of the present invention may also have the following additional technical features: In one embodiment of the present invention, the input of a continuous video sequence to an image encoder to construct a multimodal feature sequence including geometric tokens, camera tokens, and motion tokens includes: S11, extract image patch features for each frame using the DINO image encoder. ;in, This indicates the number of image patches in each frame. This represents the dimension of each feature vector; S12, initialize motion tokens for each frame. At that time, a query based on the target time is used. The dynamic position coding mechanism; among which, This represents the dimension of each feature vector. Indicates the number of sports tokens. t 1,..., tr This indicates a specific time value.

[0014] In one embodiment of the present invention, the step of performing cross-frame interactive modeling of the multimodal feature sequence through alternating intra-frame attention layers and global attention layers to generate an intermediate representation that integrates geometric structure and motion information includes: S21, the intra-frame attention layer calculates image patch features. Local correlation with geometric tokens generates intra-frame geometric consistency constraints; S22, Global Attention Layer Based on Motion Tokens With camera token Cross-frame interaction establishes a spatiotemporal motion-geometric coupling relationship.

[0015] In one embodiment of the present invention, the step of generating a pixel-level three-dimensional motion field based on the intermediate representation, using motion tokens as conditional signals to dynamically modulate geometric features through adaptive layer normalization, and then decoding with a conditional dense prediction head includes: S31, using AdaLN for geometric features Adjust parameters, scaling factor and offset factor By sports token Generated via a fully connected layer; The S32, DPTHead decoder employs a multilayer perceptron structure, achieving pixel-level motion field resolution through residual connections and feature pyramids. The gradual refinement; among them, Indicates the number of video clips. This indicates the frame number of each segment, and 3 indicates the dimension of the motion vector. Indicates the height of the sports field in space. It indicates the width of the sports field in space.

[0016] In one embodiment of the present invention, the joint output via a single forward propagation includes: S41, Three-dimensional sports field The representation is based on the world coordinate system rather than the camera coordinate system; among them, This represents the amount of motion along the x-axis in the world coordinate system. This represents the amount of motion along the y-axis in the world coordinate system. This represents the amount of motion along the z-axis in the world coordinate system. Represents pixel coordinates; S42, the output includes camera intrinsic and extrinsic parameters. Depth map Point cloud map With sports field Joint gradient consistency constraints.

[0017] In one embodiment of the present invention, it further includes: S5 executes a multi-stage progressive training strategy: S51, Phase 1, using supervised 3D motion loss Establish initial motion perception ability; S52, Phase 2 introduces geometric reprojection consistency loss and spatial gradient smoothing constraint; S53, Phase 3: Unfreeze all network modules and perform joint optimization of the geometry head and motion head.

[0018] To achieve the above objectives, another aspect of the present invention provides an end-to-end unified sensing device for dynamic three-dimensional scenes, comprising: The image encoding and multimodal feature construction module is used to input continuous video sequences into the image encoder and construct a multimodal feature sequence containing geometric tokens, camera tokens, and motion tokens; The intra-frame and global attention interaction module is used to perform cross-frame interactive modeling of the multimodal feature sequence through alternating intra-frame attention layers and global attention layers, generating an intermediate representation that integrates geometric structure and motion information. The motion condition geometry decoding module is used to generate a pixel-level three-dimensional motion field based on the intermediate representation, using motion tokens as conditional signals through adaptive layer normalization and dynamic modulation of geometric features, and then decoding them through a conditional dense prediction head. The Joint Output and Unified Representation module is used to jointly output camera parameters, depth maps, point cloud maps, and 3D motion fields of a continuous video sequence through a single forward propagation, thereby achieving a unified representation of geometry and motion.

[0019] In one embodiment of the present invention, it further includes: A multi-stage progressive training module is used to execute multi-stage progressive training strategies, including: Phase 1 uses supervised 3D motion loss Establish initial motion perception ability; Phase 2 introduces geometric reprojection consistency loss and spatial gradient smoothing constraint; Phase 3 involves unfreezing all network modules and performing joint optimization of the geometry head and motion head.

[0020] This invention discloses an end-to-end unified perception method and apparatus for dynamic 3D scenes. By constructing multimodal feature sequences and cross-frame interactive modeling, it achieves deep fusion and unified representation of geometric structure and motion information in 3D scenes. It effectively overcomes the technical limitations of traditional schemes that separate geometric estimation and motion analysis, and significantly improves the integrity and spatial consistency of 3D motion field reconstruction through an end-to-end joint optimization framework. It simultaneously outputs camera parameters, depth maps, point cloud maps, and 3D motion fields in a single forward propagation, not only enhancing the computational efficiency of the perception system but also ensuring geometric consistency between output results, providing a more accurate and reliable solution for understanding dynamic 3D scenes.

[0021] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing an end-to-end unified perception method for a dynamic three-dimensional scene as described in the first aspect embodiment.

[0022] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an end-to-end unified perception method for a dynamic three-dimensional scene as described in the first aspect embodiment.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of an end-to-end unified perception method for a dynamic three-dimensional scene according to an embodiment of the present invention; Figure 2 This is a model architecture diagram of an end-to-end unified perception method for dynamic three-dimensional scenes according to an embodiment of the present invention; Figure 3 This is a comparison diagram of an end-to-end unified perception method for dynamic 3D scenes according to an embodiment of the present invention and existing methods; Figure 4 This is a schematic diagram of the structure of an end-to-end unified sensing device for a dynamic three-dimensional scene according to an embodiment of the present invention. Figure 5 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] The following description, with reference to the accompanying drawings, describes an end-to-end unified perception method, apparatus, device, and storage medium for dynamic three-dimensional scenes according to embodiments of the present invention.

[0028] The core idea of ​​this invention is to construct a dynamic graph model covering multiple levels of the battery pack, from individual cells to clusters to the entire pack. This model integrates electrical connections and physical proximity to establish a spatial topology, and introduces temporal edges to characterize the system's dynamic operation. Based on this, an attribute-decoupled encoder is used to divide node features into health, anomaly, and trend subspaces. Orthogonal constraints and mutual information constraints are used to achieve feature decoupling and redundancy suppression. Furthermore, a cross-layer message passing mechanism establishes bidirectional information paths from bottom to top and from top to bottom. Gated attention is used to dynamically adjust the interaction intensity between levels, achieving deep fusion of multi-level features and collaborative updating of global representations. Finally, health status and lifetime prediction heads are set based on shared representations, and a joint loss function containing multiple constraints is constructed. This is combined with a sliding time window and edge weight update mechanism for integrated training. This transforms the traditionally separate state assessment and lifetime prediction into an intelligent prediction system that can fully exploit hierarchical relationships, suppress feature interference, and achieve multi-task collaborative optimization, significantly improving the accuracy, stability, and engineering applicability of battery pack health status and remaining lifetime predictions.

[0029] Example 1 To achieve the above invention, embodiments of the present invention provide an end-to-end unified perception method for dynamic 3D scenes, such as... Figure 1 As shown, it includes: S1, input a continuous video sequence to the image encoder to construct a multimodal feature sequence containing geometric tokens, camera tokens and motion tokens.

[0030] Specifically, in this step, the system first divides the length into... video sequence Each frame of the image The data is input into an image encoder based on the DINO architecture. This encoder divides each frame of image into fixed-size image patches and extracts the features of each image patch through a multi-layer Transformer encoder. ,in Indicates the number of image patches. Indicates the feature dimension.

[0031] Furthermore, based on image patch feature extraction, the system constructs three types of key tokens for each image frame: geometric tokens, camera tokens, and motion tokens. Geometric tokens encode 3D geometric information such as depth and point clouds; camera tokens aggregate and predict camera intrinsic and extrinsic parameters; and motion tokens explicitly carry temporal dynamic information, such as object displacement and rotation. These tokens are concatenated with image patch features in the sequence dimension to form a multimodal feature sequence, which serves as the input to the subsequent geometry-aware backbone network.

[0032] Specifically, camera token Typically composed of learnable embedding vectors, used for initializing and optimizing camera pose; motion tokens It consists of multiple learnable temporal embeddings, among which The number of categories of motion patterns or the number of samples of motion vectors are used to model the motion trajectory of dynamic objects; geometric tokens are generated through the interaction of image patch features and camera parameters to support the prediction of depth maps and point clouds.

[0033] Specifically, by unifying geometric, camera, and motion information into a multimodal token sequence, the model can establish spatiotemporal consistency during cross-frame interactions, thus providing structured input for subsequent intra-frame attention and global attention layers. This design avoids the two-stage "reconstruction first, then tracking" process in traditional methods, enabling early fusion of geometric and motion information in the feature space, improving the model's perception efficiency and robustness in dynamic scenes. Furthermore, this step provides a unified feature representation for the subsequent conditional dense prediction head, allowing motion information to be injected into geometric features in a differentiable manner, achieving end-to-end prediction of pixel-level 3D motion fields.

[0034] Furthermore, S1 includes: S11, extract image patch features for each frame using the DINO image encoder. ;in, This indicates the number of image patches in each frame. This represents the dimension of each feature vector.

[0035] Specifically, this step is a key input processing step in the entire MotionVGGT model architecture, and its technical implementation is based on the image block encoding mechanism of Visual Transformer (ViT).

[0036] Specifically, input image First, the image is divided into fixed-size patches, typically... or Pixels, forming Image patches ( (where the block size is [size]). Each image block is linearly embedded as [size]. A dimensional vector, where The value is typically set to 384 or 768, depending on the version of the DINO model used (such as the base or large model in DINOv2). The DINO encoder is pre-trained through self-supervised learning and can extract visual features with strong discriminative power, making it suitable for unsupervised or weakly supervised 3D reconstruction tasks.

[0037] Furthermore, the block features output by this step With camera token and sports tokens After being stitched together, the data serves as input to the geometry-aware backbone network. This design allows the model to integrate static geometry and dynamic motion information during the encoding stage, laying the foundation for subsequent cross-frame attention interactions and unified decoding. In practical applications, this step is suitable for continuous video input, especially in dynamic scenes with large changes in viewpoint and occlusion, providing robust feature representations and thus improving the accuracy of 3D motion field reconstruction.

[0038] Specifically, through this step, the present invention achieves efficient mapping from the original image to high-dimensional semantic features, providing a unified feature space for subsequent inter-frame modeling and motion prediction. Its technical value lies in: eliminating the need for external feature extraction modules, reducing system complexity, and enhancing the model's ability to perceive dynamic scenes through the strong representational capabilities of the DINO encoder.

[0039] S12, initialize motion tokens for each frame. At that time, a query based on the target time is used. The dynamic position coding mechanism; among which, This represents the dimension of each feature vector. Indicates the number of sports tokens. t 1,..., tr This indicates a specific time value.

[0040] Specifically, the core of this step lies in introducing a dynamic positional coding mechanism to initialize the motion token corresponding to each video frame. In particular, traditional static positional coding methods struggle to effectively capture complex motion patterns that change dynamically over time in video sequences. This step addresses this by pre-setting a set of target time queries. This query set defines the key or consecutive time points in the video clip that need to be modeled. Technically, the system dynamically generates high-dimensional location-encoded vectors based on these time queries and fuses this temporal information with a learnable motion token prototype, thereby initializing a unique motion token for each frame. This mechanism allows motion tokens to be deeply integrated with temporal context information from the very beginning of their generation, laying a solid foundation for subsequent cross-frame motion information interaction and modeling.

[0041] Furthermore, in practical applications of dynamic 3D scene perception, such as autonomous driving, robot navigation, and augmented reality, motion in the scene is often continuous and complex. This design directly enhances the model's ability to model long-term temporal dependencies and complex motion trajectories. Motion tokens initialized through dynamic position encoding guide the model to focus on motion changes at different time scales, effectively establishing spatiotemporal relationships between frames regardless of video sequence length. This allows the invention to maintain robust motion estimation performance even in complex scenes involving rapid acceleration, deceleration, or nonlinear motion, significantly improving the practicality and reliability of the technology in real-world applications.

[0042] Specifically, the key parameters involved in this step include the number of motion tokens. and the dimension of the feature vector and target time query The granularity. Among them, the number of motion tokens. It is a configurable hyperparameter that determines the complexity of the motion patterns or moving entities that the model can represent. The higher the value, the stronger the model's ability to represent multiple independently moving objects in the scene. Feature Dimension This requires maintaining consistency with the dimension of the image patch features to ensure that multimodal features interact within a unified vector space. (Target time query) density (i.e.) The values ​​and distributions (uniform or nonlinear) can be adjusted according to specific task requirements. For example, for high-speed motion scenarios, denser queries can be used to achieve more refined modeling of the motion process.

[0043] Specifically, this step explicitly injects crucial prior information about temporal sequence into the model through dynamic positional encoding, compensating for the shortcomings of traditional encoding methods in temporal modeling. Secondly, the motion token initialized in this way serves as the core carrier for subsequent cross-frame attention interactions and conditional feature modulation; its quality directly determines the accuracy and spatiotemporal consistency of the final 3D motion field estimation. This step, by introducing a specific technical means—a "dynamic positional encoding mechanism based on target time query"—solves the technical problem of "how to effectively embed motion tokens into the temporal context," constituting an improvement over existing technologies and producing the positive effect of "enhancing the accuracy of motion estimation," demonstrating the invention's inventiveness and novelty.

[0044] S2, by alternating intra-frame attention layers and global attention layers, cross-frame interactive modeling of the multimodal feature sequence is performed to generate an intermediate representation that integrates geometric structure and motion information.

[0045] Specifically, this step is the core link in realizing end-to-end unified 3D geometry and motion perception. Its technical implementation is based on the self-attention mechanism of the Transformer architecture, which aims to enhance the coupling expression capability of intra-frame local structure and inter-frame global relationship.

[0046] Specifically, the input multimodal feature sequence consists of image patch features. It consists of three types of tokens (geometry, camera, motion), among which This indicates the number of blocks in each frame of the image. The feature dimension is represented by the intra-frame attention layer, which models local features within a single frame and enhances the perception of geometric structure within the image by calculating attention weights between blocks. The global attention layer, on the other hand, establishes connections across frames, learning motion patterns and geometric consistency over time by aggregating features from multiple frames. These two attention mechanisms are stacked alternately to form a multi-layered perceptual structure, enabling the model to simultaneously capture intra-frame details and inter-frame dynamic changes when processing consecutive video frames.

[0047] Furthermore, the intra-frame attention layer typically employs a multi-head self-attention (MHSA) structure, with the attention weights for each head calculated as follows: ; in, These are query, key, and value matrices, respectively. This represents the dimension of the key vector. The global attention layer extends this mechanism across frames by introducing inter-frame query vectors. This enables the modeling of motion information at the target time step.

[0048] Specifically, this step is particularly suitable for dynamic video scenes with large changes in viewpoint and occlusion in practical applications. By fusing geometric and motion information in the feature space, the model can effectively alleviate the reconstruction inconsistency caused by occlusion or sudden changes in viewpoint, and improve the spatiotemporal continuity and robustness of the 3D motion field. Its technical value lies in the fact that it can complete cross-frame modeling in a unified Transformer architecture without relying on external optical flow estimation or trackers, thereby significantly reducing computational complexity and improving overall perception efficiency.

[0049] Furthermore, S2 includes: S21, the intra-frame attention layer calculates image patch features. Local correlation with geometric tokens generates intra-frame geometric consistency constraints.

[0050] Specifically, the core objective of this step is to enhance the context awareness of each frame in geometric modeling by enabling local information interaction between image patch features and camera tokens through an intra-frame attention mechanism. This step is a key component of the geometry-aware backbone network in the MotionVGGT model, providing structured and consistent feature representations for subsequent cross-frame modeling and motion prediction.

[0051] Specifically, this step first extracts image patch features from the input image using an image encoder (such as DINO). ,in This indicates the number of image blocks into which each frame is divided. The feature dimensions for each image patch are then determined. Subsequently, a camera token is introduced for each frame. This token is used to encode the camera's intrinsic and extrinsic parameters corresponding to that frame. The intra-frame attention layer concatenates the image patch features with the camera token to form a sequence of length [length missing]. The feature sequences are processed and local features are aggregated through a multi-head self-attention mechanism. Specifically, each image patch feature learns its geometric context relationship within the current frame by interacting with the camera token, thereby improving the perception accuracy of geometric information such as depth and point cloud.

[0052] Furthermore, intra-frame attention layers are typically configured as multi-head attention structures, where the number of attention heads is... It can be set to 8 or 16, the dimensions of each head. Furthermore, the length of the feature sequence in this step... It is usually determined by the output of the image encoder, for example, when using the ViT-Base encoder. (correspond Image patch partitioning). Camera token dimensions. Maintain consistency with image patch features to ensure dimensionality matching and information fusion in the attention mechanism.

[0053] Specifically, this step is widely applicable to 3D reconstruction tasks of dynamic video sequences, especially in complex scenes with large changes in viewpoint and occlusion. By explicitly injecting camera parameter information into image patch features, the model can more accurately understand the geometric structure of each frame, providing a stable spatiotemporal basis for subsequent cross-frame motion modeling.

[0054] Specifically, the intra-frame attention mechanism significantly enhances the coupling between image patches and camera parameters through local feature aggregation, thereby improving the model's ability to model the geometry of a single frame. This step provides high-quality intermediate feature representations for subsequent global attention layers and conditional dense prediction heads, and is a key step in achieving end-to-end unified geometry and motion perception.

[0055] S22, Global Attention Layer Based on Motion Tokens With camera token Cross-frame interaction establishes a spatiotemporal motion-geometric coupling relationship.

[0056] Specifically, in this step, the model introduces a multi-head attention (MHA) mechanism to achieve global interaction of cross-frame features, thereby learning the spatiotemporal consistency relationship between geometry and motion within a unified Transformer architecture.

[0057] Specifically, the core of this step lies in using the sports token. With geometric tokens (such as image patch features) and camera token Cross-frame aggregation is performed in the global attention layer. Specifically, the feature sequence of each frame (including image patches, camera data, and motion tokens) is input into the global attention layer, which globally models the features of all frames through a multi-head attention mechanism. Each attention head independently computes the query, key, and value vectors, and then uses the attention weight matrix... To achieve cross-frame feature interaction, where For the number of image block tokens, For the number of sports tokens, This represents the camera token. The outputs of multi-head attention are fused through a linear projection layer to form an intermediate representation with spatiotemporal consistency.

[0058] Specifically, the number of sports tokens It is usually set to the number of image block tokens. Matching sparse sampling points to ensure both coverage of motion information and computational efficiency. Number of attention heads. It is generally set to 8 or 16, the dimensions of each head. , and Usually ,in This represents the number of feature channels (e.g., 256 or 512). Furthermore, the output dimension of the global attention layer remains unchanged. This is to ensure compatibility with subsequent geometry decoders.

[0059] Specifically, this step is applicable to 3D reconstruction tasks of dynamic video sequences, especially in complex scenes with occlusion, drastic changes in viewpoint, or object deformation. Through cross-frame attention mechanism, the model can effectively capture the long-range dependency between motion and geometry, thereby improving the spatiotemporal consistency and robustness of reconstruction.

[0060] Specifically, through the global attention mechanism, the model can establish cross-frame geometry-motion coupling relationships in the feature space, avoiding the error accumulation and computational overhead caused by relying on external optical flow or trackers in traditional methods. At the same time, the multi-head attention mechanism enhances the model's ability to express different motion modes, providing high-quality spatiotemporally consistent features for subsequent conditional dense prediction heads, thereby achieving end-to-end unified dynamic 3D perception.

[0061] S3, based on the intermediate representation, the motion token is used as a conditional signal to dynamically modulate the geometric features through adaptive layer normalization, and then the pixel-level three-dimensional motion field is generated by decoding through a conditional dense prediction head.

[0062] Specifically, this step is a key step in achieving end-to-end unified 3D geometry and motion perception, and its technical implementation principle and operation method are as follows: Furthermore, the Adaptive Layer Normalization Module (AdaLN) introduces motion tokens. As a conditional signal, the characteristics of the geometric backbone output Dynamic modulation is performed. Specifically, AdaLN employs learnable affine transformation parameters. and Its input is geometric features With sports tokens The combination of these factors results in a modulated feature output. Its mathematical expression is: ; in, This represents the standard layer normalization operation. For target time queries, specify the time target for motion prediction. and It is a sports token Time query The modulation parameters generated by the multilayer perceptron (MLP) have dimensions and features Consistency is achieved, thereby enabling personalized adjustments to the geometric features at each pixel location.

[0063] Furthermore, the modulated features The data is fed into a DPTHead-style conditional dense decoder, which is based on the Transformer architecture and gradually reconstructs a high-resolution, pixel-level 3D motion field through multi-layer decoding attention mechanisms and upsampling operations. The output dimension of DPTHead is... ,in For the number of input frames, For the target number of time steps, Represents a three-dimensional motion vector , and This represents the height and width of the image.

[0064] Specifically, by directly injecting motion information into the decoding process of geometric features, the model can more accurately capture pixel-level motion changes, avoiding the error accumulation and computational overhead caused by relying on external optical flow or point trackers in traditional methods.

[0065] Specifically, this design significantly improves the spatiotemporal consistency of the motion field and the robustness of geometric reconstruction. Compared to a fixed-weight decoder, the conditional dense prediction head achieves dynamic coupling between motion and geometric features through AdaLN, thus outputting high-quality pixel-level 3D motion vectors even in scenarios with complex occlusion and changing viewpoints.

[0066] Furthermore, S3 includes: S31, using AdaLN for geometric features Adjust parameters, scaling factor and offset factor By sports token It is generated through a fully connected layer.

[0067] Specifically, the core of this step lies in utilizing the specific mechanism of adaptive layer normalization to achieve dynamic modulation of geometric features by motion information. Traditional layer normalization uses fixed scaling and offset parameters, which are difficult to adapt to dynamically changing scenes in video sequences. This step innovatively uses a motion token M as a conditional signal, mapping it through one or more fully connected layers to generate the scaling factor γ and offset factor β required to modulate the geometric features F. Technically, this process can be expressed as AdaLN(F) = γ(M). Norm(F) + β(M), where Norm(F) represents the normalization result of the geometric features. This means that the distribution of geometric features is no longer static, but is adjusted in real time and in a content-related manner by the temporal motion information encoded by the motion tokens, thereby deeply coupling the geometric structure with the motion trajectory at the feature level.

[0068] Furthermore, different parts of a scene often exhibit distinct motion patterns. This step's design enables the model to adaptively enhance or suppress the representation of geometric features based on the motion state of each region. For example, for high-speed moving objects, this mechanism can strengthen the saliency of their geometric edge features to ensure clear motion field boundaries; for static backgrounds, it can maintain the stability of their features. This dynamic modulation capability significantly improves the model's robustness in complex real-world scenarios (such as multiple vehicles merging at intersections and pedestrians crossing), ensuring high-fidelity 3D motion field estimation for both fast-moving foreground objects and slowly changing background environments, directly improving the performance and safety of downstream tasks such as autonomous driving and video understanding.

[0069] Specifically, the core parameter involved in this step is the specific configuration of the fully connected layers that generate the scaling factor γ and the offset factor β. The input dimension of these fully connected layers matches the dimension C of the motion token M, while the output dimension must match the number of channels of the geometric feature F to be modulated, ensuring channel-wise modulation. Furthermore, the hidden layer dimension and the choice of activation function (such as GELU) within the fully connected layer itself determine the mapping capability of the conditional signal M to the modulation parameters γ and β. A deeper or wider fully connected layer can capture more complex motion-geometry coupling relationships, but a trade-off must be made between model performance and computational cost. The number of motion tokens K indirectly affects the richness of the conditional signal.

[0070] Specifically, the core value of this step lies in the fact that it does not simply concatenate motion and geometric features, but rather utilizes the efficient mechanism of conditional normalization to achieve refined and dynamic control of the representation of geometric features by motion information. This solves the problems of feature alignment difficulties and information loss caused by the separation of motion and geometric modules in traditional methods. By introducing the specific technical means of "generating adaptive layer normalization parameters through a fully connected layer based on motion tokens," this step successfully solves the technical problem of "how to effectively and efficiently modulate geometric features with motion information," resulting in a significant effect of "improving the accuracy and spatiotemporal consistency of 3D motion field estimation." This constitutes a substantial improvement over existing technologies and reflects the inventiveness and technological advancement of the invention.

[0071] The S32, DPTHead decoder employs a multilayer perceptron structure, achieving pixel-level motion field resolution through residual connections and feature pyramids. The gradual refinement; among them, Indicates the number of video clips. This indicates the frame number of each segment, and 3 indicates the dimension of the motion vector. Indicates the height of the sports field in space. It indicates the width of the sports field in space.

[0072] Specifically, the technical implementation of this step is based on the decoupled design of the Adaptive Layer Normalization (AdaLN) mechanism and the Transformer architecture, which enables the joint decoding of geometric and motion information.

[0073] Specifically, this step first receives image features output from the geometry-aware backbone. With sports tokens And query in conjunction with the target time. The features are dynamically modulated using the AdaLN module. AdaLN modulates the features using motion tokens. As a conditional signal, for geometric features The mean and variance are adaptively adjusted to inject timing information during decoding. Specifically, the output of AdaLN is: ; in, The modulated geometric features still have the same dimension. However, it has already incorporated motion and time query information. Subsequently, the modulated features are input into the DPTHead decoder, which consists of a multi-layer MLP and an upsampling module, recovering the spatial resolution of the features layer by layer, and finally outputting a pixel-level three-dimensional motion field. Its shape is , representing each source frame To target frame Each pixel Three-dimensional motion vector in world coordinate system .

[0074] Optionally, the upsampling module in DPTHead can employ transposed convolution or interpolation to progressively restore the spatial dimension of the feature map. The MLP layer typically consists of several fully connected layers and activation functions (such as GELU) for nonlinear mapping and feature fusion. Furthermore, this decoder supports multi-target frame queries, enabling the model to simultaneously predict motion information from multiple target frames, thereby improving inference efficiency and temporal consistency of the motion field.

[0075] Specifically, this step plays a crucial role in the entire MotionVGGT model. Through a conditional decoding mechanism, it directly embeds motion information into the decoding process of geometric features, avoiding redundant steps that rely on external optical flow or trackers in traditional methods. This significantly improves the model's end-to-end inference capability and the accuracy of dynamic scene reconstruction. In practical applications, this module can be widely used for dynamic obstacle modeling in autonomous driving, real-time scene updates in augmented reality, and 3D motion perception tasks in robot navigation.

[0076] S4, through a single forward propagation, jointly outputs camera parameters, depth maps, point cloud maps, and 3D motion fields of a continuous video sequence, achieving a unified representation of geometry and motion.

[0077] Specifically, this step is the core link in achieving end-to-end unified 3D geometry and motion perception, and its technical implementation is based on the feedforward Transformer architecture and multi-task joint decoding mechanism.

[0078] Furthermore, the model receives continuous video sequences during the inference phase. Each frame of the image The size is , representing a three-channel RGB image with a height of Width is The image is first processed by a DINO encoder to extract image patch features. ,in The number of image patches, For the feature dimension. Subsequently, the features of each frame image are compared with the camera token. Sports tokens The sequences are spliced ​​together to form a multimodal feature sequence, which is then input into the geometric perception backbone network for cross-frame modeling and feature aggregation.

[0079] Furthermore, the model outputs a three-dimensional motion field through a Conditional Dense Prediction Head. The head uses a motion token. As a conditional signal, geometric features are dynamically adjusted through adaptive layer normalization (AdaLN). This enables a differentiable mapping from geometric representation to pixel-level 3D motion. Specifically, the model utilizes target time queries... For each pixel in each frame Generate from source frame To target frame 3D motion vector in world coordinate system This design avoids the complex process of motion estimation that relies on external optical flow or point trackers in traditional methods, significantly improving inference efficiency and system robustness.

[0080] Specifically, the camera parameters output by the model With external references Usually and The matrix, depth map With point clouds The resolution is the same as the input image, that is... Three-dimensional sports field The output dimension is ,in For the number of input frames, For the target number of frames, This represents the 3D motion vector for each pixel. The model inference process is completed in a single forward propagation, with a computational complexity of O(n log n). Compared with the traditional two-stage method, the reasoning speed is improved by more than 30%.

[0081] Specifically, this step is particularly suitable for scenarios requiring real-time processing of dynamic video sequences, such as scene understanding in autonomous driving, dynamic object tracking in augmented reality, and 3D motion perception in robot navigation. Since no external optical flow estimation or tracker is required, the model can be deployed on resource-constrained edge devices, achieving low-latency, high-precision 3D dynamic reconstruction.

[0082] Specifically, by employing a unified feature representation and conditional decoding mechanism, the joint output of geometric and motion information is achieved, avoiding error accumulation and computational redundancy caused by task decoupling in traditional methods. Simultaneously, the model outputs a 3D motion field with pixel-level accuracy, effectively supporting temporal consistency modeling of dynamic scenes and downstream 3D understanding tasks.

[0083] Furthermore, S4 includes: S41, Three-dimensional sports field The representation is based on the world coordinate system rather than the camera coordinate system; among them, This represents the amount of motion along the x-axis in the world coordinate system. This represents the amount of motion along the y-axis in the world coordinate system. This represents the amount of motion along the z-axis in the world coordinate system. Represents pixel coordinates.

[0084] Specifically, the technical implementation of this step is based on the Conditional Dense Prediction Head and the Adaptive Layer Normalization (AdaLN) mechanism.

[0085] Specifically, this step first relies on the image features output by the geometry-aware backbone network. With sports tokens Sports token Depend on It consists of learnable embedding vectors, each vector having a dimension of . It is used to encode the motion patterns of dynamic objects in a scene. During the decoding phase, the motion token dynamically modulates geometric features through the AdaLN mechanism. This enhances motion perception of geometric information. Specifically, AdaLN achieves this by combining motion tokens with time queries. By combining inputs, learnable scaling and offset parameters are generated, thereby adaptively normalizing geometric features and enhancing their ability to perceive motion in target frames.

[0086] Furthermore, the output dimension of this sports field is ,in Indicates the number of source frames. Indicates the number of target frames. and These represent the height and width of the image, respectively. Each pixel... In the source frame To target frame motion vector By using a world coordinate system as a reference, this design avoids the viewpoint bias and occlusion problems caused by frame-by-frame tracking based on the camera coordinate system in traditional methods. This design has significant advantages in applications such as augmented reality and autonomous driving that require high-precision spatiotemporal consistency modeling. It can directly support 3D reconstruction and motion prediction of dynamic scenes without relying on external optical flow estimation or point trackers, thereby improving the end-to-end efficiency and robustness of the system.

[0087] S42, the output includes camera intrinsic and extrinsic parameters. Depth map Point cloud map With sports field Joint gradient consistency constraints.

[0088] Specifically, in terms of technical implementation, this is not a simple matter of independent outputs. Instead, during the backpropagation process of model training, a joint loss function is constructed that connects all output terms. This function enforces that the geometric and motion information derived from different outputs must be mathematically and physically consistent. For example, the camera parameters from adjacent frames... Depth map and sports field The calculated next frame image should be consistent with the actual next frame image at the pixel level (i.e., reprojection consistency); at the same time, based on the point cloud map... With sports field The calculated 3D scene flow must also satisfy spatial smoothness and continuity constraints. This coupling optimization achieved through gradients ensures that a self-consistent geometric system is formed among the outputs.

[0089] Specifically, in practical applications such as autonomous driving and robot navigation, the reliability of a perception system depends on the consistency, rather than contradiction, of the information output by its various modules. This step, through joint gradient consistency constraints, fundamentally solves the information misalignment problem commonly found in traditional separate perception modules. For example, it avoids logical fallacies such as estimated motion vectors pointing to an object that appears as infinitely far away on a depth map. This inherent consistency ensures that when the system provides an environmental model for downstream path planning or decision control modules, the output camera pose, 3D map, and dynamic object trajectory are highly coordinated and unified, greatly improving the safety and stability of the entire intelligent system operating in complex and dynamic environments.

[0090] Furthermore, this step is implemented without directly introducing new network structure parameters that require manual adjustment. Instead, it is reflected in the weight configuration of various constraint losses in the joint optimization objective. During training, carefully balanced weight coefficients need to be set for different constraint terms such as reprojection consistency loss, depth smoothing loss, and motion field smoothing loss. The settings of these hyperparameters determine the model's trade-off tendency between "fitting the observed data" and "satisfying physical geometric laws." An appropriate parameter set can guide the model to converge to an ideal state where all outputs conform to the input video observations and are highly consistent internally, thereby avoiding overall performance degradation due to excessively strong or weak constraints.

[0091] Specifically, this step not only improves the overall efficiency of the system, but more importantly, it creates a powerful self-supervised signal through internal consistency constraints, reducing the dependence on massive, finely labeled ground truth data, while significantly improving the accuracy and robustness of the results. This step clarifies the specific technical means of achieving unified optimization of multiple outputs through joint gradient consistency constraints, successfully solving the technical challenge of "conflicting output results in multi-task perception systems," resulting in a significant improvement in the overall performance and reliability of the system, and constituting a complete and licensable technical solution.

[0092] S5 executes a multi-stage progressive training strategy.

[0093] Specifically, this strategy gradually enhances the model's ability to model dynamic scenes by introducing different types of supervision signals in stages, thereby achieving a smooth transition from static geometric learning to dynamic motion perception.

[0094] Furthermore, in the first stage (supervised motion learning), the model is trained using a dataset with 2D point tracking annotations. Specifically, after inputting a video sequence, the geometric backbone network first recovers the 3D coordinates of pixels in each frame. Subsequently, through cross-frame correspondences, the displacements of these 3D points are converted into ground truth 3D motion values ​​in the world coordinate system. The motion head undergoes supervised training with a loss function in this stage to establish preliminary motion perception capabilities. The loss function in this stage typically includes a 3D motion error term, such as: ; in, Indicates the model's prediction from the source frame To target frame pixels The three-dimensional motion vector, This represents the labeled truth vector. The training objective at this stage is to enable the model to have a preliminary understanding of motion patterns, laying the foundation for subsequent self-supervised training.

[0095] Furthermore, in the second stage (self-supervised reprojection consistency), the model is trained on large-scale unlabeled video data. By introducing geometric reprojection consistency loss, the model can still optimize motion prediction through cross-frame geometric consistency constraints even without explicit motion annotations. Additionally, spatial gradient smoothing constraints can be added to enhance the continuity and stability of the motion field in local regions. The training focus in this stage is to improve the model's generalization ability to real-world dynamic scenes (such as occlusion and viewpoint changes).

[0096] Furthermore, in the third stage (joint fine-tuning), all network modules (including the geometry head and motion head) are unfrozen and jointly optimized under a unified loss function. This stage, through an end-to-end approach, enables the geometry and motion tasks to mutually reinforce each other, thereby improving the overall reconstruction accuracy and temporal consistency. For example, more accurate depth estimation helps eliminate motion ambiguity, while a more consistent motion field, in turn, stabilizes the geometry reconstruction process.

[0097] Specifically, this multi-stage training strategy has significant advantages in practical applications, especially for processing large-scale dynamic video data. By gradually introducing supervisory signals, the model can effectively utilize unsupervised information based on limited labeled data, achieving efficient and robust 3D dynamic scene perception.

[0098] Furthermore, S5 includes: S51, Phase 1, using supervised 3D motion loss Establish initial motion perception capabilities.

[0099] Specifically, the core objective of this stage is to enable the model to learn the pixel-level 3D motion field based on geometric reconstruction, thereby laying the foundation for the subsequent self-supervised and joint optimization stages.

[0100] Specifically, this stage first uses an image encoder (such as DINO) to extract block features from each frame of the image. ,in Indicates the number of image patches. The feature dimension is defined as follows. Subsequently, the model concatenates three types of tokens for each frame: geometry tokens, camera tokens, and motion tokens, which are then input into the geometry-aware backbone network. The backbone network aggregates and interacts with cross-frame information through alternating intra-frame attention and global attention layers, thereby learning intermediate representations consistent with 3D geometry. Based on this, the model decodes the camera parameters for each frame. , Depth map and point cloud map Furthermore, it converts the 2D pixel correspondence across frames into the true values ​​of 3D motion in the world coordinate system. It is used to supervise the output of the motion prediction head.

[0101] Furthermore, the loss function at this stage Mean squared error (MSE) or L1 loss is typically used to predict the three-dimensional motion field. Align the data pixel-by-pixel with the ground truth. Specifically, the loss function can be expressed as: ; in, To input the number of video frames, For the target time query count, , For image resolution, This represents the 3D motion vector predicted by the model. This loss function ensures that the model can accurately learn pixel-level motion changes with labeled data, thereby improving the accuracy and robustness of motion perception.

[0102] Specifically, this stage is suitable for scenarios requiring high-precision motion modeling, such as dynamic object tracking in augmented reality and pedestrian / vehicle motion prediction in autonomous driving. Through supervised learning, the model can quickly adapt to the motion patterns of specific tasks, and can maintain the continuity and consistency of the motion field, especially in the presence of significant deformation or occlusion.

[0103] Specifically, this stage is achieved through The introduction of this approach enables the model to perceive dynamic scenes early in training, avoiding the two-stage process of reconstruction followed by tracking in traditional methods. Furthermore, because motion information is modeled in the world coordinate system, rather than being limited to the camera coordinate system, it can more effectively handle viewpoint changes and occlusion issues, providing a stable foundation for subsequent self-supervised and joint optimization.

[0104] S52, Phase 2 introduces geometric reprojection consistency loss and spatial gradient smoothing constraint.

[0105] Specifically, in the second phase of training, this invention introduces a self-supervised reprojection consistency loss. Spatial gradient smoothing constraint This aims to optimize the spatiotemporal correspondence across frames. The core objective of this stage is to improve the model's robustness in modeling geometric and motion information in dynamic scenes through unsupervised or weakly supervised methods, especially under complex conditions such as occlusion and drastic changes in viewpoint.

[0106] Specifically, The construction is based on camera parameters output from the geometry-aware backbone. , and depth map Specifically, the model uses the depth map of the current frame and camera parameters to backproject the pixels of the target frame onto the world coordinate system, and then uses motion tokens to... Predicted three-dimensional motion field The point is then reprojected back onto the image plane of the source frame. If the reconstructed result differs significantly from the original pixel value of the source frame in the feature space or color space, the loss is calculated using the reprojection error, thereby constraining the motion field prediction to conform to geometric consistency. This process does not require manually labeled motion fields and can achieve self-supervised training solely based on the input of the video sequence.

[0107] Furthermore, to enhance the continuity and stability of the sports field in the spatial dimension, this invention introduces a spatial gradient smoothing constraint. This constraint, by calculating the gradient magnitude of the motion field in the image space, encourages similarity in the motion vectors of adjacent pixels, thereby suppressing noise and abnormal motion prediction. Specifically, It can be represented as: ; in, , The height and width of the image. Indicates the pixel position The gradient vector of the motion field at that location. This loss term is used in training... Joint optimization further enhances the spatial consistency of the sports field.

[0108] Specifically, this stage is particularly suitable for training large-scale unlabeled video data, such as in-vehicle videos in autonomous driving and drone aerial footage sequences. Through self-supervised learning, the model can adapt to dynamic scenes with varying lighting, occlusion, and viewpoint changes, without relying on expensive motion-annotated data.

[0109] Specifically, through and Through joint optimization, the model can learn spatiotemporal consistency across frames under unsupervised conditions, significantly improving the robustness and geometric consistency of motion field prediction, and laying a solid foundation for the subsequent joint fine-tuning stage.

[0110] S53, Phase 3: Unfreeze all network modules and perform joint optimization of the geometry head and motion head.

[0111] Specifically, this stage is a key link in the entire training process. Its core lies in breaking the freezing strategy of network modules in the first two stages, so that geometric and motion tasks form a two-way promotion mechanism in the feature space.

[0112] Specifically, this stage first unfreezes the partially trained model parameters from the first and second stages, including the image encoder, the intra-frame and global attention layers in the geometry-aware backbone network, and the geometry head (used to predict camera intrinsic and extrinsic parameters, depth maps, and point cloud maps) and motion head (used to predict pixel-level 3D motion fields). During decoding, motion tokens... As a conditional signal, geometric features are dynamically modulated through an adaptive layer normalization (AdaLN) mechanism. And query in conjunction with the target time. DPTHead outputs a dense three-dimensional motion field. This process, by sharing gradient information, allows the geometry and motion tasks to influence each other during backpropagation, thereby enhancing the model's ability to perceive complex dynamic scenes.

[0113] Furthermore, training at this stage is typically performed on large-scale unlabeled video datasets, with input video sequence lengths of... It is generally set to 5 to 10 frames, target time query. It can be set to 1 to 3 future frames. The optimization objective of the model is a joint loss function, which includes 3D motion loss and geometric reprojection consistency loss, specifically in the form of: ; in, and For loss weights, they are usually set to , To balance the optimization priorities of motion prediction and geometric reconstruction.

[0114] Specifically, the joint optimization strategy in this stage is particularly suitable for dynamic scenes with drastic changes in viewpoint, occlusion, or object deformation, such as pedestrian motion modeling in autonomous driving and real-time scene updates in augmented reality. Through a unified loss function and a shared gradient mechanism, the model can more accurately recover the 3D motion trajectory of objects while maintaining the stability of the geometric structure, avoiding the error accumulation problem caused by the decoupling of geometry and motion in traditional methods.

[0115] Specifically, end-to-end joint optimization significantly improved the model's reconstruction accuracy and robustness in dynamic scenes. The collaborative optimization of geometry and motion tasks not only enhanced the model's adaptability to complex motion patterns but also effectively improved the spatiotemporal consistency between pixel-level motion fields and point cloud maps, providing a high-quality unified representation for downstream tasks such as dynamic scene understanding and 3D video generation.

[0116] This invention presents an end-to-end unified perception method for dynamic 3D scenes, effectively solving the core problems of separation and error accumulation in traditional dynamic scene perception, namely geometric reconstruction and motion estimation. It achieves deep fusion and bidirectional promotion of motion information and geometric features during the decoding process, significantly improving the spatiotemporal consistency and geometric accuracy of 3D motion field estimation. It exhibits stronger adaptability and robustness in complex dynamic scenes, providing a more accurate and consistent scene understanding solution for applications such as autonomous driving and 3D vision.

[0117] Example 2 To achieve the above invention, embodiments of the present invention also provide a model architecture for an end-to-end unified perception method for dynamic 3D scenes, such as... Figure 2 As shown, it includes: The overall idea of ​​this model architecture is as follows: inputting a continuous video sequence or image sequence, the model outputs the geometric information (camera parameters, depth map, point cloud map) and 3D motion information (pixel-level motion vector field) of the scene in a single forward inference through a unified encoding and decoding structure, without the need for additional optical flow estimation or external trackers.

[0118] Specifically, the system takes a continuous video sequence as input and employs a feedforward Transformer structure of "geometric perception backbone + conditional motion prediction head". In one forward inference, it simultaneously outputs camera parameters, depth / point cloud, and pixel-level 3D motion field. The specific steps are as follows: S101, Data input and token construction during the training phase.

[0119] Specifically, the length is video sequence The image is fed into an image encoder (DINO) to obtain the patch features of each frame. Three types of tokens are concatenated for each frame: camera token. Sports tokens The image tokens are combined to form a sequence and then input into the geometry-aware backbone network for cross-frame interaction and global modeling.

[0120] S102, cross-frame modeling of the geometry-aware backbone.

[0121] Specifically, the backbone employs alternating intra-frame attention and global attention layers to aggregate tokens from multiple frames, learning intermediate representations consistent with 3D geometry. Then, geometry-related multi-head decoding predicts camera intrinsics and extrinsic parameters, depth, and point maps. This backbone inherits VGGT's multi-task supervision in multi-view geometry, enabling intermediate features to naturally encode geometry and maintain cross-frame consistency.

[0122] S103, 3D motion decoding of conditionally dense prediction heads.

[0123] Specifically, based on the features output by the geometric backbone, a Conditional Dense Prediction Head is introduced: using motion tokens as conditional signals, temporal / motion information is injected into the geometric features through Adaptive Layer Normalization (AdaLN), and then a DPT-style dense decoder outputs a dense 3D motion field. ; in, For image tokens, For sports tokens, For target time queries. This design allows the model to generate pixel-level 3D motion directly without external optical flow or trackers.

[0124] S104, Joint output and expression form of the reasoning stage.

[0125] Specifically, the model can output simultaneously in a single forward pass: ; in, For internal reference, As an external reference, For depth, For point maps / point clouds, This is a three-dimensional motion field. For any pixel... From the source frame To target frame The world system three-dimensional motion is defined as: .

[0126] Compared with existing tracking methods, this invention (e.g.) Figure 3As shown in the figure, without the need for complicated post-processing, it achieves end-to-end unified perception of geometry and motion.

[0127] This invention provides a model architecture for an end-to-end unified perception method for dynamic 3D scenes. Through a unified encoding and decoding architecture, it achieves the joint output of geometric and motion information, effectively addressing the technical shortcomings of traditional methods that separate geometric reconstruction and motion estimation, and rely on complex post-processing. This method innovatively employs a conditional motion prediction mechanism, using motion information as a conditional signal for geometric feature modulation. In a single forward propagation, it simultaneously outputs camera parameters, depth maps, point cloud maps, and a 3D motion field, significantly improving the completeness and spatiotemporal consistency of 3D scene perception. It demonstrates stronger environmental understanding and higher output accuracy in complex dynamic scenarios such as autonomous driving and robot navigation, providing a more reliable and unified environmental perception solution for downstream applications.

[0128] Example 3 To achieve the above invention, such as Figure 4 As shown, this embodiment also provides an end-to-end unified sensing device 10 for dynamic three-dimensional scenes, the device 10 including: The image encoding and multimodal feature construction module 100 is used to input continuous video sequences into the image encoder and construct a multimodal feature sequence containing geometric tokens, camera tokens, and motion tokens.

[0129] The intra-frame and global attention interaction module 200 is used to perform cross-frame interactive modeling of the multimodal feature sequence through alternating intra-frame attention layers and global attention layers to generate an intermediate representation that integrates geometric structure and motion information.

[0130] The motion condition geometry decoding module 300 is used to generate a pixel-level three-dimensional motion field based on the intermediate representation, using motion tokens as conditional signals to dynamically modulate geometric features through adaptive layer normalization, and then decoding them through a conditional dense prediction head.

[0131] The joint output and unified representation module 400 is used to jointly output camera parameters, depth maps, point cloud maps and three-dimensional motion fields of a continuous video sequence through a single forward propagation, so as to achieve a unified representation of geometry and motion.

[0132] In one embodiment of the present invention, it further includes: a multi-stage progressive training module for executing a multi-stage progressive training strategy, including: Stage 1 using supervised three-dimensional motion loss. The initial motion perception capability is established; in the second stage, geometric reprojection consistency loss and spatial gradient smoothing constraint are introduced; in the third stage, all network modules are unfrozen, and the geometric head and motion head are jointly optimized.

[0133] This invention discloses a multi-level collaborative prediction device for battery pack health status and lifespan. By constructing a multi-level dynamic graph model and an attribute decoupling encoding mechanism, it effectively solves the problem of insufficient prediction accuracy caused by hierarchical isolation and feature aliasing in traditional methods. This device achieves end-to-end collaborative modeling from spatial topology construction and feature decoupling to cross-layer information fusion, significantly improving the accuracy and consistency of health status and lifespan prediction. Modular design enhances the system's engineering applicability and adaptability, providing a more reliable solution for battery management system status assessment and lifespan prediction.

[0134] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 5 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads the executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the end-to-end unified perception method for a dynamic three-dimensional scene described above.

[0135] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an end-to-end unified perception method for a dynamic three-dimensional scene as described in the foregoing embodiments.

[0136] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0137] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. An end-to-end unified perception method for dynamic 3D scenes, characterized in that, include: S1, Input a continuous video sequence to the image encoder to construct a multimodal feature sequence containing geometric tokens, camera tokens and motion tokens; S2, cross-frame interactive modeling of the multimodal feature sequence is performed through alternating intra-frame attention layers and global attention layers to generate an intermediate representation that integrates geometric structure and motion information; S3, based on the intermediate representation, the motion token is used as a conditional signal to dynamically modulate the geometric features through adaptive layer normalization, and then the pixel-level three-dimensional motion field is generated by decoding through a conditional dense prediction head. S4, through a single forward propagation, jointly outputs camera parameters, depth maps, point cloud maps, and 3D motion fields of a continuous video sequence, achieving a unified representation of geometry and motion.

2. The method as described in claim 1, characterized in that, The input continuous video sequence is fed to the image encoder to construct a multimodal feature sequence containing geometric tokens, camera tokens, and motion tokens, including: S11, extract image patch features for each frame using the DINO image encoder. ;in, This indicates the number of image patches in each frame. This represents the dimension of each feature vector; S12, initialize motion tokens for each frame. At that time, a query based on the target time is used. The dynamic position coding mechanism; among which, This represents the dimension of each feature vector. Indicates the number of sports tokens. t 1,..., tr This indicates a specific time value.

3. The method as described in claim 1, characterized in that, The process of performing cross-frame interactive modeling of the multimodal feature sequence through alternating intra-frame attention layers and global attention layers to generate an intermediate representation that integrates geometric structure and motion information includes: S21, the intra-frame attention layer calculates image patch features. Local correlation with geometric tokens generates intra-frame geometric consistency constraints; S22, Global Attention Layer Based on Motion Tokens With camera token Cross-frame interaction establishes a spatiotemporal motion-geometric coupling relationship.

4. The method as described in claim 1, characterized in that, The process of generating a pixel-level 3D motion field based on the intermediate representation, using motion tokens as conditional signals, dynamically modulating geometric features through adaptive layer normalization, and then decoding them via a conditional dense prediction head, includes: S31, using AdaLN for geometric features Adjust parameters, scaling factor and offset factor By sports token Generated via a fully connected layer; The S32, DPTHead decoder employs a multilayer perceptron structure, achieving pixel-level motion field resolution through residual connections and feature pyramids. The gradual refinement; among them, Indicates the number of video clips. This indicates the frame number of each segment, and 3 indicates the dimension of the motion vector. Indicates the height of the sports field in space. It indicates the width of the sports field in space.

5. The method as described in claim 1, characterized in that, The joint output through a single forward propagation includes: S41, Three-dimensional sports field The representation is based on the world coordinate system rather than the camera coordinate system; among them, This represents the amount of motion along the x-axis in the world coordinate system. This represents the amount of motion along the y-axis in the world coordinate system. This represents the amount of motion along the z-axis in the world coordinate system. Represents pixel coordinates; S42, the output includes camera intrinsic and extrinsic parameters. Depth map Point cloud map With sports field Joint gradient consistency constraints.

6. The method as described in claim 1, characterized in that, Also includes: S5 executes a multi-stage progressive training strategy: S51, Phase 1, using supervised 3D motion loss Establish initial motion perception ability; S52, Phase 2 introduces geometric reprojection consistency loss and spatial gradient smoothing constraint; S53, Phase 3: Unfreeze all network modules and perform joint optimization of the geometry head and motion head.

7. A dynamic three-dimensional scene end-to-end unified sensing device, characterized in that, include: The image encoding and multimodal feature construction module is used to input continuous video sequences into the image encoder and construct a multimodal feature sequence containing geometric tokens, camera tokens, and motion tokens; The intra-frame and global attention interaction module is used to perform cross-frame interactive modeling of the multimodal feature sequence through alternating intra-frame attention layers and global attention layers, generating an intermediate representation that integrates geometric structure and motion information. The motion condition geometry decoding module is used to generate a pixel-level three-dimensional motion field based on the intermediate representation, using motion tokens as conditional signals through adaptive layer normalization and dynamic modulation of geometric features, and then decoding them through a conditional dense prediction head. The Joint Output and Unified Representation module is used to jointly output camera parameters, depth maps, point cloud maps, and 3D motion fields of a continuous video sequence through a single forward propagation, thereby achieving a unified representation of geometry and motion.

8. The apparatus as claimed in claim 7, characterized in that, Also includes: A multi-stage progressive training module is used to execute multi-stage progressive training strategies, including: Phase 1 uses supervised 3D motion loss Establish initial motion perception ability; Phase 2 introduces geometric reprojection consistency loss and spatial gradient smoothing constraint; Phase 3 involves unfreezing all network modules and performing joint optimization of the geometry head and motion head.

9. An electronic device, comprising: processor; The memory stores executable instructions; when the processor executes the instructions, it implements the dynamic three-dimensional scene end-to-end unified perception method as described in any one of claims 1-6.

10. A computer-readable storage medium storing a computer program, which, when executed by a processor, implements a dynamic three-dimensional scene end-to-end unified perception method as claimed in any one of claims 1-6.

Images