Spatiotemporal State Prediction Method and System for Autonomous Driving World Model Incorporating High-Frequency Inertial Physical Constraints

By fusing high-frequency inertial data and multi-sensor information to generate physically enhanced semantic features, the problem of low sampling frequency of visual sensors in autonomous driving models is solved, enabling more accurate prediction of future scene states and reducing computation and storage requirements.

CN123069256APending Publication Date: 2026-11-10TONGJI UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202611209927.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-11
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing autonomous driving world models are insufficient in terms of motion state consistency. Visual sensors have low sampling frequency and are easily affected by occlusion and changes in lighting, making it difficult to accurately predict changes in position, speed, and attitude in emergency situations.

Method used

By synchronously acquiring vehicle-mounted multi-source sensor data, utilizing the carrier motion state information fused from a high-frequency inertial measurement unit and multiple sensors, a physically enhanced semantic feature representation is generated. Topological alignment and fusion are then performed in the latent space to construct a motion state consistency loss function to constrain the prediction results.

Benefits of technology

It improves the accuracy of prediction results, reduces computational and storage overhead, lowers prediction latency, and adapts to the resource constraints of in-vehicle computing platforms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069256A_ABST
    Figure CN123069256A_ABST
Patent Text Reader

Abstract

This invention relates to a method and system for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints. The method includes: simultaneously acquiring a continuous sequence of image frames, as well as position, attitude, and velocity information obtained from IMU high-frequency inertial data or multi-sensor fusion; extracting latent visual features and generating motion state physical features from the vehicle's motion state information, determining a confidence coefficient based on the state estimation covariance; aligning two types of features through contrastive learning and adjusting the fusion weights using the confidence coefficient; predicting future latent features based on the fused features and vehicle control commands under a joint embedding prediction architecture; and constructing a motion state consistency loss weighted by the confidence coefficient during the training phase to constrain the deviation between the predicted position, velocity, and attitude and the reference motion state, thereby obtaining the future scene state prediction result. This invention eliminates the need for pixel-level image reconstruction, simultaneously using vehicle motion state information for feature fusion and prediction constraints.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving environment simulation and embodied intelligence technology, and in particular to a method and system for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints. Background Technology

[0002] With the development of autonomous driving technology, world models can predict future traffic scenarios by learning the spatiotemporal evolution relationships of historical driving data, providing environmental state predictions for decision-making and planning. Visual joint embedding prediction architectures can perform feature prediction within a non-reconstructed latent space, thereby reducing the computational cost required for pixel-level image reconstruction.

[0003] Existing visual world models still have shortcomings in terms of consistency of motion state in environmental projection. Visual sensors have relatively low sampling frequencies and may be affected by occlusion or changes in lighting. When relying solely on latent visual features for spatiotemporal prediction, it is difficult to timely characterize changes in position, velocity, and attitude under conditions such as emergency braking, road bumps, or slippage. The prediction results may show abrupt changes in position, discontinuous velocity, or unreasonable attitude changes.

[0004] Inertial measurement units (IMUs) can output the specific force and angular velocity of a carrier at a high frequency, and multi-sensor fusion positioning can also output position, attitude, and velocity status. Existing prediction frameworks still need to address the alignment and fusion of visual features and motion state information in the latent space, as well as the use of motion state information to constrain future prediction results.

[0005] Chinese invention patent application CN122300550A discloses an end-to-end autonomous driving method based on a physical information fusion model. This method models and predicts physical indicators in a world model and uses these physical indicators as constraints during policy learning. For potential spatial predictions that do not perform pixel-level reconstruction, further research is needed on how to simultaneously use high-frequency inertial data or multi-sensor fusion states for input feature enhancement and output state consistency constraints.

[0006] Therefore, there is a need for a method and system for predicting the spatiotemporal state of an autonomous driving world model, which integrates visual information and carrier motion state information within the potential space, and constrains the consistency between the predicted position, velocity, and attitude and the reference motion state. Summary of the Invention

[0007] The present invention aims to provide a method and system for predicting the spatiotemporal state of an autonomous driving world model by incorporating high-frequency inertial physical constraints, so as to use the position, attitude and velocity information obtained by fusing IMU high-frequency inertial data or multiple sensors for latent feature fusion and prediction result constraints without performing pixel-level image reconstruction.

[0008] The objective of this invention can be achieved through the following technical solutions: According to one aspect of the present invention, a method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints is provided, the method comprising: S1. Simultaneously acquire a continuous image frame sequence output by the vehicle-mounted multi-source vision sensor, as well as carrier motion state information, wherein the carrier motion state information includes a high-frequency inertial data stream output by the inertial measurement unit, or position, attitude and velocity information obtained by fusing at least two vehicle-mounted sensors; S2. Extract the latent spatial features of the continuous image frame sequence using a visual encoder; simultaneously determine the position increment, velocity increment, and attitude change of the carrier based on the carrier motion state information, generate motion state physical features, and determine the confidence coefficient based on the corresponding state estimation covariance; S3. Construct an isomorphic manifold space, perform topological alignment between the latent space features and the motion state physical features through a contrastive learning mechanism, and adjust the fusion weights of the two types of features after alignment using the credibility coefficient to obtain a physically enhanced semantic feature representation. S4. Under the joint embedding prediction architecture, the spatiotemporal evolution network is used to infer the predictable potential features within the future time window based on the physically enhanced semantic feature representation and the current vehicle control command, without performing pixel-level image reconstruction. S5. Construct a motion state consistency loss function weighted by the credibility coefficient, calculate the residuals between the predicted position, velocity and attitude predicted by the predicted latent features and the reference position, reference velocity and reference attitude determined based on the carrier motion state information, construct a total training loss function, and obtain the scene state prediction result within the future time window through the total training loss function and the prediction output of the backpropagation constrained spatiotemporal evolution network.

[0009] As a preferred technical solution, the carrier motion state information includes at least one of the following: real-time specific force and angular velocity in the high-frequency inertial data stream; and position, attitude and velocity information obtained by fusing at least two vehicle-mounted sensors from the camera, inertial measurement unit, wheel speed gauge, satellite positioning device and lidar.

[0010] As a preferred technical solution, the increment of the carrier motion state is expressed as: in, , and These represent the time intervals. The displacement increment, velocity increment, and attitude transformation increment are obtained by pre-integration of observations from the inertial measurement unit, where T represents the matrix transpose; when using multi-sensor fusion... , and It is determined by the changes in the fusion state sequence within the corresponding time interval.

[0011] As a preferred technical solution, the physical features of the motion state are generated from the position, attitude, and velocity information obtained by pre-integration of the observations of the inertial measurement unit or fusion of multiple sensors; a confidence coefficient is determined based on the pre-integrated covariance of the observations of the inertial measurement unit or the estimated covariance of the fused state of the multiple sensors, and the confidence coefficient is used to adjust the weight of the physical features of the motion state in feature fusion and consistency constraints.

[0012] As a preferred technical solution, the topology alignment process includes achieving consistency in the distribution of visual features and motion state features by minimizing the mutual information loss function, expressed as: Where E represents the expectation, Representing visual features Characteristics of motion state The joint probability distribution, and Representing visual features respectively Characteristics of motion state The marginal probability distribution is given by d, where d is the discriminant function, used to measure the topological similarity between the visual semantic manifold and the motion state manifold in the latent space.

[0013] As a preferred technical solution, the prediction of latent features Represented as: in, Representative parameters are The conversion model, For the current moment Physically enhanced semantic feature representation, For the current moment, To predict the number of steps, Given a sequence of future action controls.

[0014] As a preferred technical solution, the mathematical expression of the motion state consistency loss function is: in, This is the physical state decoding head, used to map predicted latent features back to the physical motion space; ODE represents the ordinary differential equation based on Newtonian kinematics. , and Representing time respectively The carrier's physical state vector, specific force, and angular velocity, wherein the carrier's physical state vector includes position, velocity, and attitude. The L2 norm is represented; when the carrier motion state information is a multi-sensor fusion state, the reference physical trajectory is determined by the position, velocity and attitude state sequence of each time stamp.

[0015] As a preferred technical solution, the total training loss function includes semantic prediction loss and motion state consistency loss, expressed as: in, To predict the semantic prediction loss between latent features and target latent features, The motion state consistency loss between the position, velocity, and attitude predictions decoded from the predicted latent features and the reference position, reference velocity, and reference attitude is calculated. These are the physical constraint weighting coefficients. Representation of time The potential features of the target corresponding to the actual future observation.

[0016] As a preferred technical solution, the motion state consistency loss function is calculated only during the training phase based on the carrier motion state information corresponding to the prediction time window; during the inference phase, only the image and carrier motion state information up to the current moment are used, without inputting observation data from future moments, to obtain the scene state prediction result within the future time window.

[0017] According to another aspect of the present invention, a spatiotemporal state prediction system for an autonomous driving world model incorporating high-frequency inertial physical constraints is provided. The system is used to implement the above-described method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints. The system includes: The data perception module is used to synchronously acquire the continuous image frame sequence output by the vehicle-mounted multi-source vision sensor, as well as the carrier motion state information. The carrier motion state information includes the high-frequency inertial data stream output by the inertial measurement unit, or the position, attitude and velocity information obtained by fusing at least two vehicle-mounted sensors. The feature alignment processor, connected to the data perception module, is used to perform cross-modal alignment of visual latent features with motion state physical features generated based on carrier motion state information in the latent space, and adjust the fusion weights according to the confidence coefficient determined by the state estimation covariance. The lightweight evolutionary deduction module is connected to the feature alignment processor and has a built-in spatiotemporal transformation model based on a joint embedding prediction architecture for performing non-pixel reconstruction latent feature sequence prediction. A physical consistency constraint, connected to the lightweight evolutionary inference module, is used to construct a motion state consistency loss function weighted by the confidence coefficient based on the reference position, reference velocity, and reference attitude during the training phase, and to constrain the prediction output of the lightweight evolutionary inference module.

[0018] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) This invention simultaneously utilizes continuous image frame sequences and carrier motion state information. The carrier motion state information can be derived from high-frequency inertial data from the IMU or from position, attitude, and velocity information obtained through multi-sensor fusion. By topologically aligning and fusing the physical features of motion state with the latent visual features, the latent features not only contain scene semantic information but also motion state information related to changes in carrier position, attitude, and velocity. This helps to compensate for the lack of transient motion information caused by the relatively low sampling frequency of visual sensors and their susceptibility to occlusion and changes in illumination.

[0019] (2) The present invention adopts a joint embedding prediction architecture to extrapolate future spatiotemporal states within a physically enhanced latent feature space, without requiring pixel-level decoding, reconstruction, and rendering of future images. Compared with world models that rely on video generation, this approach can reduce the computational and storage overhead required for image decoding and pixel rendering, which is beneficial for reducing the inference latency of world models and adapting to application environments where the computing power and storage resources of in-vehicle computing platforms are limited.

[0020] (3) This invention decodes the predicted latent features into position, velocity, and attitude predictions, and compares them with reference position, reference velocity, and reference attitude determined based on IMU pre-integration or multi-sensor fusion state sequences to construct a motion state consistency loss. Simultaneously, it determines the confidence coefficient based on the state estimation covariance and uses the confidence coefficient to adjust the weights of feature fusion and consistency constraints. This reduces the adverse effects of low-confidence motion state information on model training and imposes constraints on prediction biases such as abrupt position changes, discontinuous velocity, or unreasonable attitude changes. No future observation data is input during the inference stage, thus maintaining the feasibility of the future scene state prediction process. Attached Figure Description

[0021] Figure 1 A schematic diagram of a method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints, provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the principle of physical enhancement feature alignment and spatiotemporal evolution prediction based on the JEPA architecture provided in this embodiment of the invention; Figure 3 This is a block diagram of a spatiotemporal state prediction system for an autonomous driving world model that incorporates high-frequency inertial physical constraints, provided as an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] Example 1 To address the problems existing in the prior art, this embodiment provides a method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints, such as... Figure 1 As shown, it specifically includes: S1. Simultaneously acquire the continuous image frame sequence output by the vehicle-mounted multi-source vision sensor, as well as the carrier motion state information.

[0024] During vehicle movement or sample collection, continuous image frame sequences and carrier motion state information are synchronously acquired via an onboard computing platform. The carrier motion state information can be a high-frequency inertial data stream output by an IMU, or position, attitude, and velocity information fused from at least two onboard sensors, including cameras, IMUs, wheel speedometers, satellite positioning devices, and lidar. When using IMU data, the sampling frequency is, for example, 200Hz; the image frame sequence sampling frequency is, for example, 20Hz; and the high-frequency inertial data stream includes carrier specific force and angular velocity. The two types of data are synchronized using timestamps.

[0025] S2. Use a visual encoder to extract latent spatial features from a sequence of continuous image frames; at the same time, determine the position increment, velocity increment and attitude change of the carrier based on the carrier motion state information, generate physical features of motion state, and determine the confidence coefficient based on the corresponding state estimation covariance.

[0026] On the one hand, a pre-trained visual encoder is used to extract the latent spatial features of a continuous image frame sequence and compress them into a low-dimensional visual semantic manifold representation. On the other hand, when using IMU data, the specific force and angular velocity are extracted from the high-frequency inertial data stream and pre-integration processing is performed; when using multi-sensor fusion, the position, velocity, and attitude state sequences within the same time window are read, and the corresponding state increments are determined accordingly. The carrier motion state increment is: in, , and These represent the time intervals. The displacement increment, velocity increment, and attitude change increment are obtained by pre-integration of observations from the inertial measurement unit, where T represents the matrix transpose. When using multi-sensor fusion, , and The state increments are determined by the changes in the fused state sequence within the corresponding time intervals. The state increments are encoded as physical characteristics of the motion state, and the confidence coefficients are determined based on the IMU pre-integrated covariance or the estimated covariance of the multi-sensor fused state.

[0027] S3. Construct an isomorphic manifold space, perform topological alignment between latent spatial features and motion state physical features through a contrastive learning mechanism, and adjust the fusion weights of the two types of features after alignment using a credibility coefficient to obtain a physically enhanced semantic feature representation.

[0028] like Figure 2 As shown, the consistency of the distributions of visual features and motion state features is achieved by minimizing the mutual information loss function, expressed as: Where E represents the expectation, Representing visual features Characteristics of motion state The joint probability distribution, and Representing visual features respectively Characteristics of motion state The marginal probability distribution is given by d, where d is the discriminant function, used to measure the topological similarity between the visual semantic manifold and the motion state manifold in the latent space.

[0029] After alignment and fusion, the output is a physically enhanced semantic feature representation so that the visual latent features include motion state information related to position, pose and velocity; the credibility coefficient is used to adjust the fusion weight of the motion state physical features.

[0030] S4. Under the joint embedding prediction architecture, the spatiotemporal evolution network is used to infer the potential features for prediction within the future time window based on the physically enhanced semantic feature representation and the current vehicle control command, without performing pixel-level image reconstruction.

[0031] Spatiotemporal evolution networks predict the future Potential characteristics of steps Represented as: in, Representative parameters are The conversion model, For the current moment Physically enhanced semantic feature representation, For the current moment, To predict the number of steps, Given a sequence of future action controls.

[0032] This state transition process takes place within the latent feature space and does not involve pixel-level image decoding and rendering.

[0033] S5. Construct a motion state consistency loss function weighted by the credibility coefficient, calculate the residuals between the predicted position, velocity and attitude values ​​decoded from the predicted latent features and the reference position, reference velocity and reference attitude determined based on the carrier motion state information, construct the total training loss function, and obtain the scene state prediction results within the future time window by using the total training loss function and the prediction output of the backpropagation constrained spatiotemporal evolution network.

[0034] To reduce the occurrence of abrupt position changes, velocity discontinuities, or unreasonable attitude variations in the prediction results, a motion state consistency loss function is constructed. In the IMU branch, the reference physical trajectory is obtained by pre-integration based on specific force and angular velocity; in the multi-sensor fusion branch, the reference physical trajectory is obtained based on the position, velocity, and attitude state sequence of each time stamp within the prediction time window. Its expression is: Where φ is the physical state decoding head, used to map the predicted latent features back to the physical motion space; ODE represents the ordinary differential equation based on Newtonian kinematics. , and Representing time respectively The carrier's physical state vector, specific force, and angular velocity are given. The carrier's physical state vector includes position, velocity, and attitude. The L2 norm is represented; when the carrier motion state information is a multi-sensor fusion state, the reference physical trajectory is determined by the position, velocity and attitude state sequence of each time stamp.

[0035] The total training loss function includes the semantic prediction loss of the joint embedding prediction architecture and the motion state consistency loss: in, To predict the semantic prediction loss between latent features and target latent features, To predict the motion state consistency loss between the position, velocity, and attitude predictions obtained from the latent feature decoding and the reference position, reference velocity, and reference attitude, These are the physical constraint weighting coefficients. Representation of time The potential features of the target corresponding to the actual future observation.

[0036] During model backpropagation, the total training loss function is minimized, and gradient descent is used to penalize predicted outputs that do not conform to the reference motion state, thus updating the parameters of the spatiotemporal evolution network. During training, the reference motion state can be constructed using IMU data corresponding to the prediction time window or multi-sensor fusion data; during inference, only image and carrier motion state information up to the current moment is used, without inputting observation data from future moments.

[0037] Example 2 Regarding the aforementioned embodiment's method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints, this embodiment provides a system for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints. This system is used to implement the aforementioned method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints. The system structure is as follows: Figure 3 As shown, it specifically includes: The data sensing module is used to synchronously acquire continuous image frame sequences from the vehicle-mounted camera and carrier motion state information. The carrier motion state information includes high-frequency inertial data from the IMU, or position, attitude, and velocity information obtained by fusing at least two vehicle-mounted sensors. The feature alignment processor, connected to the data perception module, is used to perform cross-modal alignment of visual latent features with motion state physical features generated based on carrier motion state information in the latent space, and adjust the fusion weights according to the confidence coefficient determined by the state estimation covariance. The lightweight evolutionary deduction module is connected to the feature alignment processor and has a built-in spatiotemporal transformation model based on a joint embedding prediction architecture for performing non-pixel reconstruction latent feature sequence prediction. The physical consistency constraint, connected to the lightweight evolutionary inference module, is used to construct a motion state consistency loss function weighted by a confidence coefficient based on the reference position, reference velocity, and reference attitude during the training phase, and to constrain the prediction output of the lightweight evolutionary inference module.

[0038] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints, characterized in that, The method includes: S1. Simultaneously acquire a continuous image frame sequence output by the vehicle-mounted multi-source vision sensor, as well as carrier motion state information, wherein the carrier motion state information includes a high-frequency inertial data stream output by the inertial measurement unit, or position, attitude and velocity information obtained by fusing at least two vehicle-mounted sensors; S2. Extract the latent spatial features of the continuous image frame sequence using a visual encoder; simultaneously determine the carrier motion state increment based on the carrier motion state information, including position increment, velocity increment and attitude change, generate motion state physical features, and determine the confidence coefficient based on the corresponding state estimation covariance; S3. Construct an isomorphic manifold space, perform topological alignment between the latent space features and the motion state physical features through a contrastive learning mechanism, and adjust the fusion weights of the two types of features after alignment using the credibility coefficient to obtain a physically enhanced semantic feature representation. S4. Under the joint embedding prediction architecture, the spatiotemporal evolution network is used to infer the predictable potential features within the future time window based on the physically enhanced semantic feature representation and the current vehicle control command, without performing pixel-level image reconstruction. S5. Construct a motion state consistency loss function weighted by the credibility coefficient, calculate the residuals between the predicted position, velocity and attitude predicted by the predicted latent features and the reference position, reference velocity and reference attitude determined based on the carrier motion state information, construct a total training loss function, and obtain the scene state prediction result within the future time window through the total training loss function and the prediction output of the backpropagation constrained spatiotemporal evolution network.

2. The method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints according to claim 1, characterized in that, The carrier motion state information includes at least one of the following: real-time specific force and angular velocity in the high-frequency inertial data stream; and position, attitude, and velocity information obtained by fusing at least two on-board sensors from the camera, inertial measurement unit, wheel speedometer, satellite positioning device, and lidar.

3. The method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints according to claim 1, characterized in that, The increment of the carrier's motion state is represented as: in, , and These represent the time intervals. The displacement increment, velocity increment, and attitude transformation increment are obtained by pre-integration of observations from the inertial measurement unit, where T represents the matrix transpose; when using multi-sensor fusion... , and It is determined by the changes in the fusion state sequence within the corresponding time interval.

4. The method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints according to claim 3, characterized in that, The physical features of the motion state are generated from the position, attitude, and velocity information obtained by pre-integration of the observations of the inertial measurement unit or fusion of multiple sensors; the confidence coefficient is determined based on the pre-integrated covariance of the observations of the inertial measurement unit or the estimated covariance of the fused state of the multiple sensors, and the confidence coefficient is used to adjust the weight of the physical features of the motion state in feature fusion and consistency constraints.

5. The method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints according to claim 1, characterized in that, The topology alignment process includes achieving consistency in the distribution of visual features and motion state features by minimizing the mutual information loss function, expressed as: Where E represents the expectation, Representing visual features Characteristics of motion state The joint probability distribution, and Representing visual features respectively Characteristics of motion state The marginal probability distribution is given by d, where d is the discriminant function, used to measure the topological similarity between the visual semantic manifold and the motion state manifold in the latent space.

6. The method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints according to claim 1, characterized in that, The predicted latent features Represented as: in, Representative parameters are The conversion model, For the current moment Physically enhanced semantic feature representation, For the current moment, To predict the number of steps, Given a sequence of future action controls.

7. The method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints according to claim 6, characterized in that, The mathematical expression for the motion state consistency loss function is: in, This is the physical state decoding head, used to map predicted latent features back to the physical motion space; ODE represents the ordinary differential equation based on Newtonian kinematics. , and Representing time respectively The carrier's physical state vector, specific force, and angular velocity, wherein the carrier's physical state vector includes position, velocity, and attitude. The L2 norm is represented; when the carrier motion state information is a multi-sensor fusion state, the reference physical trajectory is determined by the position, velocity and attitude state sequence of each time stamp.

8. The method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints according to claim 7, characterized in that, The total training loss function includes semantic prediction loss and motion state consistency loss, expressed as: in, To predict the semantic prediction loss between latent features and target latent features, The motion state consistency loss between the position, velocity, and attitude predictions decoded from the predicted latent features and the reference position, reference velocity, and reference attitude is calculated. These are the physical constraint weighting coefficients. Representation of time The potential features of the target corresponding to the actual future observation.

9. The method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints according to claim 1, characterized in that, The motion state consistency loss function is calculated only during the training phase based on the carrier motion state information corresponding to the prediction time window. During the inference phase, only the image and carrier motion state information up to the current moment are used, without inputting observation data from future moments, to obtain the scene state prediction result within the future time window.

10. A spatiotemporal state prediction system for an autonomous driving world model incorporating high-frequency inertial physical constraints, characterized in that, The system is used to implement the method for predicting the spatiotemporal state of an autonomous driving world model incorporating high-frequency inertial physical constraints as described in any one of claims 1 to 9, and the system comprises: The data perception module is used to synchronously acquire the continuous image frame sequence output by the vehicle-mounted multi-source vision sensor, as well as the carrier motion state information. The carrier motion state information includes the high-frequency inertial data stream output by the inertial measurement unit, or the position, attitude and velocity information obtained by fusing at least two vehicle-mounted sensors. The feature alignment processor, connected to the data perception module, is used to perform cross-modal alignment of visual latent features with motion state physical features generated based on carrier motion state information in the latent space, and adjust the fusion weights according to the confidence coefficient determined by the state estimation covariance. The lightweight evolutionary deduction module is connected to the feature alignment processor and has a built-in spatiotemporal transformation model based on a joint embedding prediction architecture for performing non-pixel reconstruction latent feature sequence prediction. A physical consistency constraint, connected to the lightweight evolutionary inference module, is used to construct a motion state consistency loss function weighted by the confidence coefficient based on the reference position, reference velocity, and reference attitude during the training phase, and to constrain the prediction output of the lightweight evolutionary inference module.

Citation Information

Patent Citations

  • End-to-end automatic driving method based on physical information fusion model

    CN122300550A