A vehicle-oriented asynchronous multi-modal continuous spatio-temporal occupancy state modeling method and system

Abstract

This invention discloses an asynchronous multimodal continuous spatiotemporal occupancy state modeling method and system for vehicles. It collects timestamped asynchronous vehicle observation data and onboard platform pose information, which are preprocessed and converted into an intermediate representation in a unified reference coordinate system. Using the query time as a reference, and combining the posterior joint state and pose information from the previous time step, a current prior joint state is obtained, including occupancy, spatiotemporal evolution, uncertainty, and consistent residual prior states. The spatiotemporal corrections for each modality are extracted from the prior residual states, the effective sampling time and extrinsic parameters are determined, and the corrected observations are propagated to the query time to obtain unified observations. The residual increments of the observations and prior states are calculated and fused to obtain the posterior consistent residual state. Based on this, the update weights of each modal spatial unit are determined, and the occupancy, evolution, and uncertainty states are jointly updated. Finally, the state results are output.

Description

Technical Field

[0001] This invention relates to the fields of multimodal environment perception, image and radar data fusion, and artificial intelligence computing model technology, and more specifically to an asynchronous multimodal continuous spatiotemporal occupancy state modeling method and system for vehicles. Background Technology

[0002] Existing solutions typically include cameras, LiDAR, and / or millimeter-wave radar mounted on the vehicle. Each sensor outputs raw observations with timestamps. The raw observations first enter time synchronization, extrinsic parameter compensation, or online calibration modules, and then enter image feature extraction networks, point cloud feature extraction networks, or bird's-eye view / voxel encoders. Subsequently, they are mapped to a bird's-eye view grid or 3D voxel space through a unified coordinate system, and finally output the occupancy status, semantic category, velocity / flow information, and grid probability at the current or future time through temporal fusion, particle prediction, or generative decoding modules. Among them, the public process of Streaming Flow can be summarized as "raw multimodal data stream—bird's-eye view feature encoding—asynchronous multisensor fusion—streaming occupancy prediction", and the public process of Occ World can be summarized as "3D occupancy—scene marker—world token—spatiotemporal generative transformer—occupancy decoding / trajectory decoding", while the public process of existing technologies is "vehicle position and previous time map—current grid map—update grid based on asynchronous sensor observations".

[0003] However, existing technologies still have the following problems:

[0004] First, the existing publicly available 3D and 4D occupation / world model schemes primarily focus on current scene reconstruction, future occupation prediction, bird's-eye view feature propagation, tokenized representation, or generative temporal modeling. Surround Occ and Occ3D mainly address 3D occupation reconstruction at the current moment; Cam4DOcc primarily establishes a 4D occupation prediction benchmark; Streaming Flow's core is asynchronous multimodal bird's-eye view feature propagation; and Occ World's core is world evolution modeling using occupancy tokens and spatiotemporal transformers. Based on the publicly available abstracts and descriptions retrieved, while these schemes involve asynchronous input, temporal modeling, or future prediction, they fail to uniformly model the cross-modal inconsistencies caused by asynchronous sampling, extrinsic parameter micro-drift, and dynamic scene changes as independent, consistent residual states that are "identifiable, outputtable, and directly involved in occupation updates." In other words, existing technologies tend to absorb inconsistencies through preprocessing calibration, implicit feature alignment, historical window stacking, or state propagation, rather than explicitly stateifying the inconsistencies themselves.

[0005] Second, while existing asynchronous occupation grid update or time-aware occupation mapping schemes improve the real-time performance of map updates, most still rely on "grid probability, grid type, particle state or timestamp" as core state variables.

[0006] Third, multimodal systems objectively suffer from time synchronization errors, acquisition delays, the influence of vehicle self-motion, and sensor drift issues during long-term operation.

[0007] Fourth, existing efficiency-optimized 3D / 4D occupancy schemes have begun to emphasize sparse representation and spatiotemporal efficiency, but their focus still differs from that of this invention. Sparse Occ focuses on a fully sparse occupancy network that models only non-free regions; STCOcc focuses on sparse spatial-temporal cascade renovation based on occupied state; and I²-World focuses on improving the compression and inference efficiency of 4D occupancy prediction through intra / inter tokenization.

[0008] Therefore, although existing technologies have made progress in areas such as 3D occupancy reconstruction, 4D occupancy prediction, asynchronous occupancy grid update, spatiotemporal online calibration, and sparsity efficiency optimization, there is still a lack of a method that can directly start from asynchronous multimodal raw observations, simultaneously model occupancy state, spatiotemporal evolution, uncertainty, and consistency residuals within a continuous-time unified framework, and directly use the consistency residuals for occupancy updates. This gap is precisely the core technical problem that this invention aims to solve. Summary of the Invention

[0009] In view of this, the present invention provides an asynchronous multimodal continuous spatiotemporal occupancy state modeling method and system for vehicles, which can solve the problem that cross-modal spatiotemporal inconsistencies caused by sampling delay, observation lag, slight extrinsic parameter drift and dynamic scene changes are difficult to explicitly model and quantify, thereby reducing occupancy ghosting, spatiotemporal jitter and dynamic flow field breaks.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A method for modeling asynchronous multimodal continuous spatiotemporal occupancy states of vehicles includes:

[0012] Collect asynchronous vehicle observation data with original timestamps and pose information of the vehicle-mounted mobile platform at each time point, and preprocess the asynchronous vehicle observation data to obtain an intermediate representation in a unified reference coordinate system.

[0013] Based on the query time, the prior joint state of the current query time is obtained based on the posterior joint state and pose information of the previous query time. The prior joint state includes the occupied state, the spatiotemporal evolution state, the uncertainty state, and the consistent residual prior state.

[0014] From the consistent residual prior state, extract the time correction and spatial correction corresponding to each mode according to the prior joint state, determine the effective sampling time and effective extrinsic parameters of the corresponding mode observation, and propagate the corrected mode observation to the query time to obtain the unified query time observation;

[0015] Based on the difference between the unified query time observation and the prior joint state, the consistency residual increment at the current query time is calculated, and the consistency residual increment is fused with the consistency residual prior state to obtain the posterior consistency residual state at the current query time.

[0016] Based on the posterior consistent residual state, the update weights of different modes for different spatial units are determined, and the occupied state, spatiotemporal evolution state and uncertainty state are jointly updated to obtain the posterior joint state at the current query time.

[0017] Output the occupancy status, spatiotemporal evolution, uncertainty, and consistency residual results at the current query time.

[0018] Preferably, the asynchronous vehicle observation data includes vehicle image data, lidar data, and millimeter-wave radar data;

[0019] The preprocessing includes: performing distortion correction, brightness normalization, invalid pixel removal, and multi-scale image feature extraction on the vehicle image data, and mapping the image features to the unified reference coordinate system based on the camera intrinsic parameters and nominal extrinsic parameters;

[0020] The lidar point cloud is denoised, outlier removed, self-motion compensated, and voxelized to form the lidar spatial features.

[0021] False alarm suppression, point correlation, and coordinate transformation are performed on millimeter-wave radar data, and feature encoding is performed on range, azimuth, relative velocity, and echo intensity to form millimeter-wave radar features;

[0022] The unified reference coordinate system is the vehicle coordinate system of the mobile platform, and the target space adopts a three-dimensional voxel mesh.

[0023] Preferably, the prior joint state includes an occupied state, a spatiotemporal evolution state, an uncertain state, and a consistent residual prior state; the occupied state is used to represent the occupied probability, idle probability, or unknown state of each voxel; the spatiotemporal evolution state is used to represent the local motion direction, velocity, flow field information, or occupied change rate of each voxel; and the uncertain state is used to represent the credibility of the occupied state and the spatiotemporal evolution state.

[0024] Preferably, the propagation of the prior joint state includes: performing self-motion compensation on the occupancy state at the previous query time using the pose information; dividing each spatial unit into static and dynamic regions based on the comparison results of the motion amplitude and / or occupancy change rate of each spatial unit in the spatiotemporal evolution state at the previous query time after self-motion compensation with a preset threshold; performing state propagation based on platform pose change on the occupancy state of the static region; performing position propagation on the occupancy state of the dynamic region, and performing local motion trend propagation on the spatiotemporal evolution state of the dynamic region; performing time propagation on the uncertain state; and performing historical residual inheritance on the consistent residual prior state; wherein the propagation adopts any one of the following: continuous time state space model, piecewise linear propagation model, spline propagation model, or analytical propagation model based on kinematic constraints.

[0025] Preferably, the correction of the nominal extrinsic parameters of the original asynchronous vehicle observation data includes:

[0026] The effective sampling time for modal observation is determined based on the time correction amount of the corresponding modality.

[0027] The effective extrinsic parameters for the mode observation are determined based on the spatial correction amount of the corresponding mode and the nominal extrinsic parameters of the mode.

[0028] Based on the valid sampling time, valid extrinsic parameters, and pose information, the modal observations are propagated to the query time to obtain unified query time observations.

[0029] Preferably, the consistency residual increment includes temporal residual increment, spatial residual increment, and dynamic residual increment;

[0030] The temporal residual increment is determined by the temporal correlation deviation or temporal alignment deviation between the post-propagation observation and the prior joint state;

[0031] The spatial residual increment is determined by the reprojection error, voxel alignment bias, or region offset between the post-propagation observation and the prior joint state;

[0032] The dynamic residual increment is determined by the local occupancy difference, velocity direction difference, flow field deviation, or dynamic continuity difference between the post-propagation observation and the prior joint state.

[0033] Preferably, the formula for obtaining the posterior consistent residual state at the current query time is as follows:

[0034] R(tq) = Γ(R-(tq), ΔR(tq));

[0035] Where Γ represents the residual fusion operator, R(tq) is the posterior consistent residual state, R-(tq) is the consistent residual prior state, and ΔR(tq) is the consistent residual increment.

[0036] Preferably, joint updating of prior joint states includes:

[0037] Based on the posterior consistent residual state R(tq) and prior uncertainty state of each mode in each spatial unit, the update weights of different modes are determined.

[0038] Write the unified query time observation into the occupied state according to the update weight to obtain the occupied probability, idle probability or unknown state of the current query time;

[0039] By combining radar velocity observation, lidar geometric changes, and image temporal characteristics of dynamic regions, the local motion direction, local flow field, or rate of change of occupation can be updated.

[0040] When the temporal residual, spatial residual, or dynamic residual of a certain region exceeds a preset threshold, the region is marked as a high mismatch region, and methods such as delayed writing, local reestimation, or outputting unknown states are adopted to suppress erroneous observations from being directly written into the occupied results.

[0041] Preferably, it further includes: a vehicle-side local update step, which determines residual hot zones based on the posterior consistency residual state and uncertainty state; performs local incremental updates on residual hot zones, and performs low-frequency updates, sparse refreshes, or historical state preservation on non-residual hot zones; wherein, a higher update frequency is used for dynamic regions than for static regions.

[0042] Preferably, an asynchronous multimodal continuous spatiotemporal occupancy state modeling system for vehicles includes: a camera, a lidar, a millimeter-wave radar, a pose information source, a time and pose management unit, a preprocessing and unified expression unit, a continuous time state modeling unit, a unified query time propagation unit, a consistent residual estimation unit, a joint update unit, a result output unit, a processor, and a memory.

[0043] The camera, lidar, millimeter-wave radar, and pose information source are electrically connected to the processor, which is electrically connected to the memory. The memory stores program instructions, sensor nominal intrinsic and extrinsic parameters, historical states, query time sequences, and model parameters. After the processor executes the program instructions, it implements the functions of a time and pose management unit, a preprocessing and unified expression unit, a continuous time state modeling unit, a unified query time propagation unit, a consistent residual estimation unit, a joint update unit, and a result output unit.

[0044] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses an asynchronous multimodal continuous spatiotemporal occupancy state modeling method and system for vehicles. Compared with the existing technologies that mainly adopt discrete frame synchronization, fixed time window fusion, implicit feature alignment, or separate processing of online calibration and occupancy modeling, the present invention, through the technical closed loop of "continuous time unified propagation - explicit consistent residual modeling - residual participation in joint update - multi-state unified output", can produce the following technical effects:

[0045] (1) This invention first constructs a continuous-time joint state and uniformly propagates asynchronous multimodal observations to the same query time for processing, thus solving the problem of inconsistent sampling times of different modal observations and the difficulty in directly fusing them under the same spatiotemporal reference from a technical mechanism perspective. As a result, images, point clouds and radar observations that were originally scattered at different sampling times can be uniformly mapped to the occupancy state update process at the same query time, thereby improving the temporal continuity and spatial consistency of continuous spatiotemporal occupancy state modeling.

[0046] (2) This invention explicitly models the inconsistencies caused by sampling delay, observation lag, slight extrinsic parameter drift, and dynamic scene changes as temporal residuals, spatial residuals, and dynamic residuals, rather than implicitly absorbing them through feature layer alignment or empirical compensation. In this way, cross-modal mismatches, which are difficult to identify and quantify individually in existing technologies, are transformed into computable, updatable, and outputtable state variables. Therefore, this invention can not only describe "what" the scene is, but also "where the inconsistencies are and why," thereby enhancing the interpretability and diagnosability of continuous spatiotemporal occupancy modeling.

[0047] (3) This invention further utilizes the consistency residuals to directly participate in the occupancy state update and uncertainty correction, so that the consistency residuals are no longer just error analysis results, but an effective adjustment quantity that enters the state update closed loop. For regions with high consistency, the update effect of corresponding observations on the occupancy state can be enhanced; for regions with poor consistency, the direct writing of mismatched observations can be suppressed, and the uncertainty characterization of the region can be improved simultaneously. Thus, the erroneous occupancy writing caused by asynchronous mismatch and slight drift can be reduced from the update mechanism, reducing the probability of problems such as occupancy ghosting, boundary jitter, and dynamic flow field breakage, and improving the stability and reliability of the results.

[0048] (4) This invention does not only output a single occupancy category or occupancy probability, but also uniformly outputs the occupancy state, spatiotemporal evolution state, uncertainty state, and consistency residual state. This output method enables the system to not only characterize the spatial occupancy relationship at the current moment, but also to characterize the changing trend of the scene over time, the reliability of the results, and the degree of cross-modal alignment. Therefore, the output results of this invention are more complete in engineering, can be directly used for environmental understanding, facilitate subsequent use of environmental world models, and provide a direct basis for system integration, fault diagnosis, and calibration maintenance.

[0049] (5) In a preferred embodiment, the present invention introduces controlled perturbations such as time delay, random frame loss, and extrinsic parameter perturbations during the training phase through a perturbation-restoration training mechanism. This enables the model to learn to restore a stable occupancy state under non-ideal observation conditions and synchronously output the corresponding residuals. This technique allows the model to be designed to adapt to asynchronous, multi-perturbation, and slight drift conditions, rather than relying on ideal synchronization and ideal calibration conditions by default. Therefore, the present invention has stronger robustness and engineering adaptability to short-term mismatch, sensor jitter, and local observation loss in real deployment environments.

[0050] (6) In an optional implementation, the present invention determines residual hot zones based on consistency residuals and uncertainties, and performs incremental updates only on local areas with significant changes or strong mismatches. Compared with the globally unified high-frequency refresh method, this method prioritizes the allocation of computing resources to areas that truly need correction, thereby reducing invalid updates and redundant calculations. Without changing the core modeling mechanism of the present invention, this implementation further enhances the deployment feasibility of the present invention on vehicle-side or other resource-constrained platforms.

[0051] (7) From the perspective of overall technical effectiveness, this invention does not simply combine asynchronous fusion, online compensation, and occupancy prediction in parallel. Instead, it establishes a unified state modeling mechanism oriented towards asynchronous multimodal inputs at the environmental modeling layer. This mechanism uses the previous continuous state as a priori, consistent residuals as an intermediary, joint updates as a closed loop, and unified multi-state outputs as the result, thus enabling the occupancy state modeling process to simultaneously possess continuity, correctability, interpretability, and scalability. Especially under non-ideal working conditions such as asynchronous sampling, slight calibration drift, dense dynamic targets, and frequent local occlusion, this invention can better demonstrate its technical advantages.

[0052] (8) Therefore, this invention can provide a complete, logically closed-loop, and engineering-implementable technical solution to address the problems in the prior art, such as "difficulty in unifying asynchronous observations, difficulty in explicitly modeling mismatch errors, susceptibility of occupation updates to interference from erroneous observations, and difficulty in balancing accuracy and interpretability of output results." Its technical effect comes directly from the synergistic effect between the various technical features, rather than from the isolated improvement of a single module. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 The flowchart of the asynchronous multimodal continuous spatiotemporal occupancy state modeling method provided by the present invention is shown.

[0055] Figure 2 This is a schematic diagram of the continuous-time joint state and consistency residual closed loop provided by the present invention.

[0056] Figure 3 A schematic diagram of the asynchronous multimodal continuous spatiotemporal occupancy state modeling system provided by the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] This invention uses a unified query time tq as the state update benchmark, defines the environmental modeling result as a continuous-time joint state, rather than only performing occupancy estimation on discrete synchronous frames; and elevates the cross-modal inconsistencies caused by asynchronous sampling, observation lag, slight extrinsic drift, and dynamic scene changes from the traditional implicit alignment error to an explicit consistent residual state in the joint state. This residual state participates in observation correction, posterior update, and result output simultaneously, thus forming a technical closed loop of "prior propagation - residual prior correction - residual increment estimation - residual-driven joint update".

[0059] The continuous-time joint state representation of query time tq is as follows:

[0060] X(tq)={O(tq), E(tq), U(tq), R(tq)}

[0061] Wherein, O(tq) represents the occupied state, indicating the occupied, idle, or unknown state of the target spatial unit at the query time and its probability; E(tq) represents the spatiotemporal evolution state, indicating the local motion trend, occupancy change rate, or flow field information of the target spatial unit in the time dimension; U(tq) represents the uncertainty state, indicating the reliability of the occupied and spatiotemporal evolution states; and R(tq) represents the consistency residual state, indicating the temporal, spatial, and dynamic inconsistencies between the asynchronous multimodal observation and the prior state at the query time. The consistency residual state is not an independent calibration parameter, nor is it merely an error quantity used for offline analysis; rather, it is a component of the continuous-time joint state.

[0062] This invention uses the vehicle coordinate system as a unified reference coordinate system and a three-dimensional voxel mesh as the target space representation. Multiple cameras are preferably surround-view cameras; the lidar is preferably located on the roof or in a high position; the millimeter-wave radar is preferably located at the front, rear, or four corners of the vehicle; and the pose information source is preferably composed of one or more of an inertial measurement unit, a combined navigation unit, a wheel speedometer, a visual odometer, or a laser odometer.

[0063] The role of this hardware combination is as follows: the camera provides dense texture and semantic information, the lidar provides relatively accurate spatial geometric point clouds, the millimeter-wave radar provides distance and velocity observation, and the pose information source provides the self-motion information required for historical state propagation and observation of spatiotemporal transformation.

[0064] like Figure 1 As shown in the figure, this invention discloses an asynchronous multimodal continuous spatiotemporal occupancy state modeling method for vehicles, including:

[0065] Collect asynchronous vehicle observation data with original timestamps and pose information of the vehicle-mounted mobile platform at each time point, and preprocess the asynchronous vehicle observation data to obtain an intermediate representation in a unified reference coordinate system.

[0066] Based on the query time, the prior joint state of the current query time is obtained based on the posterior joint state and pose information of the previous query time. The prior joint state includes the occupied state, the spatiotemporal evolution state, the uncertainty state, and the consistent residual prior state.

[0067] From the consistent residual prior state, extract the time correction and spatial correction corresponding to each mode according to the prior joint state, determine the effective sampling time and effective extrinsic parameters of the corresponding mode observation, and propagate the corrected mode observation to the query time to obtain the unified query time observation;

[0068] Based on the difference between the unified query time observation and the prior joint state, the consistency residual increment at the current query time is calculated, and the consistency residual increment is fused with the consistency residual prior state to obtain the posterior consistency residual state at the current query time.

[0069] Based on the posterior consistent residual state, the update weights of different modes for different spatial units are determined, and the occupied state, spatiotemporal evolution state and uncertainty state are jointly updated to obtain the posterior joint state at the current query time.

[0070] Output the occupancy status, spatiotemporal evolution, uncertainty, and consistency residual results at the current query time.

[0071] Specifically, the asynchronous vehicle observation data includes vehicle image data, lidar data, and millimeter-wave radar data;

[0072] The preprocessing includes: performing distortion correction, brightness normalization, invalid pixel removal, and multi-scale image feature extraction on the vehicle image data, and mapping the image features to the unified reference coordinate system based on the camera intrinsic parameters and nominal extrinsic parameters;

[0073] The lidar point cloud is denoised, outlier removed, self-motion compensated, and voxelized to form the lidar spatial features.

[0074] False alarm suppression, point correlation, and coordinate transformation are performed on millimeter-wave radar data, and feature encoding is performed on range, azimuth, relative velocity, and echo intensity to form millimeter-wave radar features;

[0075] The unified reference coordinate system is the vehicle coordinate system of the mobile platform, and the target space adopts a three-dimensional voxel mesh.

[0076] In a specific embodiment of the present invention, image data collected by multiple cameras at their respective exposure times, point cloud data collected by LiDAR at their respective scanning times, and point trace data or range-azimuth-velocity data output by millimeter-wave radar at their respective detection times are acquired, and vehicle pose information output by pose information source is acquired simultaneously. The time and pose management unit writes original timestamps to the original observations of each modality and associates and stores the original timestamps with the corresponding pose information. The purpose of this step is to preserve the original asynchronous time attributes of different modalities, rather than forcibly cropping all observations into the same discrete frame, thus preserving the true sampling time sequence from the source and providing a basis for subsequent continuous time propagation.

[0077] The preprocessing and unified representation units preprocess observations of different modalities and form a unified intermediate representation. For image data, distortion correction, invalid pixel removal, brightness normalization, and multi-scale image feature extraction are performed, and the image features are projected onto a three-dimensional voxel space under a unified reference coordinate system based on the camera's nominal intrinsic and extrinsic parameters. For lidar point clouds, denoising, outlier removal, self-motion compensation, and voxel encoding are performed to form lidar spatial features. For millimeter-wave radar data, false alarm suppression, point correlation, and coordinate transformation are performed, and the range, azimuth, relative velocity, and echo intensity are encoded as radar features.

[0078] The original observations from different modalities are converted into intermediate representations that can be compared and fused under the same reference coordinate system. This step aims to eliminate differences in data format, sampling method, and coordinate representation between different modalities, establishing a common input for unified query timing processing.

[0079] Specifically, the prior joint state includes occupied state, spatiotemporal evolution state, uncertainty state, and consistent residual prior state; occupied state is used to represent the occupied probability, idle probability, or unknown state of each voxel; spatiotemporal evolution state is used to represent the local motion direction, velocity, flow field information, or occupation change rate of each voxel; uncertainty state is used to represent the credibility of occupied state and spatiotemporal evolution state.

[0080] In a specific embodiment of the present invention, the continuous-time state modeling unit constructs the prior joint state of the current query time tq based on the posterior joint state X(tq-1) of the previous query time tq-1 and the vehicle pose change information P(tq-1:tq) within the time interval [tq-1, tq], thereby obtaining:

[0081] X-(tq)=F(X(tq-1),P(tq-1:tq),Δt)

[0082] Where X-(tq) represents the prior joint state at the current query time, F represents the continuous-time propagation operator, and Δt represents the propagation time interval.

[0083] Furthermore, the prior joint state can be further written as:

[0084] X-(tq)={O-(tq), E-(tq), U-(tq), R-(tq)}

[0085] Here, R-(tq) represents the consistent residual prior state at the current query time. The occupied state O-(tq), after propagation, provides the scene occupancy prior at the query time; the spatiotemporal evolution state E-(tq), after propagation, provides the local motion trend prior; the uncertain state U-(tq), after propagation, provides the current state credibility prior; and the consistent residual prior state R-(tq), after propagation, provides the historical residual prior required for observation correction at the current time. The purpose of this step is to first establish a continuous state benchmark for a unified query time, enabling subsequent asynchronous observations to be corrected and updated around the same query time.

[0086] Specifically, the propagation of the prior joint state includes: performing self-motion compensation on the occupancy state at the previous query time using the pose information; dividing each spatial unit into static and dynamic regions based on the comparison results of the motion amplitude and / or occupancy change rate of each spatial unit in the spatiotemporal evolution state at the previous query time after self-motion compensation with a preset threshold; performing state propagation based on platform pose change on the occupancy state of the static region; performing position propagation on the occupancy state of the dynamic region, and performing local motion trend propagation on the spatiotemporal evolution state of the dynamic region; performing time propagation on the uncertain state; and performing historical residual inheritance on the consistent residual prior state; wherein the propagation adopts any one of the following: continuous time state space model, piecewise linear propagation model, spline propagation model, or analytical propagation model based on kinematic constraints.

[0087] In a specific embodiment of the present invention, each spatial unit after self-motion compensation is divided according to the motion amplitude and occupancy change rate in the spatiotemporal evolution state at the previous query time: when the motion amplitude of the spatial unit is less than a first threshold and the occupancy change rate is less than a second threshold, it is divided into a static region; otherwise, it is divided into a dynamic region. For static regions, state propagation with platform pose changes is performed on its occupancy state; for dynamic regions, position propagation is performed on its occupancy state, and local motion trend propagation is performed on its spatiotemporal evolution state; for uncertain states, time propagation is performed according to the time interval Δt; for consistent residual prior states, historical residual inheritance is performed.

[0088] Specifically, the correction of the nominal extrinsic parameters of the original asynchronous vehicle observation data includes:

[0089] The effective sampling time for modal observation is determined based on the time correction amount of the corresponding modality.

[0090] The effective extrinsic parameters for the mode observation are determined based on the spatial correction amount of the corresponding mode and the nominal extrinsic parameters of the mode.

[0091] Based on the valid sampling time, valid extrinsic parameters, and pose information, the modal observations are propagated to the query time to obtain unified query time observations.

[0092] In a specific embodiment of the present invention, asynchronous observations based on residual priors are used to unify query time propagation.

[0093] The unified query time propagation unit performs query time propagation on the observation zi(ti) acquired by each mode i at the original sampling time ti. In order to avoid directly using the residual that has not yet been estimated at the current time for the current correction, the obtained residual prior state R-(tq) is used instead of the posterior residual state R(tq) at the current time.

[0094] Specifically, the time correction δti-(tq) and spatial correction δTi-(tq) corresponding to mode i are extracted from the residual prior state R-(tq) to obtain the effective sampling time and effective extrinsic parameters of the mode observation:

[0095] t i = ti + δti - (tq)

[0096] T i = Ti0⊕δTi-(tq)

[0097] Where Ti0 represents the nominal extrinsic parameter of mode i, and "⊕" represents the incremental correction based on the nominal extrinsic parameter. Then, based on the vehicle pose change within the time interval [ti, tq], the effective sampling time ti, and the effective extrinsic parameter Ti, the observation zi(ti) of mode i is propagated to the unified query time tq to obtain the unified query time observation z. i(tq).

[0098] For the image mode, image features are reprojected onto the 3D voxel space at the query time using corrected camera extrinsic parameters and vehicle pose changes. For the lidar mode, motion compensation of the point cloud is performed using vehicle pose changes and mapped to the voxel space at the query time. For the millimeter-wave radar mode, point traces and velocity information are transformed to the query time based on corrected extrinsic parameters and vehicle pose changes. The purpose of this step is to perform spatiotemporal correction on asynchronous observations using historical residual priors, so that observations from different modes and different sampling times are uniformly converted into observational representations that can be directly compared at the current query time.

[0099] Specifically, the consistency residual increment includes temporal residual increment, spatial residual increment, and dynamic residual increment;

[0100] The temporal residual increment is determined by the temporal correlation deviation or temporal alignment deviation between the post-propagation observation and the prior joint state;

[0101] The spatial residual increment is determined by the reprojection error, voxel alignment bias, or region offset between the post-propagation observation and the prior joint state;

[0102] The dynamic residual increment is determined by the local occupancy difference, velocity direction difference, flow field deviation, or dynamic continuity difference between the post-propagation observation and the prior joint state.

[0103] In a specific embodiment of the present invention, the consistency residual estimation unit obtains the unified query time observation z. i(tq) is compared with the prior joint state X-(tq) to estimate the consistency residual increment ΔR(tq) at the current query time, i.e.:

[0104] ΔR(tq)=Ψ(X-(tq), {z i(tq)})

[0105] Where Ψ represents the residual increment estimation operator.

[0106] The consistency residual increment ΔR(tq) includes at least the time residual increment ΔRt(tq), the spatial residual increment ΔRs(tq), and the dynamic residual increment ΔRd(tq), that is:

[0107] ΔR(tq)={ΔRt(tq), ΔRs(tq), ΔRd(tq)}

[0108] Specifically, the temporal residual increment characterizes the alignment deviation between the post-propagation observation and the prior state in the time dimension; the spatial residual increment characterizes the spatial deviation between the post-propagation observation and the prior state on the projection, mapping, or voxel boundary; and the dynamic residual increment characterizes the dynamic deviation between the post-propagation observation and the prior evolved state in terms of velocity, flow field, and local occupancy changes. Preferably, the residual increment can be estimated at the voxel level, region level, or target level. The purpose of this step is to explicitly extract the cross-modal inconsistency quantity that actually occurs at the current moment, rather than continuing to be absorbed by implicit feature alignment.

[0109] Specifically, the formula for obtaining the posterior consistent residual state at the current query time is as follows:

[0110] R(tq) = Γ(R-(tq), ΔR(tq));

[0111] Where Γ represents the residual fusion operator, R(tq) is the posterior consistent residual state, R-(tq) is the consistent residual prior state, and ΔR(tq) is the consistent residual increment.

[0112] Specifically, joint updates to the prior joint states include:

[0113] Based on the posterior consistent residual state R(tq) and prior uncertainty state of each mode in each spatial unit, the update weights of different modes are determined.

[0114] Write the unified query time observation into the occupied state according to the update weight to obtain the occupied probability, idle probability or unknown state of the current query time;

[0115] By combining radar velocity observation, lidar geometric changes, and image temporal characteristics of dynamic regions, the local motion direction, local flow field, or rate of change of occupation can be updated.

[0116] When the temporal residual, spatial residual, or dynamic residual of a certain region exceeds a preset threshold, the region is marked as a high mismatch region, and methods such as delayed writing, local reestimation, or outputting unknown states are adopted to suppress erroneous observations from being directly written into the occupied results.

[0117] Specifically, it also includes: a vehicle-side local update step, which determines residual hot zones based on the posterior consistency residual state and uncertainty state; performs local incremental updates on residual hot zones, and performs low-frequency updates, sparse refreshes, or historical state preservation on non-residual hot zones; wherein, a higher update frequency is used for dynamic regions than for static regions.

[0118] In a specific embodiment of the present invention, a combined update of consistent residual posterior and residual-driven update is used.

[0119] The joint update unit first forms the posterior-consistent residual state R(tq) at the current query time based on the obtained residual prior state R-(tq) and the obtained residual increment ΔR(tq), that is:

[0120] R(tq)=Γ(R-(tq),ΔR(tq))

[0121] Where Γ represents the residual fusion operator.

[0122] Subsequently, the joint update unit directly incorporates the posterior consistent residual state R(tq) into the state update process at the current query time, jointly updating the occupied state O(tq), the spatiotemporal evolution state E(tq), and the uncertain state U(tq) to obtain the posterior joint state at the current query time:

[0123] X(tq) = H(X - (tq), {z} i(tq)},R(tq))

[0124] Here, H represents the joint update operator.

[0125] The joint update operator includes at least the following sub-procedures:

[0126] First, modal weight allocation: based on the posterior consistent residual state R(tq) and prior uncertainty state U-(tq) of each mode in each spatial cell, the update weight of different modes is determined; the smaller the residual and the lower the prior uncertainty, the greater the update weight of the corresponding mode for that spatial cell.

[0127] Secondly, the occupancy status is updated by writing the unified query time observation into the occupancy status according to the update weight, thereby obtaining the occupancy probability, idle probability, or unknown status at the current query time.

[0128] Third, the spatiotemporal evolution state is updated by combining radar velocity observation, lidar geometric changes and image temporal characteristics of dynamic regions to update the local motion direction, local flow field or occupancy rate of change.

[0129] Fourth, regarding the update of uncertain states, for regions with large consistency residuals, the uncertainty state is increased; for regions with small consistency residuals and high multimodal consistency, the uncertainty state is reduced or the confidence level is maintained.

[0130] Fifth, high mismatch region suppression: when the time residual, spatial residual or dynamic residual of a certain region exceeds the preset threshold, the region is marked as a high mismatch region. For this region, delayed writing, local reestimation or output of unknown state are adopted to suppress the direct writing of erroneous observations into the occupied results.

[0131] In summary, the purpose of this embodiment is to make the consistency residual no longer just an error analysis result, but a direct adjustment quantity in the occupied state update closed loop, thereby suppressing erroneous occupied writes caused by asynchronous mismatch and slight drift from the update mechanism.

[0132] Furthermore, the results are output. The results output unit outputs the occupancy status result, spatiotemporal evolution result, uncertainty result, and consistency residual result at the current query time. The occupancy status result can be represented as a 3D voxel occupancy map or occupancy probability map at the query time; the spatiotemporal evolution result can be represented as a short-term occupancy change trend, local flow field, or velocity distribution; the uncertainty result can be represented as a confidence map, variance map, or entropy map; and the consistency residual result can be represented as a temporal residual map, spatial residual map, dynamic residual map, or a combination thereof. The purpose of this step is not only to output "what occupies the scene," but also "how the scene changes," "whether the result is reliable," and "which regions exhibit cross-modal inconsistencies."

[0133] Furthermore, in a preferred embodiment, the continuous-time propagation operator F is implemented using a continuous-time state-space model, a learnable piecewise propagation model, or an analytical propagation model based on kinematic constraints; however, regardless of the propagation form used, the consistent residual state is required to participate in the propagation as part of the continuous-time joint state, rather than being treated as an external independent calibration parameter.

[0134] Furthermore, in a preferred embodiment, the consistency residual estimation can be trained using explicit supervision or weak supervision.

[0135] Preferably, during the training phase, controlled time delays, random frame drops, and extrinsic parameter perturbations are injected into the original training samples to construct a non-ideal asynchronous operating condition. Under this non-ideal asynchronous operating condition, the occupied state output, spatiotemporal evolution output, uncertainty output, and consistency residual output are simultaneously monitored, enabling the model to identify asynchronous mismatches and slight drifts. This implementation method is used to enhance the robustness of the system.

[0136] Furthermore, in a preferred embodiment, residual hot zones can be determined based on the consistent residual state and the uncertain state. Local incremental updates are performed only on these residual hot zones, while low-frequency updates or historical state preservation are performed on long-term stable regions. This embodiment is used to improve deployment efficiency on resource-constrained platforms.

[0137] Furthermore, without changing the core concept of the present invention, the sensor combination can be replaced by a combination of camera and lidar, camera and millimeter-wave radar, or lidar and millimeter-wave radar; the spatial representation can be replaced by a sparse voxel grid, an octree structure, or a bird's-eye view superimposed height layer structure; however, the essential technical feature of the present invention always lies in: establishing a continuous-time joint state including occupied state, spatiotemporal evolution state, uncertain state, and consistent residual state, using residual prior to perform unified query time correction on asynchronous observations, estimating the residual increment at the current time, and using posterior consistent residual to drive joint update of occupied state.

[0138] Further explanation of terms and abbreviations:

[0139] O(tq): Occupancy State;

[0140] E(tq): Evolution State;

[0141] U(tq): Uncertainty;

[0142] R(tq): Consistency Residual;

[0143] BEV: Bird's Eye View;

[0144] IMU: Inertial Measurement Unit;

[0145] GNSS: Global Navigation Satellite System;

[0146] tq: Query time, refers to the target time when the present invention performs unified state update and outputs results;

[0147] Ti0: Nominal extrinsic parameter of mode i;

[0148] δti-(tq): The time correction for mode i given by the residual prior state;

[0149] δTi-(tq): Modal i-space correction given by the residual prior state.

[0150] Furthermore, the model used to estimate the consistency residual increment is obtained as follows: a controlled perturbation is injected into the training samples, the controlled perturbation including one or more of time delay perturbation, random frame loss perturbation, and extrinsic parameter micro-perturbation; the training samples after the controlled perturbation are input into the modeling method to obtain the occupied state output, spatiotemporal evolution output, uncertainty output, and consistency residual output; the occupied state output, spatiotemporal evolution output, uncertainty output, and consistency residual output are jointly supervised and trained based on reference labels to enable the model to have the ability to identify asynchronous mismatch and slight drift.

[0151] Specifically, such as Figure 3 As shown, an asynchronous multimodal continuous spatiotemporal occupancy state modeling system for vehicles includes: a camera, a lidar, a millimeter-wave radar, a pose information source, a time and pose management unit, a preprocessing and unified expression unit, a continuous time state modeling unit, a unified query time propagation unit, a consistent residual estimation unit, a joint update unit, a result output unit, a processor, and a memory.

[0152] The camera, lidar, millimeter-wave radar, and pose information source are electrically connected to the processor, which is electrically connected to the memory. The memory stores program instructions, sensor nominal intrinsic and extrinsic parameters, historical states, query time sequences, and model parameters. After the processor executes the program instructions, it implements the functions of a time and pose management unit, a preprocessing and unified expression unit, a continuous time state modeling unit, a unified query time propagation unit, a consistent residual estimation unit, a joint update unit, and a result output unit.

[0153] Figure 3 The main functional units of the system of the present invention and their logical connections are illustrated. In one embodiment, the system further includes a processor and a memory; the memory is electrically connected to the processor, and the processor calls program instructions stored in the memory to implement the functions of a time and pose management unit, a preprocessing and unified representation unit, a continuous time state modeling unit, a unified query time propagation unit, a consistent residual estimation unit, a joint update unit, and a result output unit. A camera, LiDAR, millimeter-wave radar, and pose information source are communicatively connected to the processor via a vehicle bus or data interface.

[0154] Furthermore, such as Figure 2 The diagram illustrates a closed-loop representation of the continuous-time joint state and the consistency residual in this embodiment of the invention. It shows the process by which the posterior joint state X(tq-1) of the previous query time is processed by the continuous-time state modeling unit to obtain the prior joint state X-(tq). The prior joint state X-(tq) includes the occupied state, the spatiotemporal evolution state, the uncertain state, and the consistency residual prior. The asynchronous unified expression observation {zi(ti)}, after being processed by the unified query time propagation unit, is input together with the prior joint state into the consistency residual estimation unit and the joint update unit to complete the consistency residual estimation and joint state update at the current query time, resulting in the posterior joint state X(tq) at the current query time. Figure 2 The dashed feedback path further illustrates the recursive relationship in which the posterior joint state X(tq) at the current query time serves as the input for the next query time and participates in subsequent cyclic updates. Figure 2 It is mainly used to explain the core innovative mechanism of the present invention, namely the closed-loop modeling process formed by continuous-time joint state construction, explicit consistency residual introduction, and residual-driven joint update.

[0155] Specific embodiments of the method of the present invention are as follows:

[0156] Example 1: Implementation of an Onboard Asynchronous Multimodal Continuous Spatiotemporal Occupancy State Modeling System

[0157] The asynchronous multimodal continuous spatiotemporal occupancy state modeling system of this embodiment is set on a vehicle platform and includes multiple cameras, LiDAR, millimeter-wave radar, time and pose management unit, preprocessing and unified expression unit, continuous time state modeling unit, unified query time propagation unit, consistent residual estimation unit, joint update unit, result output unit, processor, memory and pose information source.

[0158] Among them, multiple cameras are used to collect images of the environment around the vehicle, preferably including surround-view cameras arranged in the front, rear, left, right and diagonal directions; LiDAR is used to output three-dimensional point clouds of the scene; millimeter-wave radar is used to output target distance, orientation, relative speed and point information; pose information source is used to output the pose information of the vehicle at different times, preferably one or more combinations of inertial measurement unit, integrated navigation unit, wheel speed meter, visual odometer and laser odometer.

[0159] Multiple cameras, LiDAR, millimeter-wave radar, and pose information sources are electrically connected to the processor. This electrical connection can be achieved via vehicle Ethernet, Controller Area Network (CAN) bus, or other vehicle communication buses. The processor is electrically connected to a memory containing executable program instructions and model parameters. After executing the program instructions in the memory, the processor implements the functions of a time and pose management unit, a preprocessing and unified representation unit, a continuous-time state modeling unit, a unified query time propagation unit, a consistent residual estimation unit, a joint update unit, and a result output unit. In other words, these functional units can be implemented separately by hardware circuits, or preferably by the same processor calling different software modules.

[0160] In terms of static relationships, multiple cameras, LiDAR, millimeter-wave radar, and pose information sources serve as front-end observation sources; the time and pose management unit and the preprocessing and unified representation unit serve as front-end data processing and unified representation modules; the continuous-time state modeling unit, the unified query time propagation unit, the consistent residual estimation unit, and the joint update unit serve as core modeling modules; and the result output unit serves as the back-end result generation module. In terms of dynamic relationships, at each query time, the front-end observation sources output asynchronous vehicle observation data. The time and pose management unit assigns a unified time scale and pose reference to this data. The preprocessing and unified representation unit forms a fusionable unified intermediate representation. The continuous-time state modeling unit constructs and propagates the prior state of the current query time. The unified query time propagation unit maps different modal observations to the same query time. The consistent residual estimation unit calculates the temporal residual, spatial residual, and dynamic residual. The joint update unit updates the occupied state, spatiotemporal evolution state, and uncertain state based on the residuals. Finally, the result output unit outputs the continuous spatiotemporal occupied state result and its associated results.

[0161] In this embodiment, the processor may be a central processing unit, a graphics processing unit, a neural network processor, or a combination thereof; the memory may include random access memory, read-only memory, flash memory, or solid-state memory. The result output unit may send the output results to the environmental understanding module, environmental world model module, or display terminal on the vehicle, but does not involve planning, decision-making, or control execution.

[0162] Example 2: An Example of an Asynchronous Multimodal Continuous Spatiotemporal Occupancy State Modeling Method

[0163] This embodiment provides an asynchronous multimodal continuous spatiotemporal occupancy state modeling method based on the system implemented in Embodiment 1. For ease of explanation, the following description uses a vehicle platform simultaneously equipped with multiple cameras, a LiDAR, multiple millimeter-wave radars, and a pose information source as an example.

[0164] In this embodiment, the vehicle coordinate system is used as the unified reference coordinate system, where the forward direction is the positive X-axis, the left direction is the positive Y-axis, and the top direction is the positive Z-axis. The space of interest can be set as a three-dimensional region within a certain range in front of, to the side of, and behind the vehicle, and discretized into a three-dimensional voxel mesh; in another embodiment, a bird's-eye view overlaid with height layers can also be used. The query time can be set with a fixed step size, for example, triggering a state update every preset time interval; alternatively, the query time can be triggered by a sensor arrival event.

[0165] The continuous-time joint state in this embodiment is denoted as:

[0166] X(tq)={O(tq),E(tq),U(tq),R(tq)}

[0167] Where X(tq) represents the joint state at query time tq; O(tq) represents the occupied state, used to describe the occupied probability, idle probability, or unknown state of each spatial unit at query time; E(tq) represents the spatiotemporal evolution state, used to describe the short-term motion trend, flow field direction, velocity information, or occupancy change rate of each spatial unit; U(tq) represents the uncertainty state, used to describe the reliability of the occupied and evolution states; and R(tq) represents the consistency residual state, used to describe the temporal inconsistency, spatial inconsistency, and dynamic inconsistency between asynchronous observations and prior states.

[0168] This embodiment includes the following steps:

[0169] Step S101: Asynchronous multimodal observation and acquisition

[0170] Multiple cameras acquire environmental images at their respective exposure times, LiDAR acquires scene point clouds at its respective scanning times, and millimeter-wave radar outputs point traces, distance, azimuth, and relative velocity information at its respective detection times. A pose information source simultaneously outputs the vehicle's attitude and displacement information at each moment. The time and pose management unit writes raw timestamps to each modal data and associates and stores the vehicle pose information with the observation data.

[0171] Because cameras, lidar, and millimeter-wave radars have different sampling frequencies, transmission delays, and exposure or scanning durations in actual operation, step S101 yields an asynchronous multimodal observation sequence, rather than strictly synchronized observation frames at the same time. This invention does not force them to be cropped into the same discrete frame, but rather retains the original asynchronous temporal attributes, providing a basis for subsequent continuous-time propagation.

[0172] Step S102: Raw data preprocessing and unified representation

[0173] The preprocessing and unified representation units preprocess data of different modalities respectively.

[0174] For image data, distortion correction, brightness normalization, and invalid pixel removal are performed first. Then, an image feature extraction network extracts multi-scale image features. The image feature extraction network can be implemented using a convolutional neural network, a visual Transformer network, or a combination of both. The extracted image features, combined with camera intrinsic parameters and nominal extrinsic parameters, can be projected onto a three-dimensional volume space or bird's-eye view space under a unified reference coordinate system.

[0175] For lidar point clouds, denoising, outlier removal, and self-motion compensation are performed first, followed by voxelization, columnarization, or point-level coding to form the spatial features of the lidar.

[0176] For millimeter-wave radar data, false alarm points are first removed, points are associated, and coordinates are transformed. Then, the range, azimuth, relative velocity, and echo intensity are encoded as radar features. For implementations with multi-frame radar tracking capabilities, radar time-series features with velocity priors can also be formed.

[0177] After the above processing, image features, LiDAR features, and millimeter-wave radar features are all expressed as intermediate features that can participate in fusion under a unified reference coordinate system. This step aims to eliminate the differences in the original data formats of each modality, establishing a common input format for subsequent unified query time propagation and joint updates.

[0178] Step S103: Construct a continuous-time joint state and perform prior propagation

[0179] The continuous-time state modeling unit, based on the joint state X(tq-1) of the previous query time and combined with the vehicle's pose changes within the time interval [tq-1,tq], propagates the prior state of the current query time tq to obtain:

[0180] X-(tq)=F(X(tq-1),P(tq-1:tq),Δt)

[0181] Where X-(tq) represents the prior joint state at the current query time, F represents the continuous time propagation operator, P(tq-1:tq) represents the pose change information of the vehicle between the previous query time and the current query time, and Δt represents the propagation time interval.

[0182] Specifically, when the occupancy state O(tq-1) propagates to the current query time, it mainly performs self-motion compensation for static regions, and propagates the position and shape of dynamic regions in combination with historical evolution trends; the spatiotemporal evolution state E(tq-1) is used to predict the future short-term occupancy change trend during propagation; the uncertainty state U(tq-1) can be appropriately attenuated or expanded according to the increase of time interval during propagation; the consistency residual state R(tq-1) can be retained as historical residual information to the current time to characterize whether historical inconsistency is still continuing.

[0183] In a preferred embodiment, the continuous-time propagation operator F can be implemented by a learnable continuous-time state-space model; in another embodiment, it can also be implemented by a piecewise linear propagation model, a spline interpolation propagation model, or an analytical propagation model based on kinematic constraints.

[0184] Step S104: Asynchronous observations propagate to the unified query time.

[0185] To avoid directly using the posterior residual, which has not yet been estimated at the current query time, for the current correction, this embodiment calls the consistent residual prior state R obtained in step S103. - (tq), instead of the posterior consistent residual state R(tq) at the current query time. For any mode i, its original observation is denoted as zi(ti), where ti is the original sampling time of the observation. Considering sampling delay and slight extrinsic parameter drift, this invention uses the effective sampling time and effective extrinsic parameters after residual correction for this observation, resulting in:

[0186] t'i=ti+Δti

[0187] T'i=Ti0⊕ΔTi

[0188] Where Δti represents the time residual component, Ti0 represents the nominal extrinsic parameter of mode i, ΔTi represents the spatial residual component, and "⊕" represents the incremental correction based on the nominal extrinsic parameter.

[0189] After correction, the unified query time propagation unit propagates zi(ti) to tq based on the vehicle's pose change and modal projection relationship within the time interval [ti,tq], forming the unified query time observation z'i(tq).

[0190] For image modalities, the image features are projected onto a three-dimensional voxel space or bird's-eye view space under a unified reference coordinate system using the corrected camera extrinsic parameters T'i and vehicle pose changes.

[0191] For the lidar mode, motion compensation is performed on the scanned point cloud by utilizing the vehicle pose change, and the point cloud is mapped to a unified reference coordinate system through the corrected extrinsic parameter T'i.

[0192] For millimeter-wave radar modes, the point trace and velocity information are transformed to a unified query time based on the vehicle pose change and the corrected extrinsic parameter T'i.

[0193] Through step S104, observations obtained from different modalities and different sampling times are uniformly converted into observation expressions that can be directly compared and fused at the current query time, providing a common reference system for explicit residual estimation.

[0194] Step S105: Explicit Consistent Residual Estimation

[0195] The consistent residual estimation unit compares the unified query time observations z'i(tq) of each modality with the prior joint state X-(tq) to obtain the consistent residual state R(tq). In this embodiment, the consistent residual state includes at least three parts: the temporal residual Rt, the spatial residual Rs, and the dynamic residual Rd, namely:

[0196] R(tq) = {Rt(tq), Rs(tq), Rd(tq)}

[0197] Among them, the temporal residual Rt is used to characterize the degree of misalignment of the observation in the time dimension; the spatial residual Rs is used to characterize the degree of deviation of the observation in spatial projection, voxel mapping or extrinsic parameter transformation; and the dynamic residual Rd is used to characterize the degree of dynamic inconsistency between the observation and the prior evolutionary state.

[0198] Specifically, the temporal residual Rt can be calculated by observing the arrival time, the temporal correlation deviation between the propagation features and the prior state, or the consistency of the target position change before and after propagation; the spatial residual Rs can be calculated by reprojection error, voxel alignment deviation, local region offset, or multimodal boundary misalignment; and the dynamic residual Rd can be calculated by local occupancy difference, velocity direction difference, flow field deviation, and dynamic target continuity loss.

[0199] In a preferred embodiment, the residuals can be estimated separately at the voxel level, region level, or target level, and then fused to form a unified residual map. The core of this step is to explicitly convert the cross-modal inconsistencies, which are usually absorbed by implicit feature alignment in existing technologies, into state variables that can participate in subsequent updates.

[0200] Step S106: Joint update based on consistent residuals

[0201] The joint update unit performs a joint update on the occupancy state, spatiotemporal evolution state, uncertainty state, and consistency residual state at the current query time based on the prior joint state X-(tq), the observation z'i(tq) at the unified query time, and the consistency residual R(tq), resulting in:

[0202] X(tq)=H(X-(tq),{z'i(tq)},R(tq))

[0203] Here, H represents the joint update operator.

[0204] In this embodiment, the joint update operator H preferably includes five sub-processes: weight allocation, occupancy update, evolution update, uncertainty update, and residual update.

[0205] First, weight allocation. The update weight αi(v) for each mode i in each spatial cell v can be determined by the residual size and prior uncertainty corresponding to that spatial cell. The smaller the residual and the lower the prior uncertainty, the larger the update weight of the corresponding mode; the larger the residual and the higher the prior uncertainty, the smaller the update weight of the corresponding mode.

[0206] Second, occupation update. The unified query time observations of different modalities are written into the occupation state O(tq) according to the update weight, thereby obtaining the occupation probability, idle probability or unknown state of each spatial unit at the current query time.

[0207] Third, evolution and update. For spatial units with motion trends, the spatiotemporal evolution state E(tq) is updated using radar velocity information, historical point cloud changes, and image temporal features, thereby obtaining the local flow field, velocity direction, or occupation change trend.

[0208] Fourth, uncertainty update. For regions with large residuals, increase the uncertainty state U(tq) for that region to reflect the relatively low confidence level of the results in that region; for regions with high multimodal consistency, reduce uncertainty or maintain a high confidence level.

[0209] Fifth, residual update. The consistent residuals obtained from the current estimation are merged with the historical residual states to form R(tq) at the current time step, which is used for propagation in the next time step and subsequent output.

[0210] In one optional implementation, when the time residual, spatial residual or dynamic residual of a certain region exceeds a preset threshold, the joint update unit can mark the region as a high mismatch region and trigger local re-estimation, delayed writing or temporary output of an unknown state for the region, so as to further avoid errors occupying the write.

[0211] Step S107: Output the result

[0212] The result output unit outputs the occupancy status result, spatiotemporal evolution result, uncertainty result, and consistency residual result at the current query time.

[0213] Among them, the occupancy status result can be represented as an occupancy probability map, occupancy category map, or accessibility result of three-dimensional voxels; the spatiotemporal evolution result can be represented as the short-term future occupancy change trend, local flow field, or velocity distribution; the uncertainty result can be represented as a variance map, entropy map, confidence map, or interval estimation result; the consistency residual result can be represented as a time residual map, spatial residual map, dynamic residual map, or a combination of the three.

[0214] The above output results can be used directly by the environmental understanding system, or as a unified input basis for the environmental world model. It should be noted that the environmental world model described in this invention refers only to the upper-level modeling module that provides a unified description of the spatiotemporal state of the external environment, and does not involve planning, decision-making, or control execution.

[0215] Example 3: Perturbation-Recovery Training Example

[0216] To improve the stability and residual identification capability of the present invention under non-ideal asynchronous operating conditions, this embodiment provides a preferred training method.

[0217] First, training data containing images, point clouds, millimeter-wave radar, and vehicle pose information is collected or constructed. Training labels can be generated from high-precision offline multi-frame fusion results, offline high-precision reconstruction results, or high-quality occupancy annotation results.

[0218] Secondly, the perturbation injection unit applies controlled perturbations to the original training samples. The controlled perturbations include at least one or more of the following: time delay perturbation, random frame loss perturbation, extrinsic micro-perturbation, and local dynamic perturbation.

[0219] Time delay perturbation refers to increasing or decreasing a preset time deviation in the effective sampling time of a certain mode observation to simulate asynchronous sampling and transmission lag.

[0220] Random frame loss perturbation refers to the blocking of a certain mode observation at a certain time under a preset probability to simulate data loss under conditions of temporary sensor failure or bandwidth limitation.

[0221] External parameter micro-perturbation refers to applying small-range translational and rotational increments to the nominal external parameters of a sensor to simulate slight calibration drift caused by vehicle vibration, thermal deformation, or long-term operation.

[0222] Local dynamic perturbation refers to perturbing the observed position, velocity, or visibility of a local dynamic target to simulate sudden occlusion or sudden change in target motion.

[0223] Subsequently, the perturbed training samples are input into the method flow of Example 2 to obtain the occupancy state results, spatiotemporal evolution results, uncertainty results, and consistency residual results, which are then jointly supervised with the reference labels. The joint loss function can be expressed as:

[0224] L=λ1·Locc+λ2·Levo+λ3·Lunc+λ4·Lres+λ5·Lcons

[0225] Where L represents the total loss; Locc represents the occupied state supervision loss; Levo represents the spatiotemporal evolution supervision loss; Luc represents the uncertainty constraint loss; Lres represents the consistency residual supervision loss; Lcons represents the cross-modal consistency constraint loss; and λ1 to λ5 are the corresponding loss weight coefficients.

[0226] In the presence of known injected perturbations, the temporal and spatial residuals can be directly monitored using the amount of injected perturbation as the monitoring target. In the absence of explicit residual labels, the residuals can also be indirectly monitored through the consistency error between the propagated multimodal observations and the high-precision reference state.

[0227] The purpose of this training method is to allow the model to experience non-ideal conditions such as asynchrony, missing frames, and slight drift during the training phase, so that it can actively identify residuals and stably output continuous spatiotemporal occupancy states during the deployment phase, rather than being effective only under ideal synchronization conditions.

[0228] Example 4: Optional Partial Update Implementation for Vehicle Deployment

[0229] In another optional embodiment of the present invention, residual hot zones and stable regions can be generated based on the consistency residual results and the uncertainty results.

[0230] The local incremental update unit uses a higher frequency and finer granularity of state updates for residual hot zones, and a lower frequency of updates, historical state preservation, or sparse refresh for stable regions. For static background regions, low-frequency propagation combined with low-frequency updates is preferred; for dynamic target regions, high-frequency propagation combined with high-frequency updates is preferred. This reduces invalid global updates and improves deployment efficiency while maintaining the core modeling mechanism of this invention.

[0231] It should be noted that Embodiment 4 is a preferred embodiment of the present invention, used to illustrate the vehicle-side deployment configuration of the present invention, and does not constitute a restrictive technical feature that must be adopted by the present invention.

[0232] Summary of the static relationships, dynamic relationships, and effects of this invention:

[0233] From a static perspective, this invention has an implementable hardware foundation consisting of multiple cameras, LiDAR and millimeter-wave radar, pose information source, processor and memory, and a clear functional chain consisting of a time and pose management unit, a preprocessing and unified expression unit, a continuous time state modeling unit, a unified query time propagation unit, a consistent residual estimation unit, a joint update unit and a result output unit.

[0234] From a dynamic perspective, this invention executes a closed-loop process of "asynchronous acquisition - unified expression - prior propagation - unified query time propagation - residual estimation - joint update - result output" at each query time. The consistency residual is not an independent analytical quantity, but directly participates in occupation update and uncertainty correction.

[0235] In terms of its effects, this invention can explicitly state the cross-modal inconsistencies caused by asynchronous sampling, slight calibration drift, and dynamic scene changes, and incorporate them into the continuous-time joint state modeling framework, thereby outputting the occupied state, spatiotemporal evolution state, uncertain state, and consistent residual state more stably.

[0236] Terminology and English Abbreviations

[0237] Occupancy State: refers to the state of a target space unit at a certain moment, which is occupied by an obstacle, is empty, or is unknown, and the probability thereof.

[0238] Evolution State: refers to the trend of the occupied state changing over time, which can be reflected in velocity, flow field, direction or rate of change of occupation.

[0239] Uncertainty: refers to the model's estimate of the reliability of the output results.

[0240] Consistency Residual: refers to the spatiotemporal inconsistency between the prior states at different modal observation times and the unified query time.

[0241] Bird's Eye View (BEV): A two-dimensional representation of a scene viewed from above.

[0242] Inertial Measurement Unit (IMU): A sensor unit used to output acceleration and angular velocity.

[0243] Global Navigation Satellite System (GNSS): refers to a satellite navigation system used to provide absolute positioning information.

[0244] Controller Area Network (CAN): refers to the bus communication method commonly used inside vehicles.

[0245] Voxel: refers to a volume unit after three-dimensional space is discretized.

[0246] Query time: refers to the target time when the present invention performs unified state update and result output, which can be a fixed period time or an event trigger time.

[0247] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0248] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for modeling asynchronous multimodal continuous spatiotemporal occupancy states of vehicles, characterized in that, include: Collect asynchronous vehicle observation data with original timestamps and pose information of the vehicle-mounted mobile platform at each time point, and preprocess the asynchronous vehicle observation data to obtain an intermediate representation in a unified reference coordinate system. Based on the query time, the prior joint state of the current query time is obtained based on the posterior joint state and pose information of the previous query time. The prior joint state includes the occupied state, the spatiotemporal evolution state, the uncertainty state, and the consistent residual prior state. From the consistent residual prior state, extract the time correction and spatial correction corresponding to each mode according to the prior joint state, determine the effective sampling time and effective extrinsic parameters of the corresponding mode observation, and propagate the corrected mode observation to the query time to obtain the unified query time observation; Based on the difference between the unified query time observation and the prior joint state, the consistency residual increment at the current query time is calculated, and the consistency residual increment is fused with the consistency residual prior state to obtain the posterior consistency residual state at the current query time. Based on the posterior consistent residual state, the update weights of different modes for different spatial units are determined, and the occupied state, spatiotemporal evolution state and uncertainty state are jointly updated to obtain the posterior joint state at the current query time. Output the occupancy status, spatiotemporal evolution, uncertainty, and consistency residual results at the current query time.

2. The asynchronous multimodal continuous spatiotemporal occupancy state modeling method for vehicles according to claim 1, characterized in that, The asynchronous vehicle observation data includes vehicle image data, lidar data, and millimeter-wave radar data; The preprocessing includes: performing distortion correction, brightness normalization, invalid pixel removal, and multi-scale image feature extraction on the vehicle image data, and mapping the image features to the unified reference coordinate system based on the camera intrinsic parameters and nominal extrinsic parameters; The lidar point cloud is denoised, outlier removed, self-motion compensated, and voxelized to form the lidar spatial features. False alarm suppression, point correlation, and coordinate transformation are performed on millimeter-wave radar data, and feature encoding is performed on range, azimuth, relative velocity, and echo intensity to form millimeter-wave radar features; The unified reference coordinate system is the vehicle coordinate system of the mobile platform, and the target space adopts a three-dimensional voxel mesh.

3. The asynchronous multimodal continuous spatiotemporal occupancy state modeling method for vehicles according to claim 2, characterized in that, The prior joint state includes the occupied state, the spatiotemporal evolution state, the uncertainty state, and the consistent residual prior state. The occupied state is used to represent the occupied probability, idle probability, or unknown state of each voxel. The spatiotemporal evolution state is used to represent the local motion direction, velocity, flow field information, or occupation change rate of each voxel. The uncertainty state is used to represent the credibility of the occupied state and the spatiotemporal evolution state.

4. The asynchronous multimodal continuous spatiotemporal occupancy state modeling method for vehicles according to claim 3, characterized in that, The propagation of the prior joint state includes: performing self-motion compensation on the occupancy state at the previous query time using the pose information; dividing each spatial unit into static and dynamic regions based on the comparison results of the motion amplitude and / or occupancy change rate of each spatial unit in the spatiotemporal evolution state at the previous query time after self-motion compensation with a preset threshold; performing state propagation based on platform pose change on the occupancy state of the static region; performing position propagation on the occupancy state of the dynamic region, and performing local motion trend propagation on the spatiotemporal evolution state of the dynamic region; performing time propagation on the uncertain state; and performing historical residual inheritance on the consistent residual prior state; wherein the propagation adopts any one of the following: continuous time state space model, piecewise linear propagation model, spline propagation model, or analytical propagation model based on kinematic constraints.

5. The asynchronous multimodal continuous spatiotemporal occupancy state modeling method for vehicles according to claim 1, characterized in that, Correcting the nominal extrinsic parameters of the original asynchronous vehicle observation data includes: The effective sampling time for modal observation is determined based on the time correction amount of the corresponding modality. The effective extrinsic parameters for the mode observation are determined based on the spatial correction amount of the corresponding mode and the nominal extrinsic parameters of the mode. Based on the valid sampling time, valid extrinsic parameters, and pose information, the modal observations are propagated to the query time to obtain unified query time observations.

6. The asynchronous multimodal continuous spatiotemporal occupancy state modeling method for vehicles according to claim 1, characterized in that, The consistency residual increment includes the time residual increment, the spatial residual increment, and the dynamic residual increment; The temporal residual increment is determined by the temporal correlation deviation or temporal alignment deviation between the post-propagation observation and the prior joint state; The spatial residual increment is determined by the reprojection error, voxel alignment bias, or region offset between the post-propagation observation and the prior joint state; The dynamic residual increment is determined by the local occupancy difference, velocity direction difference, flow field deviation, or dynamic continuity difference between the post-propagation observation and the prior joint state.

7. A method for modeling asynchronous multimodal continuous spatiotemporal occupancy states of vehicles according to claim 6, characterized in that, The formula for obtaining the posterior consistent residual state at the current query time is as follows: R(tq) = Γ(R-(tq), ΔR(tq)); Where Γ represents the residual fusion operator, R(tq) is the posterior consistent residual state, R-(tq) is the consistent residual prior state, and ΔR(tq) is the consistent residual increment.

8. A method for modeling asynchronous multimodal continuous spatiotemporal occupancy states of vehicles according to claim 7, characterized in that, Joint updates to prior joint states include: Based on the posterior consistent residual state R(tq) and prior uncertainty state of each mode in each spatial unit, the update weights of different modes are determined. Write the unified query time observation into the occupied state according to the update weight to obtain the occupied probability, idle probability or unknown state of the current query time; By combining radar velocity observation, lidar geometric changes, and image temporal characteristics of dynamic regions, the local motion direction, local flow field, or rate of change of occupation can be updated. When the temporal residual, spatial residual, or dynamic residual of a certain region exceeds a preset threshold, the region is marked as a high mismatch region, and methods such as delayed writing, local reestimation, or outputting unknown states are adopted to suppress erroneous observations from being directly written into the occupied results.

9. A method for modeling asynchronous multimodal continuous spatiotemporal occupancy states of vehicles according to claim 1, characterized in that, Also includes: The vehicle-side partial update step involves determining residual hot zones based on the posterior consistency residual state and uncertainty state; performing local incremental updates on residual hot zones, and performing low-frequency updates, sparse refreshes, or historical state preservation on non-residual hot zones; wherein, dynamic regions adopt a higher update frequency than static regions.

10. An asynchronous multimodal continuous spatiotemporal occupancy state modeling system for vehicles, used to execute the method according to any one of claims 1-9, characterized in that, include: The system includes a camera, lidar, millimeter-wave radar, pose information source, time and pose management unit, preprocessing and unified representation unit, continuous time state modeling unit, unified query time propagation unit, consistent residual estimation unit, joint update unit, result output unit, processor, and memory. The camera, lidar, millimeter-wave radar, and pose information source are electrically connected to the processor, which is electrically connected to the memory. The memory stores program instructions, sensor nominal intrinsic and extrinsic parameters, historical states, query time sequences, and model parameters. After the processor executes the program instructions, it implements the functions of the time and pose management unit, the preprocessing and unified expression unit, the continuous time state modeling unit, the unified query time propagation unit, the consistency residual estimation unit, the joint update unit, and the result output unit.

Images