An autonomous driving occupancy prediction method and system with domain adaptation and bidirectional flow constraints

Abstract

This invention discloses an autonomous driving occupancy prediction method and system based on domain adaptation and bidirectional flow constraints. The method includes: acquiring multi-source environmental perception data of the target object, preprocessing it to generate a voxel occupancy grid and domain identifiers; generating forward and backward flows as supervision labels based on the voxel occupancy grid, and constructing a historical observation sequence based on the voxel occupancy grid and forward flows; inputting the historical observation sequence and domain identifiers into an occupancy-flow joint prediction network model, outputting a predicted occupancy grid, predicted forward flow, and predicted backward flow; training the occupancy-flow joint prediction network model using a joint loss function; calculating a truth-independent closed-loop evaluation index based on the predicted occupancy grid, predicted forward flow, predicted backward flow, and spatial geometric configuration data, and outputting a comprehensive quality score. The method and system provided by this invention can improve the accuracy and generalization performance of occupancy prediction.

Description

Technical Field

[0001] This invention relates to the field of 3D environment perception technology for autonomous driving, and in particular to an autonomous driving occupancy prediction method and system based on domain adaptation and bidirectional flow constraints. Background Technology

[0002] In the field of 3D environmental perception for autonomous driving, occupancy prediction has become the mainstream representation method for depicting the spatiotemporal distribution of obstacles of arbitrary shapes. Current mainstream solutions are divided into four categories: single dataset prediction, voxel flow temporal prediction, multi-source data unified benchmark, and collaborative perception, but many technical shortcomings still exist.

[0003] Existing single-dataset schemes have weak cross-domain transfer capabilities and cannot adapt to mixed training of multi-source heterogeneous data. At the same time, traditional voxel flow schemes independently supervise forward and backward flows, without constructing bidirectional flow closed-loop cyclic constraints. Flow field drift is prone to occur at dynamic and static boundaries and occlusion areas, leading to the accumulation of occupancy prediction errors.

[0004] Furthermore, existing occupancy-flow joint learning suffers from low coupling, achieving only shallow feature overlay and lacking a two-way mutually reinforcing closed-loop optimization mechanism. It also lacks a single ground truth evaluation dimension, failing to cover key dimensions such as reprojection consistency and physical reachability, thus unable to comprehensively characterize prediction accuracy. Simultaneously, existing collaborative and single-vehicle perception pipelines are independent, with fragmented interfaces, high deployment and reuse costs, and difficulty in achieving unified inference switching, hindering the engineering implementation and generalization performance of occupancy prediction technology. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of current autonomous driving 3D environment perception technology, the present invention provides an autonomous driving occupancy prediction method and system with domain adaptation and bidirectional flow constraints, which can improve the accuracy of occupancy prediction and generalization performance.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: An autonomous driving occupancy prediction method with domain adaptation and bidirectional flow constraints, the method comprising: Acquire multi-source environmental perception data of the target object, perform domain-adaptive cross-domain unified occupancy grid protocol processing on it, and generate voxel occupancy grid, spatial geometric configuration data and domain identifier; The forward and backward flows are generated as supervision labels based on the voxel occupancy grid, and a historical observation sequence is constructed based on the voxel occupancy grid and the forward flow. The historical observation sequence and the domain identifier are input into the occupancy-flow joint prediction network model. The network model includes a dual-branch spatiotemporal coding module and a warp-gated cross-attention module. After the data is fused by the dual-branch spatiotemporal coding module and the warp-gated cross-attention module, the predicted occupancy grid, predicted forward flow, and predicted backward flow are output. The occupancy-flow joint prediction network model is trained using a joint loss function based on the forward flow, the backward flow, the predicted forward flow, and the predicted backward flow. Based on the predicted occupied grid, the predicted forward flow, the predicted backward flow, and the spatial geometry configuration data, calculate a truth-independent closed-loop evaluation index and output a comprehensive quality score.

[0007] According to one aspect of the invention, the multi-domain environment perception data includes at least one real data source and at least one simulated data source, and the spatial geometry configuration data includes the vehicle's position and pose.

[0008] According to one aspect of the present invention, the domain adaptive cross-domain unified occupancy grid protocol processing specifically involves performing unified coordinate system, coordinate transformation, resolution resampling, semantic category mapping, visible domain mask generation, and velocity distribution normalization processing on the multi-domain environment perception data.

[0009] According to one aspect of the present invention, the step of generating forward and backward flows as supervision labels based on the voxel occupancy grid specifically comprises: according to the semantic category of each voxel unit in the voxel occupancy grid, marking voxel units belonging to a preset dynamic foreground category and spatially adjacent as dynamic voxel clusters, and marking voxel units in the voxel occupancy grid other than the dynamic voxel clusters as static background voxels; based on the dynamic voxel clusters and the static background voxels at adjacent time points, estimating the rigid body transformation displacement between adjacent frames for the dynamic voxel clusters to generate dynamic flow components, calculating motion compensation displacement for the static background voxels based on the vehicle pose to generate static flow components, and merging the dynamic flow components and the static flow components to generate forward and backward flows.

[0010] According to one aspect of the present invention, the dual-branch spatiotemporal coding module includes an occupation encoder and a stream encoder, which are used to extract the occupation token sequence and the stream token sequence, respectively; the warp-gated cross-attention module fusion specifically involves: calculating cross-attention with the occupation token sequence as the query and the stream token sequence as the key and value, and weighting the result of the cross-attention with the occupation token sequence through gating weights.

[0011] According to one aspect of the invention, the joint loss function includes occupancy segmentation loss, flow regression loss, bidirectional flow cycle consistency loss, occupancy warp consistency loss, temporal consistency loss, and inter-domain alignment loss.

[0012] According to one aspect of the invention, the bidirectional flow loop consistency loss is used to constrain the difference between the closed-loop mapping and the identity mapping formed by the predicted forward flow and the predicted backward flow; the occupancy warp consistency loss is used to constrain the consistency between the predicted occupancy grid after warping by the predicted forward flow and the true occupancy grid in the next time step of the voxel occupancy grid.

[0013] According to one aspect of the present invention, the truth-independent closed-loop evaluation metrics include a multi-view reprojection consistency metric, a semantic reachability metric, a visible domain occlusion consistency metric, and a flow loop residual metric.

[0014] According to one aspect of the present invention, the comprehensive quality score is obtained by adaptively weighting and summarizing the multi-view reprojection consistency index, the semantic reachability index, the visible domain occlusion consistency index, and the flow loop residual index based on the scene dynamic ratio.

[0015] An autonomous driving occupancy prediction system with domain adaptation and bidirectional flow constraints, the system being applied to any of the autonomous driving occupancy prediction methods described above, the system comprising: The data processing module acquires multi-source environmental perception data of the target object, performs domain-adaptive cross-domain unified occupancy grid protocol processing on it, and generates voxel occupancy grid, spatial geometric configuration data and domain identifier; The temporal supervision flow generation module generates forward and backward flows as supervision labels based on the voxel occupancy grid, and constructs a historical observation sequence based on the voxel occupancy grid and the forward flow. The model prediction module inputs the historical observation sequence and the domain identifier into the occupancy-flow joint prediction network model. The network model includes a dual-branch spatiotemporal coding module and a warp-gated cross-attention module. After the data is fused by the warp-gated cross-attention module after feature extraction by the dual-branch spatiotemporal coding module, the predicted occupancy grid, predicted forward flow, and predicted backward flow are output. The occupancy-flow joint prediction network model is trained using a joint loss function based on the forward flow, the backward flow, the predicted forward flow, and the predicted backward flow. The truth-free closed-loop evaluation module calculates a truth-free closed-loop evaluation index based on the predicted occupied grid, the predicted forward flow, the predicted backward flow, and the spatial geometric configuration data, and outputs a comprehensive quality score.

[0016] The advantages of this invention are as follows: By using a domain-adaptive cross-domain unified occupancy grid protocol, it effectively eliminates the data distribution offset problem between the simulation domain and the real domain, significantly improving the generalization ability of the model trained in multi-domain hybrid training. Through an occupancy-flow joint prediction network model, high-dimensional spatiotemporal features are extracted from historical occupancy sequences and voxel flow sequences respectively. An adaptive deep fusion of semantic and motion features is achieved through a warp-gated cross-attention module. Compared to the shallow superposition joint scheme of existing flow encoders, this significantly strengthens the coupling relationship between occupancy semantics and temporal motion.

[0017] The network employs a parallel joint decoding structure to synchronously output the predicted occupancy grid and bidirectional prediction stream. Combined with a multi-component joint loss function, it forms a closed loop of mutually reinforcing optimization between semantic prediction and motion prediction, effectively addressing the shortcomings of existing schemes such as independent supervision of bidirectional streams, easy drift at occlusion boundaries, and insufficient temporal geometric consistency. Furthermore, this invention constructs a multi-dimensional, ground-value-free closed-loop evaluation system that can adaptively assess prediction quality under unlabeled conditions. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of an autonomous driving occupancy prediction method with domain adaptation and bidirectional flow constraints as described in this invention. Figure 2 This is a schematic diagram of the structure of an autonomous driving occupancy prediction system with domain adaptation and bidirectional flow constraints as described in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1

[0022] like Figure 1 As shown, an autonomous driving occupancy prediction method based on domain adaptation and bidirectional flow cycle constraints is presented. The method includes the following specific steps: Step S1: Obtain multi-source environmental perception data of the target object, perform domain-adaptive cross-domain unified occupancy grid protocol processing on it, and generate voxel occupancy grid, spatial geometric configuration data and domain identifier; The execution flow of step S1 is as follows: Step S11: Obtain multi-source environmental perception data of the target object; Multi-domain environmental perception data includes at least one real data source and at least one simulated data source. Specifically, the acquisition method is as follows: the real data source synchronously collects raw point clouds, image detection boxes, and ground truth values ​​of dynamic target velocities from real-world road scenes using vehicle-mounted perception hardware such as multi-line LiDAR, surround-view cameras, and millimeter-wave radar, supplemented by publicly available real-world driving datasets such as nuScenes and Waymo; the simulated data source generates virtual scene point clouds, camera images, and dynamic obstacle motion information through rendering on an autonomous driving simulation platform, constructing the simulated domain perception data. A complete data set from a single data source includes: raw 3D point coordinates... Original occupying tag Dynamic voxel three-dimensional velocity field Calibration parameters of vehicle-mounted surround view camera (internal reference) External reference This includes the field of view of each camera, the vehicle's pose, etc. Simultaneously, domain identifiers are generated, assigning a unique number to each type of data source to distinguish between the simulation domain and different real datasets.

[0023] Step S12: Perform domain-adaptive cross-domain unified occupation grid protocol processing on multi-domain environmental perception data; The domain-adaptive cross-domain unified occupancy grid protocol processing includes unified coordinate system, coordinate transformation, resolution resampling, semantic category mapping, visible domain mask generation, and velocity distribution normalization, specifically: (1) Unified coordinate system definition; Define a unified Ego (Ego Vehicle) coordinate system, with the positive x-axis pointing to the direction of travel of the target vehicle, the positive y-axis pointing to the left side of the target vehicle, and the positive Z-axis pointing to the top side of the target vehicle.

[0024] Uniform voxel mesh size voxel resolution And spatial range. In this embodiment, the voxel grid size is preferably... Horizontal voxel resolution 0.4m, spatial range axis z-axis .

[0025] (2) Coordinate transformation and resolution resampling; Multi-domain environmental perception data The original occupying tag Perform coordinate transformation and resolution resampling:

[0026] in, The transformation matrix; For multi-domain environmental perception data Original 3D point coordinates; For unification Coordinates of a point in three-dimensional space under a coordinate system; To unify the grid index, This is a spatial interpolation function used to obtain the original candidate label values ​​corresponding to the target grid location. Sampling is performed based on the semantic priority of nearest neighbors.

[0027] By using coordinate transformation and resolution resampling, multi-domain environmental perception data is uniformly mapped to the same set of data. A standard grid in a coordinate system enables the fusion and alignment of multi-source sensing data.

[0028] (2) Semantic category mapping; Preset globally unified standard category set The total number of standard categories is fixed at 17, serving as a unified label benchmark. The original semantic categories inherent in the multi-domain environmental perception data are collected to obtain the corresponding source domain category sets. To eliminate the differences in multi-source tagging systems, a dedicated mapping table is constructed for each data source and aligned to the standard category set. For categories with no exact match, map them to the nearest semantic parent class and assign them... The confidence weights and the semantic classification criteria for uniform voxel graticules.

[0029] (4) Generation of the visible field mask; Based on the calibration of all intrinsic and extrinsic parameters of the vehicle-mounted cameras, the center of each voxel in the unified mesh will be determined. Projected onto the image planes of each camera:

[0030] in, , The first The extrinsic and intrinsic parameter matrices of each camera Projecting the voxel center to the first voxel Two-dimensional pixel coordinates of an image.

[0031] judge Whether it falls within the camera's effective field of view, a binary visibility mask is constructed based on this determination result. :

[0032] in, Representing the The effective imaging angle of each camera This is an indicator function; a mask value of 1 indicates that the voxel is visually observed; if the voxel cannot be observed by any camera, the mask value is 0.

[0033] (5) Velocity distribution normalization; Calculate the median of the velocity amplitude of all dynamic voxels across all data sources. Define the normalization factor :

[0034] in, The median flow rate for the nuScenes and Waymo mixed real-world dataset. To minimize the division by zero constant, in this embodiment, .

[0035] Next, using the normalization factor Multi-domain environmental perception data The original flow field Perform a global scaling update to align the velocity distribution scale between the simulation domain and the real domain.

[0036] Finally, the standard voxel grid obtained by coordinate transformation and resampling The voxels occupy the core of the grid, and each 3D voxel in the grid stores a unified standard semantic label. On this basis, dynamic flow velocity features after velocity distribution normalization and camera visibility mask are superimposed. The three are fused to form a complete voxel-occupied grid. The spatial geometric configuration data includes camera intrinsic and extrinsic parameters, vehicle pose, and other data.

[0037] Step S2: Generate forward and backward flows as supervision labels based on the voxel occupancy grid, and construct a historical observation sequence based on the voxel occupancy grid and the forward flow; In step S2, generating forward and backward flows as supervisory labels based on the voxel-occupied grid specifically involves: Based on the semantic category of each voxel unit in the voxel-occupied grid, voxel units belonging to a preset dynamic foreground category and spatially adjacent are marked as dynamic voxel clusters; voxel units in the voxel-occupied grid other than dynamic voxel clusters are marked as static background voxels; based on the dynamic voxel clusters and static background voxels at adjacent time points, rigid body transformation displacements between adjacent frames are estimated for the dynamic voxel clusters to generate dynamic flow components; motion compensation displacements are calculated for the static background voxels based on the vehicle pose to generate static flow components; the dynamic flow components and static flow components are merged to generate forward and backward flows. Specifically: Step S21: Separate dynamic voxels and static voxels; The voxel occupies the grid at the current time frame t. Perform 6-Connected Component Labeling (CCL) to segment the 3D raster into regions, specifically: (1) Perform connected component label calculation: (2) Extract the set of all voxel indices contained in each connected component:

[0038] (3) Preset dynamic foreground category set Traverse all connected components; if a connected component... The semantic category of any voxel belongs to Then the connected component is marked as a dynamic voxel cluster, and all dynamic voxel clusters constitute a set. The remaining voxels are labeled as static background voxels, and the set is denoted as . In this embodiment, Preferred .

[0039] Step S22: For all dynamic voxel clusters within each frame, calculate the inter-frame rigid body displacement cluster by cluster to generate the dynamic flow forward component: (1) For dynamic voxel clusters, if in Frames have matching clusters; the Hungarian algorithm is used to match dynamic clusters in frame t. and Frame dynamic clusters In this embodiment, the matching distance threshold is set differently according to the category: 0.5m is preferred for Car or Cyclist, and 0.2m is preferred for Pedestrian.

[0040] (2) For the successfully matched dynamic clusters, solve for the SE(3) transformation matrix that minimizes the voxel spatial distance error between the two clusters. The solution formula is:

[0041] Where v represents the three-dimensional coordinates of the voxel center within the dynamic cluster in frame t. for The three-dimensional coordinates of the voxel center within the frame matching cluster; the SE(3) transformation includes three-dimensional rotation and translation, which fully describes the overall inter-frame displacement of the obstacle.

[0042] (3) The forward flow per voxel is: for each voxel in the dynamic cluster The three-dimensional dynamic velocity of the voxel can be obtained by using the rigid body transformation difference:

[0043] in, It is a three-dimensional vector, representing the dynamic voxels from frame t to... Spatial displacement of the frame.

[0044] Step S23: Perform motion compensation on the static background voxels; The static background voxel has no self-motion; it only exhibits relative motion within the coordinate system due to the vehicle's movement. Motion compensation is achieved using the vehicle's global pose to generate static flow velocity, specifically: (1) Obtain pose: Read t, The SE (3) pose of the vehicle in the world coordinate system at any time , , for Transformation from one coordinate system to the world coordinate system, inverse matrix Return to world coordinate system Coordinate system.

[0045] (2) Derivation of static background voxel displacement: For static background voxels Calculate t to The relative displacement of the frame due to the movement of the vehicle results in a static flow:

[0046] Step S24: Merge dynamic and static flows, and generate forward and backward flows; (1) Forward flow merging:

[0047] Each valid voxel within the grid is assigned a unique three-dimensional velocity vector. Dynamic voxels store the target's own motion velocity, while static voxels store the vehicle's compensated velocity.

[0048] (2) Backflow generation: Backflow Overall computational logic and forward flow Completely identical, t and Frame matching, first frame Last frame That is, all velocity vectors are set to zero.

[0049] Step S25: Perform bidirectional flow cycle consistency tag verification on the forward and backward flows; To ensure the accuracy of forward and backward flow labels, cyclic residual verification is introduced as a supervisory constraint for subsequent model training. Specifically: (1) Define the voxel warp operator: , which represents the voxel coordinates v superimposed with the flow velocity f to obtain the predicted coordinates for the next frame; (2) Screening effective visual voxel sets Cyclic residuals are calculated only for camera-visible voxels; (3) Formula for calculating cyclic residuals:

[0050] The current voxel is mapped to the next frame along the forward flow, and then mapped back to the original frame along the backward flow of the next frame. The L2 error between the coordinates and the original coordinates is calculated. The smaller the residual, the more accurate the bidirectional flow matching. During the training phase, a cyclic consistency loss is constructed based on this cyclic residual to constrain the alignment of the network's predicted flow field with the real supervised flow labels.

[0051] Step S26: Construct historical observation sequences based on voxel occupancy grid and forward flow; Using consecutive multi-frame time sequences as units, the normalized voxel occupancy mesh Gt and the complete forward flow field are bound frame by frame. Multiple sets of frame data are stacked in chronological order to construct a fixed-length time-series historical observation sequence.

[0052] Step S3: Input the historical observation sequence and the domain identifier into the occupancy-flow joint prediction network model; the network model includes a dual-branch spatiotemporal coding module and a warp-gated cross-attention module. After the data is fused by the dual-branch spatiotemporal coding module and the warp-gated cross-attention module, the predicted occupancy grid, predicted forward flow, and predicted backward flow are output; and the occupancy-flow joint prediction network model is trained using a joint loss function based on the forward flow, the backward flow, the predicted forward flow, and the predicted backward flow. The dual-branch spatiotemporal coding module in step S3 includes an occupation encoder and a stream encoder, used to extract the occupation token sequence and the stream token sequence, respectively. The warp-gated cross-attention module fusion specifically involves: calculating cross-attention using the occupation token sequence as the query and the stream token sequence as the key and value, and then weighting the cross-attention result with the occupation token sequence using gating weights. Specifically: Step S31: Dual-branch spatiotemporal coding; The historical observation sequence and the domain identifier corresponding to the sample are input into two encoders respectively; the occupation encoder outputs the temporal occupation token sequence, and the stream encoder outputs the temporal stream token sequence. Both encoders introduce domain embedding vectors to perceive the differences in data sources and extract cross-domain compatible spatiotemporal features.

[0053] Step S32: Warp-gated cross-attention fusion; Using the occupied token as the query and the stream token sequence as the key and value vector, the scaling dot product cross attention calculation is completed. The occupied token and the stream token are concatenated and activated by sigmoid to generate gating weights in the range of 0 to 1. The gating weights are used to weight the cross attention output and superimpose the original occupied token to obtain the fused spatiotemporal features. The fusion weights of semantic information and temporal motion information are adaptively balanced.

[0054] Step S33: Joint decoding prediction.

[0055] The Transformer decoder is generated spatiotemporally from the fused feature input, and two types of decoder heads are set up in parallel. The occupancy decoder head outputs the predicted voxel occupancy grid for multiple future time steps, while the streaming decoder head synchronously outputs the predicted forward stream and predicted backward stream for the corresponding duration.

[0056] Step S34: Training constraints for the joint loss function.

[0057] The joint loss function in step S3 includes occupancy segmentation loss, flow regression loss, bidirectional flow cycle consistency loss, occupancy warp consistency loss, temporal consistency loss, and inter-domain alignment loss. Specifically, the bidirectional flow cycle consistency loss constrains the difference between the closed-loop mapping formed by the predicted forward flow and the predicted backward flow and the identity mapping; the occupancy warp consistency loss constrains the consistency between the predicted occupancy grid after warping by the predicted forward flow and the ground truth occupancy grid in the next time step of the voxel occupancy grid. Construct a multi-task weighted total loss, simultaneously optimizing all objectives: occupation prediction, streaming prediction, temporal consistency, and cross-domain alignment. The total loss expression is as follows:

[0058] The preferred hyperparameter weights in this embodiment are: , , , , Only camera-visible voxel set The constraint logic for each sub-item of loss, which is involved in the overall loss calculation, is as follows: (1) Occupy segmentation loss (enabled only for labeled training domain); Using weighted multi-class cross-entropy, The category confidence weights assigned to the semantic mapping in step S1 This is a unique hot tag for the voxel truth value category. This is used to predict class probabilities for the network, constraining the semantic classification accuracy between predicted and actual occupied grids. Occupancy segmentation loss. The calculation formula is:

[0059] (2) Flow regression loss; Simultaneously constrain the predicted values ​​of forward and backward flows. With the truth stream generated in step S2 The L2 distance is used to directly supervise the network output, ensuring the predicted forward and backward flows match the ground truth supervision labels. Flow regression loss. The calculation formula is:

[0060] (3) Consistency loss in bidirectional flow cycles; The calculation process is the same as described in step S25.

[0061] (4) Loss of warp consistency; Using trilinear interpolation sampling operator The predicted grid cell of the current frame is warped along the prediction forward flow to the next frame, and compared with the true grid cell of the next time step. Distance constraints ensure that motion flow predictions and occupancy semantic predictions match, preventing semantic category jumps after object movement. Occupancy warp consistency loss. The calculation formula is:

[0062] (5) Temporal consistency loss.

[0063] For static background voxel sets Using truth static stream The static background of the current frame is warped to the next frame to constrain the temporal prediction of static background areas such as roads and buildings to be stable, avoiding false motion predictions in static scenes. The Intersection over Union (IoU) loss is calculated.

[0064] (6) Domain adaptive alignment loss; During model training, the MMD (Maximum Mean Difference Loss) method is used to reduce the voxel feature distribution offset between the simulation domain and different real datasets, achieving cross-domain feature alignment. Combined with domain embedding vectors to weaken domain-specific features, this allows the shared encoder to extract 3D voxel features that are independent of the data source and have stronger generalization capabilities, solving the domain offset problem in simulation-to-real domain transfer and multi-dataset mixed training. In practical applications, MMD can be replaced with CORAL covariance alignment or a domain adversarial discriminator (DANN).

[0065] Step S4: Calculate a truth-independent closed-loop evaluation index based on the predicted occupied grid, the predicted forward flow, the predicted backward flow, and the spatial geometry configuration data, and output the comprehensive quality score.

[0066] The truth-value-independent closed-loop evaluation metrics in step S4 include: multi-view reprojection consistency metric, semantic reachability metric, visible domain occlusion consistency metric, and flow loop residual metric. The comprehensive quality score is obtained by adaptively weighting and summing the multi-view reprojection consistency metric, the semantic reachability metric, the visible domain occlusion consistency metric, and the flow loop residual metric according to the scene's dynamic proportions. Specifically: (1) Calculate the consistency index of multi-view reprojection; Reprojection consistency index The calculation formula is as follows:

[0067] in, Number of vehicle-mounted cameras; To filter the set of valid occupied voxels (i.e., to extract the set of voxels in the predicted occupied grid that are determined to be occupied by objects); For each camera, a semantic category texture / edge response map is provided. These are the two-dimensional pixel coordinates after voxel projection; As a preset threshold, this embodiment preferably uses... ; This is an indicator function used to determine whether the pixel response intensity is higher than a threshold. .

[0068] The index ranges from [0,1]. The closer the value is to 1, the higher the matching degree between the predicted 3D voxel occupancy and the semantic texture of the multi-camera image. For dynamic categories Using the prediction of forward flow Estimating voxel velocity , with the maximum speed of the category (For example, 30 m / s is preferred for Car, and 3 m / s for Pedestrian.) The squared error of a single voxel's velocity exceeding the upper limit is calculated, and then the index is normalized using an exponential decay function to mitigate the negative impact of velocity anomalies. The semantic reachability index is calculated as follows:

[0069] Indicator value range The closer the value is to 1, the more it means that the predicted speed of the dynamic target is in line with the constraints of reality and there is no unreasonable prediction of overspeed.

[0070] (3) Calculate the visible domain occlusion consistency index; Within the camera's visible field of view Inside, calculate t and Frame prediction occupancy crossover ratio :

[0071] in, For the next frame's backflow, To occupy the grid for the next frame, The visible field mask for the current frame. The visible field mask for the next frame. For spatial warp mapping, utilize the backflow of the next frame. The next frame prediction will occupy the grid. Reverse map back to the current frame coordinate system.

[0072] Indicator value range A higher IoU indicates that the occlusion and occupancy states of objects within the visible area of ​​adjacent frames are sequentially stable and without abrupt artifacts.

[0073] (4) Calculate the flow cycle residual index.

[0074] The calculation method is the same as in step S25, with the inputs being the predicted forward flow and the predicted backward flow.

[0075] Range of values ​​for flow cycle residual index The closer the value is to 1, the smaller the closed-loop residual of the bidirectional predicted flow and the better the self-consistency of the flow field prediction.

[0076] (5) Calculate the scene adaptive comprehensive quality score Q.

[0077] The weighted sum of the overall quality scores is:

[0078] in, For reprojection consistency weights, Semantic reachability weights , Visible domain occlusion consistency weight , ; Circular residual with fixed weight .and . For dynamic scaling of the scene, .

[0079] At the same time, set a minimum quality threshold. In this embodiment, the preferred embodiment is... When the calculated comprehensive score If the prediction result of the current frame is deemed to have insufficient credibility, a model degradation strategy or an alarm will be triggered.

[0080] Example 2

[0081] This embodiment, based on the multi-domain occupancy flow joint prediction method in Embodiment 1, adds a unified deployment interface function for single-vehicle collaboration. It is compatible with two operating modes: independent perception reasoning for single vehicles and multi-vehicle collaborative fusion reasoning. Both modes share the same preprocessing, prediction, and evaluation logic, and output standardized and unified results. The specific implementation method is as follows: Define a standardized and unified input tensor package The tensor package contains the vehicle's time-series observation G and the camera's intrinsic and extrinsic parameters. Self-driving position Collaborative slot set With pattern mark The mode flag has two possible values: the first value corresponds to the single-vehicle mode and the second value corresponds to the cooperative mode. When the mode flag is the first value, the cooperative slot is empty, and only the vehicle's own perception and observation data are input. The bidirectional voxel flow construction, occupancy-flow joint prediction and truth-free closed-loop evaluation process are directly executed. When the mode flag is the second value, the coordinated slot is filled with the aligned observation data of multiple neighboring vehicles; the neighboring vehicle-to-own coordinate system transformation matrix is ​​used to uniformly align the grid occupied by the neighboring vehicles to the own vehicle's Ego coordinate system; the neighboring vehicle confidence contribution mask is generated by combining communication delay and positioning covariance; the observation grids of the own vehicle and all neighboring vehicles are fused by the multi-source priority merging operator to obtain the global fused observation sequence, and then the subsequent prediction and evaluation process is executed. Regardless of whether the mode flag takes the first or second value, after the model inference is completed, it outputs the predicted grid occupancy, predicted forward flow, predicted backward flow, and scene comprehensive quality score in a completely uniform format. The results can be parsed without distinguishing between single-vehicle and collaborative scenarios, realizing a single model that is compatible with both single-vehicle and vehicle-to-vehicle collaborative autonomous driving deployment scenarios.

[0082] The advantages of this embodiment are: by cooperating with the unified deployment interface of single-vehicle collaboration, flexible inference of single model dual mode can be realized, effectively reducing engineering deployment costs, and taking into account prediction accuracy, robustness and practicality.

[0083] Example 3

[0084] like Figure 2 As shown, an autonomous driving occupancy prediction system based on domain adaptation and bidirectional flow constraints is described. The system is applied in Embodiment 1 and includes: Data processing module M1 acquires multi-source environmental perception data of the target object, performs domain-adaptive cross-domain unified occupancy grid protocol processing on it, and generates voxel occupancy grid, spatial geometric configuration data and domain identifier; The temporal supervision flow generation module M2 generates forward and backward flows as supervision labels based on the voxel occupancy grid, and constructs a historical observation sequence based on the voxel occupancy grid and the forward flow. The model prediction module M3 inputs the historical observation sequence and the domain identifier into the occupancy-flow joint prediction network model. The network model includes a dual-branch spatiotemporal coding module and a warp-gated cross-attention module. After the data is fused by the dual-branch spatiotemporal coding module and the warp-gated cross-attention module, the predicted occupancy grid, predicted forward flow, and predicted backward flow are output. The occupancy-flow joint prediction network model is trained using a joint loss function based on the forward flow, the backward flow, the predicted forward flow, and the predicted backward flow. The truth-free closed-loop evaluation module M4 calculates a truth-free closed-loop evaluation index based on the predicted occupied grid, the predicted forward flow, the predicted backward flow, and the spatial geometric configuration data, and outputs a comprehensive quality score.

[0085] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0086] Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for predicting autonomous driving occupancy based on domain adaptation and bidirectional flow constraints, characterized in that, The method includes: Acquire multi-source environmental perception data of the target object, perform domain-adaptive cross-domain unified occupancy grid protocol processing on it, and generate voxel occupancy grid, spatial geometric configuration data and domain identifier; Based on the voxel occupancy grid, forward and backward flows are generated as supervision labels, and historical observation sequences are constructed based on the voxel occupancy grid and the forward flow; The historical observation sequence and the domain identifier are input into the occupancy-flow joint prediction network model. The network model includes a dual-branch spatiotemporal coding module and a warp-gated cross-attention module. After the data is fused by the dual-branch spatiotemporal coding module and the warp-gated cross-attention module, the predicted occupancy grid, predicted forward flow, and predicted backward flow are output. The occupancy-flow joint prediction network model is trained using a joint loss function based on the forward flow, the backward flow, the predicted forward flow, and the predicted backward flow. Based on the predicted occupied grid, the predicted forward flow, the predicted backward flow, and the spatial geometry configuration data, calculate a truth-independent closed-loop evaluation index and output a comprehensive quality score.

2. The autonomous driving occupancy prediction method according to claim 1, characterized in that, The multi-domain environment perception data includes at least one real data source and at least one simulated data source, and the spatial geometry configuration data includes the vehicle's position and pose.

3. The autonomous driving occupancy prediction method according to claim 2, characterized in that, The domain adaptive cross-domain unified occupancy grid protocol processing specifically involves performing unified coordinate system, coordinate transformation, resolution resampling, semantic category mapping, visible domain mask generation, and velocity distribution normalization processing on the multi-domain environment perception data.

4. The autonomous driving occupancy prediction method according to claim 3, characterized in that, The process of generating forward and backward flows as supervisory labels based on the voxel-occupied grid specifically involves: according to the semantic category of each voxel unit in the voxel-occupied grid, voxel units belonging to a preset dynamic foreground category and spatially adjacent to each other are marked as dynamic voxel clusters, and voxel units in the voxel-occupied grid other than the dynamic voxel clusters are marked as static background voxels; based on the dynamic voxel clusters and the static background voxels at adjacent time points, the rigid body transformation displacement between adjacent frames is estimated for the dynamic voxel clusters to generate dynamic flow components, and the motion compensation displacement of the static background voxels is calculated based on the vehicle pose to generate static flow components; the dynamic flow components and the static flow components are merged to generate forward and backward flows.

5. The autonomous driving occupancy prediction method according to claim 1, characterized in that, The dual-branch spatiotemporal coding module includes an occupation encoder and a stream encoder, which are used to extract the occupation token sequence and the stream token sequence, respectively. The warp-gated cross-attention module fusion is specifically as follows: cross-attention is calculated using the occupation token sequence as the query and the stream token sequence as the key and value, and the result of the cross-attention is weighted and fused with the occupation token sequence through gating weights.

6. The autonomous driving occupancy prediction method according to claim 1, characterized in that, The joint loss function includes occupation partitioning loss, flow regression loss, bidirectional flow cycle consistency loss, occupation warp consistency loss, temporal consistency loss, and inter-domain alignment loss.

7. The autonomous driving occupancy prediction method according to claim 6, characterized in that, The bidirectional flow cycle consistency loss is used to constrain the difference between the closed-loop mapping formed by the predicted forward flow and the predicted backward flow and the identity mapping; the occupancy warp consistency loss is used to constrain the consistency between the predicted occupancy grid after warping by the predicted forward flow and the true occupancy grid in the next time step of the voxel occupancy grid.

8. The autonomous driving occupancy prediction method according to claim 1, characterized in that, The closed-loop evaluation metrics without truth value dependence include the multi-view reprojection consistency metric, the semantic reachability metric, the visible domain occlusion consistency metric, and the flow loop residual metric.

9. The autonomous driving occupancy prediction method according to claim 8, characterized in that, The overall quality score is obtained by adaptively weighting and summarizing the multi-view reprojection consistency index, the semantic reachability index, the visible domain occlusion consistency index, and the flow loop residual index according to the scene dynamic ratio.

10. An autonomous driving occupancy prediction system with domain adaptation and bidirectional flow constraints, characterized in that, The system is applied to the autonomous driving occupancy prediction method according to any one of claims 1 to 9, the system comprising: The data processing module acquires multi-source environmental perception data of the target object, performs domain-adaptive cross-domain unified occupancy grid protocol processing on it, and generates voxel occupancy grid, spatial geometric configuration data and domain identifier; The temporal supervision flow generation module generates forward and backward flows as supervision labels based on the voxel occupancy grid, and constructs a historical observation sequence based on the voxel occupancy grid and the forward flow. The model prediction module inputs the historical observation sequence and the domain identifier into the occupancy-flow joint prediction network model. The network model includes a dual-branch spatiotemporal coding module and a warp-gated cross-attention module. After the data is fused by the warp-gated cross-attention module after feature extraction by the dual-branch spatiotemporal coding module, the predicted occupancy grid, predicted forward flow, and predicted backward flow are output. The occupancy-flow joint prediction network model is trained using a joint loss function based on the forward flow, the backward flow, the predicted forward flow, and the predicted backward flow. The truth-free closed-loop evaluation module calculates a truth-free closed-loop evaluation index based on the predicted occupied grid, the predicted forward flow, the predicted backward flow, and the spatial geometric configuration data, and outputs a comprehensive quality score.

Images