An aerial layout condition-based diffusion-based time sequence semantic occupancy prediction method

Abstract

The present application belongs to the technical field of automatic driving three-dimensional environment perception, and specifically relates to a time sequence semantic occupation prediction method based on bird's-eye layout condition diffusion. The method takes continuous multiple frames of complete bird's-eye layout as input, and takes an Occ3D data set semantic occupation grid as supervision. A variational autoencoder is first trained to compress three-dimensional semantic occupation into low-dimensional hidden variables, and then a time sequence diffusion transformer is trained under the condition of complete bird's-eye layout to learn the mapping from the bird's-eye layout to the semantic occupation hidden variable. In the training process, the reference frame structure prior, hidden variable teacher loss, sensor reliability weighting, rare category enhancement and parameter exponential moving average are used to improve the generation quality. In the inference stage, only the bird's-eye layout needs to be input, and the three-dimensional semantic occupation grid is generated by decoding after random Gaussian hidden variable sampling through reverse diffusion. The present application can realize accurate prediction of the semantic occupation grid with the bird's-eye layout as input, and is used for supporting multi-modal scene generation and closed-loop simulation tasks of the automatic driving vehicle.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving three-dimensional environment perception technology, and specifically relates to a temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion. Background Technology

[0002] Three-dimensional semantic occupancy meshes can describe vehicles, pedestrians, roads, buildings, vegetation, and free space within a unified voxel space, serving as an important representation for autonomous driving environment understanding, scene generation, and closed-loop simulation. Accurately predicting semantic occupancy based on a bird's-eye view layout is of great value for generating multimodal scenes in autonomous driving. This process requires simultaneously restoring the consistency of road structure, traffic participant positions, vertical geometry, and consecutive time intervals.

[0003] Existing methods train generative models directly in the original voxel space, resulting in high computational costs, a large proportion of free space, and an unbalanced class distribution, leading to unsatisfactory training efficiency and generation quality. Compressing multi-channel bird's-eye view layouts into single-channel conditions results in the loss of structural information such as lanes, boundaries, pedestrian crossings, parking areas, and traffic participants, causing road misalignment, blurred boundaries, and missing small targets. Conventional diffusion training often only supervises noise prediction, lacking direct constraints on the final latent variables. Furthermore, uniform sample weights can bias the model towards large-area categories such as roads, buildings, and vegetation. When consecutive frames lack structurally stable priors, static roads and buildings may exhibit temporal jumps. Therefore, it is necessary to combine complete bird's-eye view conditions, latent variable supervision, rare class enhancement, and joint modeling of consecutive frames. Summary of the Invention

[0004] To address the above problems, this invention proposes a temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion, comprising the following steps:

[0005] Step 1: Select consecutive keyframes from the same nuScenes scene, read the complete multi-channel bird's-eye view layout and semantic occupancy annotations from the Occ3D dataset, and construct a supervised dataset; unify the coordinate direction of the bird's-eye view layout and semantic occupancy by fixing rotation and flipping, and divide the training set and validation set according to the scene;

[0006] Step 2: Compress the three-dimensional semantic occupancy into continuous low-resolution latent variables using a two-dimensional encoder, recover the voxel semantics using a three-dimensional decoder, train a variational autoencoder, jointly optimize the class-weighted cross-entropy loss, Lovász-Softmax loss and relative entropy loss, and determine the upper limit of latent variable expression based on the reconstruction performance on the validation set.

[0007] Step 3: Freeze the variational autoencoder, encode the ground truth of the Occ3D dataset and scale it into latent variables, add noise according to the noise schedule, form conditional features with complete bird's-eye view layout in multiple consecutive frames, construct a temporal latent variable diffusion transformer, and enable the temporal diffusion transformer to jointly learn the spatial structure and short temporal correlation.

[0008] Step 4: In addition to the noise prediction loss, the latent variables are directly supervised by the latent variable teacher loss. Training weights are generated using the LiDAR observation density and image structure clarity. The sample weights are increased according to the occurrence of rare categories, and the parameter exponential moving average copy is maintained to train the prediction network.

[0009] Step 5: Denoise gradually using standard Gaussian random latent variables. Turn off truth reference latent variables and truth density conditions during bird's-eye view layout inference. Perform backdiffusion sampling and semantic occupancy decoding under the condition of using only bird's-eye view layout. Recover and save the three-dimensional semantic occupancy of multiple consecutive frames by freezing the decoder.

[0010] Furthermore, in step one, the supervisory dataset is constructed as follows:

[0011] Select consecutive keyframes from the same nuScenes scene, assuming each training sample contains consecutive keyframes. The first keyframe, the The bird's-eye view layout of each sample sequence and the semantic occupancy ground truth of the Occ3D dataset are denoted as follows: and Construct a supervised dataset:

[0012]

[0013] In the formula, For monitoring datasets; The total number of sample sequences; For sequence indexing; The number of consecutive keyframes contained in each sequence; For frame time index; For the first A sequence of consecutive A collection of frame-by-frame bird's-eye view layouts; For the first A sequence of consecutive Frame semantics occupies the truth set;

[0014] Furthermore, in step one, the method for unifying the bird's-eye view layout and semantic occupancy coordinate direction is as follows:

[0015] Perform a fixed 90-degree rotation and a specified axis flip on the original bird's-eye view layout to obtain a layout aligned with the semantic occupancy grid of the Occ3D dataset:

[0016]

[0017] In the formula, For the first The sequence number is... The original bird's-eye view layout without coordinate transformation; For an operator that rotates 90 degrees counterclockwise in the bird's-eye view plane; For the bird's-eye view plane Operators that perform mirror flipping along the axis; To complete the aligned bird's-eye view layout after rotation and flipping; symbols Unless otherwise specified, Unified as .

[0018] Furthermore, step two includes: establishing a semantic occupancy variational coding distribution and performing reparameter sampling, as follows:

[0019] The first The sequence number is... The frame semantics occupies the truth input encoder, outputs the latent variable distribution parameters, obtains continuous latent variables through reparameterization, and the decoder recovers the class probabilities, as shown in the following formula:

[0020]

[0021]

[0022] In the formula, The parameter is Variational encoder; For the first The sequence number is... The Occ3D semantics of a frame occupy truth value; The mean tensor of the Gaussian distribution of latent variables; For the latent variable log-variance tensor, i.e. ; It is the element-wise positive standard deviation tensor; It is an element-wise exponential function; Semantic occupancy is a continuous latent variable; To and A standard Gaussian random tensor of the same size; This represents a tensor with zero mean. Covariance is the identity matrix The standard Gaussian distribution; This is element-wise multiplication; The parameter is A three-dimensional voxel decoder; Let be the class probability tensor.

[0023] Furthermore, step two also includes: jointly optimizing semantic reconstruction, region intersection and merging, and latent variable distribution, as follows:

[0024] Voxel-level semantic reconstruction using category-weighted cross-entropy constraints:

[0025]

[0026] In the formula, Category-weighted cross-entropy loss; The total number of sample sequences; Number of consecutive frames; , and These represent the vertical dimension, horizontal dimension, and number of height layers of the voxel mesh. For sequence indexing; For frame time index; , and These are the vertical, horizontal, and height indices of the voxel, respectively; For true value In voxels Category number at the location; The reconstruction weight corresponding to this true value category; This represents the predicted probability of the ground truth class given by the decoder at the corresponding voxel position; It is the natural logarithm function;

[0027] Using relative entropy to constrain the distribution of latent variables to approximate a standard Gaussian distribution:

[0028]

[0029] In the formula, The relative entropy regularization loss is the sum of the latent variable distribution and the standard Gaussian distribution. The index of the element after the hidden variable tensor is flattened; The number of hidden channels; and For implicit variable plane dimensions; For mean tensor In the The value at each element; For the log-variance tensor In the The value at each element; The corresponding variance; a constant. Unit variance term from a standard Gaussian distribution;

[0030] The overall training objective of the variational autoencoder is as follows:

[0031]

[0032] In the formula, This represents the total training loss of the variational autoencoder. Let be the set of all predicted probability tensors and the complete set of semantic occupancy truth values Calculated Lovász-Softmax region loss; , and These are the weights of the corresponding loss terms;

[0033] Freeze encoder parameters after training is complete and decoder parameters The upper limit of latent variable expression is determined by the reconstruction performance of the validation set.

[0034] Furthermore, step three includes: freezing the variational autoencoder, scaling the semantic latent variables, and performing forward noise addition, as follows:

[0035] The latent variables output by the frozen variational autoencoder are scaled by a fixed coefficient and then adjusted according to the diffusion time. Perform positive noise addition:

[0036]

[0037]

[0038]

[0039] In the formula, For the first The sequence number is... Latent variables in frame diffusion; The scaling factor is the implicit variable. Semantic occupancy is a continuous latent variable; For diffusion time index, The maximum number of steps for diffusion; For diffusion time Noisy latent variables; For diffusion time The cumulative signal holding coefficient; For the first Single-step signal preservation coefficient for each diffusion step; The diffusion step index used when calculating the cumulative product; Indicates and Independent standard Gaussian noise of the same size; , and Let represent the standard Gaussian distribution, the zero-mean tensor, and the unit covariance matrix, respectively; For sequence index, For frame time index.

[0040] Furthermore, step three also includes: forming temporal condition features using the complete multi-channel bird's-eye view layout, as follows:

[0041] The first Continuous sequences All frames A bird's-eye view layout, channel, and time location coding. The set of input parameters is Condition encoder , obtained the Temporal bird's-eye view of the conditional features of each sequence ; The number of passageways is determined by an aerial view;

[0042] Step three further includes: predicting noise using a time-series diffusion transformer, as follows:

[0043] By inputting the noisy latent variables, diffusion time, and temporal bird's-eye view conditions into the prediction network, the noise estimate is obtained:

[0044]

[0045] In the formula, For the first The sequence at the diffusion time The predicted noise sequence; The parameter set is A time-series diffusion converter; It is a sequence of noisy latent variables in consecutive frames. ; For the first Optional low-dimensional scene structure conditions for each sequence.

[0046] Furthermore, step four includes the following steps:

[0047] The method for constructing a priori reference frame structure and restricting its usage is as follows:

[0048] During the training phase, the latent variables of the sequence reference frame are copied to the time dimension, and random noise is added with small weights to obtain the structural prior noise:

[0049]

[0050]

[0051] In the formula, The training noise sequence after incorporating the prior knowledge of the reference frame structure; The original Gaussian noise sequence, ; These are the prior coefficients for the reference frame structure; To copy the latent variables of a single frame to The time extension operator at each moment; For the first Diffusion latent variables of each sequence reference frame; For reference frame time index; This is a training sequence of noisy latent variables reconstructed using a reference prior; This is the cumulative signal hold factor; For a continuous frame spread latent variable sequence, ; For sequence index, Index for diffusion time;

[0052] The inverse outcome is directly constrained through the latent variable teacher loss, as follows:

[0053] The diffusion time is deduced from the predicted noise. Corresponding latent variable estimation:

[0054]

[0055] In the formula, To be based on diffusion time The first noisy latent variable is obtained by back-reasoning Estimation of latent variables in a sequence; For the first The sequence at the diffusion time A noisy sequence of latent variables; To predict noisy sequences; This is the cumulative signal hold factor; For sequence indexing; Index for diffusion time;

[0056] The average noise prediction loss and the average latent variable teacher loss are as follows:

[0057]

[0058]

[0059] In the formula, For the first Average noise prediction loss for each sequence; This is the original Gaussian noise sequence added during forward diffusion; For the first The average latent variable teacher loss for each sequence; Represents the sum of squares of all elements; Number of consecutive frames; The number of hidden channels; and For implicit variable plane dimensions.

[0060] Furthermore, step four also includes:

[0061] Sample weights are set based on sensor reliability and rare categories during the training period, as follows:

[0062] The training reliability score is derived from the LiDAR observation density and image structure sharpness.

[0063]

[0064] In the formula, For the first Training reliability score of each sequence; The fusion weights are the reliability components of the lidar. The fusion weights for the image reliability components; For the first The effective point density of the lidar corresponding to each sequence; This is the normalization constant for the point density of the lidar; For the first Texture intensity of a sequence of front view images; This is the image structure intensity normalization constant; To truncate the input scalar to a closed interval The function; This serves as the source identifier for the lidar. Image source identification; images and point clouds are used only for calculating training weights and are not input into the prediction network. ;

[0065] Class augmentation coefficients are constructed based on whether rare classes appear in the semantic occupancy ground truth values ​​of the Occ3D dataset:

[0066]

[0067] In the formula, For the first Rare class enhancement coefficients for each sequence; Increase the magnitude for rare categories; This is an indicator function; it takes the value 1 if the condition inside the parentheses is true, and 0 otherwise. This indicates that at least one voxel satisfies the condition; For frame time index, , and These are the vertical, horizontal, and height indices for voxels, respectively. The truth class number for the corresponding voxel; A pre-specified set of rare categories; For the entire set of semantic categories; This indicates that the former is a subset of the latter;

[0068] Based on comprehensive noise prediction, latent variable teacher supervision, training reliability, and rare class reinforcement, the predicted total network loss is as follows:

[0069]

[0070] In the formula, The total training loss is used to predict the temporal diffusion prediction network. The total number of sample sequences; For sequence indexing; Weighted magnitude for training reliability; Predict loss for average noise; The teacher loss weight is a latent variable. The average latent variable is the teacher loss; , , and Each element controls the reference frame prior, rare class enhancement, training reliability weighting, and teacher supervision, with their meanings being distinct.

[0071] The method for maintaining an exponential moving average replica of the predicted network parameters is as follows:

[0072] Suppose the prediction network completes the first... After the secondary parameter update, its instantaneous parameters and exponential moving average parameters are updated as follows:

[0073]

[0074] In the formula, For the first The updated exponential moving average parameters; The parameters are the exponential moving average parameters from the previous update; For the first The instantaneous predicted network parameters are obtained through subgradient update; , The exponential moving average decay coefficient; The updated weights for the current real-time parameters; Update the index for the parameter; This represents the total number of parameter updates during the training phase.

[0075] Furthermore, in step five, the method for performing backdiffusion sampling based solely on the bird's-eye view layout is as follows:

[0076] During inference, we start with standard Gaussian random latent variables and use an exponential moving average parameter network to predict noise and update the latent variables at each backdiffusion time step.

[0077]

[0078]

[0079] In the formula, For the first The sequence at the maximum diffusion time The initial random latent variables; Indicates a standard Gaussian distribution; It is a zero-mean tensor; For unit covariance matrix; For the diffusion time Updated to the previous moment The subsequent sequence of latent variables; This represents the cumulative signal retention coefficient at the previous diffusion moment; For estimating latent variables; The noisy sequence is predicted using an exponential moving average parameter network. For the first The random strength of each backward sampling step, and satisfying ; Add noise to the reverse sampling; For sequence indexing; The maximum number of steps for diffusion; when Deterministic sampling is formed when Random sampling is generated at that time.

[0080] Furthermore, in step five, the method for decoding and saving the three-dimensional semantic occupancy prediction is as follows:

[0081] Divide the final latent variable obtained from backsampling by the scaling factor and input it into the frozen decoder to select the class with the highest probability for each voxel:

[0082]

[0083] In the formula, For the first The sequence number is... The predicted semantics of a frame occupy a grid. The target voxel location, where , and These are indexes for vertical, horizontal, and height respectively; An operator that returns the category number corresponding to the highest probability; Index for candidate semantic categories; A set of semantic categories; The parameter set is Frozen 3D voxel decoder; After backsampling is completed, the first Latent variables in frame diffusion; The scaling factor is the implicit variable; the square brackets are the subscripts. Indicates the type of reading and voxel location The corresponding predicted probability; For sequence index, It serves as a frame time index; the output mesh size is consistent with the Occ3D semantic occupancy truth value, and is saved according to the scene name and sample identifier.

[0084] The beneficial effects of this invention are:

[0085] This invention's method takes a complete bird's-eye view layout from multiple consecutive frames as input and uses the semantic occupancy grid of the Occ3D dataset as supervision. It first trains a variational autoencoder to establish a low-dimensional latent variable space, then trains a temporal diffusion transformer under complete bird's-eye view conditions. Furthermore, it improves generation quality by training priors using reference frames, latent variable teacher loss, sensor reliability weighting, rare class enhancement, and exponential moving average of parameters. Only the bird's-eye view layout is used during the inference phase. This invention enables accurate prediction of the semantic occupancy grid using a bird's-eye view layout as input, supporting multimodal scene generation and closed-loop simulation tasks for autonomous vehicles, and accelerating their industrialization. Attached Figure Description

[0086] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0087] Figure 2 This is a schematic diagram of the variational autoencoder process according to an embodiment of the present invention;

[0088] Figure 3 This is a comparison chart of the 3D semantic occupancy grid prediction results and the actual results for scenario 1 in the example.

[0089] Figure 4 This is a comparison chart of the 3D semantic occupancy grid prediction results and the actual results for scenario 2 in the example.

[0090] Wherein, (a) is the structured BEV information, (b) is the semantic occupancy prediction result of the present invention, and (c) is the actual semantic occupancy result of Occ3D. Detailed Implementation

[0091] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0092] Example 1

[0093] like Figure 1-2 As shown, this embodiment provides a temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion, which includes the following steps:

[0094] Step 1: Construct continuous frame samples supervised by the Occ3D dataset and unify the bird's-eye view layout and semantic occupancy coordinate orientation:

[0095] Select consecutive keyframes from the same nuScenes scene, read the complete multi-channel bird's-eye view layout and semantic occupancy annotations of the Occ3D dataset, and construct a supervised dataset; unify the coordinate direction by fixing rotation and flipping, and divide the training set and validation set according to the scene;

[0096] Furthermore, in this embodiment, the specific method of step one is as follows:

[0097] 11) Construct continuous frame supervision samples;

[0098] Select consecutive keyframes from the same nuScenes scene, assuming each training sample contains consecutive keyframes. The first keyframe, the The bird's-eye view layout of each sample sequence and the semantic occupancy ground truth of the Occ3D dataset are denoted as follows: and Construct a supervised dataset:

[0099]

[0100] In the formula, For monitoring datasets; The total number of sample sequences; For sequence indexing; The number of consecutive keyframes contained in each sequence; For frame time index; For the first A sequence of consecutive A collection of frame-by-frame bird's-eye view layouts; For the first The sequence number is... A bird's-eye view layout of the frame; The number of passageways is determined by an aerial view; and These represent the vertical and horizontal grid numbers of the bird's-eye view plan, respectively. For the first A sequence of consecutive Frame semantics occupies the truth set; For the first The sequence number is... The semantic occupancy truth value of frames in the Occ3D dataset; The number of voxel height layers; A set of semantic categories; This represents the total number of semantic categories.

[0101] 12) Unify the bird's-eye view layout and semantic occupancy coordinate direction;

[0102] Perform a fixed 90-degree rotation and a specified axis flip on the original bird's-eye view layout to obtain a layout aligned with the semantic occupancy grid of the Occ3D dataset:

[0103]

[0104] In the formula, For the first The sequence number is... The original bird's-eye view layout without coordinate transformation; For an operator that rotates 90 degrees counterclockwise in the bird's-eye view plane; For the bird's-eye view plane Operators that perform mirror flipping along the axis; To complete the aligned bird's-eye view layout after rotation and flipping; symbols Unless otherwise specified, Unified as .

[0105] 13) Divide the training set and validation set according to the scenario.

[0106] Step 2: Train a variational autoencoder using the semantic occupancy ground truth of the Occ3D dataset and determine the upper bound of latent variable representation:

[0107] The three-dimensional semantic occupancy is compressed into continuous low-resolution latent variables using a two-dimensional encoder, and the voxel semantics are recovered by a three-dimensional decoder. The class-weighted cross-entropy loss, Lovász-Softmax loss and relative entropy loss are jointly optimized, and the upper limit of the latent variable expression is determined by the reconstruction performance on the validation set.

[0108] Furthermore, in this embodiment, the specific method for step two is as follows:

[0109] 21) Establish a semantic occupancy variational coding distribution and perform reparameter sampling;

[0110] The first The sequence number is... The frame semantics occupies the truth input encoder, outputs the latent variable distribution parameters, obtains continuous latent variables through reparameterization, and the decoder recovers the class probabilities, as shown in the following formula:

[0111]

[0112]

[0113] In the formula, The parameter is Variational encoder; For the first The sequence number is... The Occ3D semantics of a frame occupy truth value; The mean tensor of the Gaussian distribution of latent variables; For the latent variable log-variance tensor, i.e. ; It is the element-wise positive standard deviation tensor; It is an element-wise exponential function; Semantic occupancy is a continuous latent variable; To and A standard Gaussian random tensor of the same size; This represents a tensor with zero mean. Covariance is the identity matrix The standard Gaussian distribution; This is element-wise multiplication; The parameter is A three-dimensional voxel decoder; Let be the class probability tensor.

[0114] 22) Jointly optimize semantic reconstruction, region intersection and merging, and latent variable distribution;

[0115] Voxel-level semantic reconstruction using category-weighted cross-entropy constraints:

[0116]

[0117] In the formula, Category-weighted cross-entropy loss; The total number of sample sequences; Number of consecutive frames; , and These represent the vertical dimension, horizontal dimension, and number of height layers of the voxel mesh. For sequence indexing; For frame time index; , and These are the vertical, horizontal, and height indices of the voxel, respectively; For true value In voxels Category number at the location; The reconstruction weight corresponding to this true value category; This represents the predicted probability of the ground truth class given by the decoder at the corresponding voxel position; It is the natural logarithm function;

[0118] Using relative entropy to constrain the distribution of latent variables to approximate a standard Gaussian distribution:

[0119]

[0120] In the formula, The relative entropy regularization loss is the sum of the latent variable distribution and the standard Gaussian distribution. The index of the element after the hidden variable tensor is flattened; The number of hidden channels; and For implicit variable plane dimensions; For mean tensor In the The value at each element; For the log-variance tensor In the The value at each element; The corresponding variance; a constant. Unit variance term from a standard Gaussian distribution;

[0121] The overall training objective of the variational autoencoder is as follows:

[0122]

[0123] In the formula, This represents the total training loss of the variational autoencoder. Let be the set of all predicted probability tensors and the complete set of semantic occupancy truth values Calculated Lovász-Softmax region loss; , and These are the weights of the corresponding loss terms;

[0124] Freeze encoder parameters after training is complete and decoder parameters The upper limit of latent variable expression is determined by the reconstruction performance of the validation set.

[0125] Step 3: Construct a temporal latent variable diffusion transformer for complete multi-channel bird's-eye view layout conditions:

[0126] Freeze the variational autoencoder, encode the ground values ​​of the Occ3D dataset and scale them into latent variables, add noise according to the noise schedule, form conditional features with a complete bird's-eye view layout in multiple consecutive frames, and enable the temporal diffusion transformer to jointly learn the spatial structure and short temporal correlation.

[0127] Furthermore, in this embodiment, the specific method for step three is as follows:

[0128] 31) Freeze the variational autoencoder, scale the semantic latent variables and perform positive noise addition;

[0129] The latent variables output by the frozen variational autoencoder are scaled by a fixed coefficient and then adjusted according to the diffusion time. Perform positive noise addition:

[0130]

[0131]

[0132]

[0133] In the formula, For the first The sequence number is... Latent variables in frame diffusion; The scaling factor is the implicit variable. Semantic occupancy is a continuous latent variable; For diffusion time index, The maximum number of steps for diffusion; For diffusion time Noisy latent variables; For diffusion time The cumulative signal holding coefficient; For the first Single-step signal preservation coefficient for each diffusion step; The diffusion step index used when calculating the cumulative product; Indicates and Independent standard Gaussian noise of the same size; , and Let represent the standard Gaussian distribution, the zero-mean tensor, and the unit covariance matrix, respectively; For sequence index, For frame time index;

[0134] 32) Utilize the complete multi-channel bird's-eye view layout to form temporal condition characteristics;

[0135] The first Continuous sequences All frames A bird's-eye view layout, channel, and time location coding. The set of input parameters is Condition encoder , obtained the Temporal bird's-eye view of the conditional features of each sequence ; The number of passageways is determined by an aerial view;

[0136] 33) Use a time-series diffusion converter to predict noise;

[0137] By inputting the noisy latent variables, diffusion time, and temporal bird's-eye view conditions into the prediction network, the noise estimate is obtained:

[0138]

[0139] In the formula, For the first The sequence at the diffusion time The predicted noise sequence; The parameter set is A time-series diffusion converter; It is a sequence of noisy latent variables in consecutive frames. ; For the first Optional low-dimensional scene structure conditions for each sequence.

[0140] Step 4: Train the prediction network using latent variable teacher supervision, multi-source reliability weighting, rare class enhancement, and parametric exponential moving average:

[0141] In addition to the noise prediction loss, the latent variables are directly supervised by the latent variable teacher loss. Training weights are generated using LiDAR observation density and image structural clarity. The sample weights are increased according to the occurrence of rare categories, and the parametric exponential moving average copy is maintained.

[0142] Furthermore, in this embodiment, the specific method for step four is as follows:

[0143] 41) Construct a priori reference frame structure and restrict its scope of use;

[0144] During the training phase, the latent variables of the sequence reference frame are copied to the time dimension, and random noise is added with small weights to obtain the structural prior noise:

[0145]

[0146]

[0147] In the formula, The training noise sequence after incorporating the prior knowledge of the reference frame structure; The original Gaussian noise sequence, ; These are the prior coefficients for the reference frame structure; To copy the latent variables of a single frame to The time extension operator at each moment; For the first Diffusion latent variables of each sequence reference frame; For reference frame time index; This is a training sequence of noisy latent variables reconstructed using a reference prior; This is the cumulative signal hold factor; For a continuous frame spread latent variable sequence, ; For sequence index, Index for diffusion time.

[0148] 42) Directly constrain the inverse outcome through the latent variable teacher loss;

[0149] The diffusion time is deduced from the predicted noise. Corresponding latent variable estimation:

[0150]

[0151] In the formula, To be based on diffusion time The first noisy latent variable is obtained by back-reasoning Estimation of latent variables in a sequence; For the first The sequence at the diffusion time A noisy sequence of latent variables; To predict noisy sequences; This is the cumulative signal hold factor; For sequence indexing; Index for diffusion time;

[0152] The average noise prediction loss and the average latent variable teacher loss are as follows:

[0153]

[0154]

[0155] In the formula, For the first Average noise prediction loss for each sequence; This is the original Gaussian noise sequence added during forward diffusion; For the first The average latent variable teacher loss for each sequence; Represents the sum of squares of all elements; Number of consecutive frames; The number of hidden channels; and For implicit variable plane dimensions;

[0156] 43) Set sample weights based on sensor reliability and rare categories during the training period;

[0157] The training reliability score is derived from the LiDAR observation density and image structure sharpness.

[0158]

[0159] In the formula, For the first Training reliability score of each sequence; The fusion weights are the reliability components of the lidar. The fusion weights for the image reliability components; For the first The effective point density of the lidar corresponding to each sequence; This is the normalization constant for the point density of the lidar; For the first Texture intensity of a sequence of front view images; This is the image structure intensity normalization constant; To truncate the input scalar to a closed interval The function; This serves as the source identifier for the lidar. Image source identification; images and point clouds are used only for calculating training weights and are not input into the prediction network. ;

[0160] Class augmentation coefficients are constructed based on whether rare classes appear in the semantic occupancy ground truth values ​​of the Occ3D dataset:

[0161]

[0162] In the formula, For the first Rare class enhancement coefficients for each sequence; Increase the magnitude for rare categories; This is an indicator function; it takes the value 1 if the condition inside the parentheses is true, and 0 otherwise. This indicates that at least one voxel satisfies the condition; For frame time index, , and These are the vertical, horizontal, and height indices for voxels, respectively. The truth class number for the corresponding voxel; A pre-specified set of rare categories; For the entire set of semantic categories; This indicates that the former is a subset of the latter;

[0163] Based on comprehensive noise prediction, latent variable teacher supervision, training reliability, and rare class reinforcement, the predicted total network loss is as follows:

[0164]

[0165] In the formula, The total training loss is used to predict the temporal diffusion prediction network. The total number of sample sequences; For sequence indexing; Weighted magnitude for training reliability; Predict loss for average noise; The teacher loss weight is a latent variable. The average latent variable is the teacher loss; , , and Each element controls the reference frame prior, rare class enhancement, training reliability weighting, and teacher supervision, with their meanings being distinct.

[0166] 44) Maintain an exponential moving average copy of the predicted network parameters;

[0167] Suppose the prediction network completes the first... After the secondary parameter update, its instantaneous parameters and exponential moving average parameters are updated as follows:

[0168]

[0169] In the formula, For the first The updated exponential moving average parameters; The parameters are the exponential moving average parameters from the previous update; For the first The instantaneous predicted network parameters are obtained through subgradient update; , The exponential moving average decay coefficient; The updated weights for the current real-time parameters; Update the index for the parameter; This represents the total number of parameter updates during the training phase.

[0170] Step 5: Perform backdiffusion sampling and semantic occupancy decoding using only the bird's-eye view layout:

[0171] The noise is gradually reduced by standard Gaussian random latent variables. The truth reference latent variables and truth density conditions are turned off during bird's-eye view layout reasoning. Then, the freeze decoder restores and saves the 3D semantic occupancy of multiple consecutive frames.

[0172] Furthermore, in this embodiment, the specific method of step five is as follows:

[0173] 51) Perform backdiffusion sampling only based on the bird's-eye view layout;

[0174] During inference, we start with standard Gaussian random latent variables and use an exponential moving average parameter network to predict noise and update the latent variables at each backdiffusion time step.

[0175]

[0176]

[0177] In the formula, For the first The sequence at the maximum diffusion time The initial random latent variables; Indicates a standard Gaussian distribution; It is a zero-mean tensor; For unit covariance matrix; For the diffusion time Updated to the previous moment The subsequent sequence of latent variables; This represents the cumulative signal retention coefficient at the previous diffusion moment; For estimating latent variables; The noisy sequence is predicted using an exponential moving average parameter network. For the first The random strength of each backward sampling step, and satisfying ; Add noise to the reverse sampling; For sequence indexing; The maximum number of steps for diffusion; when Deterministic sampling is formed when Random sampling is generated at that time.

[0178] 52) Decode and save the 3D semantic occupancy prediction;

[0179] Divide the final latent variable obtained from backsampling by the scaling factor and input it into the frozen decoder to select the class with the highest probability for each voxel:

[0180]

[0181] In the formula, For the first The sequence number is... The predicted semantics of a frame occupy a grid. The target voxel location, where , and These are indexes for vertical, horizontal, and height respectively; An operator that returns the category number corresponding to the highest probability; Index for candidate semantic categories; A set of semantic categories; The parameter set is Frozen 3D voxel decoder; After backsampling is completed, the first Latent variables in frame diffusion; The scaling factor is the implicit variable; the square brackets are the subscripts. Indicates the type of reading and voxel location The corresponding predicted probability; For sequence index, It serves as a frame time index; the output mesh size is consistent with the Occ3D semantic occupancy truth value, and is saved according to the scene name and sample identifier.

[0182] Example 2

[0183] In conjunction with the temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion described in Example 1, this example uses the Occ3D dataset for training and validation, selecting 5 consecutive frames (i.e., ...) from the same scene. ), a multi-channel bird's-eye view layout with an input size of 18×200×200 per frame (i.e. =18, =200, =200), corresponding to a 3D semantic occupancy ground truth size of 200×200×16 (i.e., Z=16). First, the bird's-eye view layout is rotated by a fixed 90 degrees and flipped along the axis to align with the coordinate direction of the semantic occupancy grid. Then, a variational autoencoder is used to encode the 3D semantic occupancy grid into 4×50×50 latent variables, and a scaling factor of 70 is used to adjust the numerical distribution of the latent variables. The temporal latent variable diffusion transformer is used as a condition for 5 consecutive frames of bird's-eye view layout, and noise addition and denoising learning is performed in the latent variable space. The model's hidden feature dimension is set to 512, the network depth is set to 12 layers, and a latent variable teacher loss with a weight of 0.5 is added to the noise prediction loss. During inference, only a continuous bird's-eye view layout is input, and back-diffusion sampling is performed starting from random Gaussian latent variables. The 3D semantic occupancy grid is generated by the frozen variational autoencoder decoder.

[0184] Figure 3 and Figure 4 The predicted and actual results of the 3D semantic occupancy grid are shown for scenarios 1 and 2, respectively. Figure 3 As can be seen, in scenarios where the road direction is relatively regular and vegetation, buildings, and vehicles are distributed on both sides of the road, the prediction results can recover the road plane, road boundaries, vegetation on both sides, building outlines, and vehicle positions quite well, and its overall spatial distribution is basically consistent with the actual results. Figure 4 As can be seen, even in scenarios with more complex road structures, vegetation distributions, and surrounding targets, the prediction results can still maintain the road orientation and main static structures, and effectively recover small-volume targets such as vehicles. Although there are a few missing or redundant voxels in some distant areas and local edges, it does not affect the expression of the overall scene structure. This indicates that the present invention can generate a 3D semantic occupancy mesh corresponding to the real scene based on the bird's-eye view layout, and maintain good spatial consistency, semantic integrity, and structural stability in different scenarios.

[0185] In summary, the temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion proposed in this invention maintains rich structural information through complete multi-channel bird's-eye view conditions, reduces computational load through latent variable spatial diffusion, and improves generation quality through multiple mechanisms such as reference frame prior, teacher supervision, reliability weighting, and rare category enhancement. It can generate accurate 3D semantic occupancy meshes by relying solely on bird's-eye view layout during the inference stage, providing an effective technical solution for autonomous driving environment perception and scene generation.

Claims

1. A temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion, characterized in that, Includes the following steps: Step 1: Select consecutive keyframes from the same nuScenes scene, read the complete multi-channel bird's-eye view layout and semantic occupancy annotations from the Occ3D dataset, and construct a supervised dataset; unify the coordinate direction of the bird's-eye view layout and semantic occupancy by fixing rotation and flipping, and divide the training set and validation set according to the scene; Step 2: Compress the three-dimensional semantic occupancy into continuous low-resolution latent variables using a two-dimensional encoder, recover the voxel semantics using a three-dimensional decoder, train a variational autoencoder, jointly optimize the class-weighted cross-entropy loss, Lovász-Softmax loss and relative entropy loss, and determine the upper limit of latent variable expression based on the reconstruction performance on the validation set. Step 3: Freeze the variational autoencoder, encode the ground truth of the Occ3D dataset and scale it into latent variables, add noise according to the noise schedule, form conditional features with complete bird's-eye view layout in multiple consecutive frames, construct a temporal latent variable diffusion transformer, and enable the temporal diffusion transformer to jointly learn the spatial structure and short temporal correlation. Step 4: In addition to the noise prediction loss, the latent variables are directly supervised by the latent variable teacher loss. Training weights are generated using the LiDAR observation density and image structure clarity. The sample weights are increased according to the occurrence of rare categories, and the parameter exponential moving average copy is maintained to train the prediction network. Step 5: Denoise gradually using standard Gaussian random latent variables. Turn off truth reference latent variables and truth density conditions during bird's-eye view layout inference. Perform backdiffusion sampling and semantic occupancy decoding under the condition of using only bird's-eye view layout. Recover and save the three-dimensional semantic occupancy of multiple consecutive frames by freezing the decoder.

2. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: In step one, the supervisory dataset is constructed as follows: Select consecutive keyframes from the same nuScenes scene, assuming each training sample contains consecutive keyframes. The first keyframe, the The bird's-eye view layout of each sample sequence and the semantic occupancy ground truth of the Occ3D dataset are denoted as follows: and Construct a supervised dataset: In the formula, For monitoring datasets; The total number of sample sequences; For sequence indexing; The number of consecutive keyframes contained in each sequence; For frame time index; For the first A sequence of consecutive A collection of frame-by-frame bird's-eye view layouts; For the first A sequence of consecutive Frame semantics occupies the truth set; The method for unifying the bird's-eye view layout and semantic occupancy coordinate direction is as follows: Perform a fixed 90-degree rotation and a specified axis flip on the original bird's-eye view layout to obtain a layout aligned with the semantic occupancy grid of the Occ3D dataset: In the formula, For the first The sequence number is... The original bird's-eye view layout without coordinate transformation; For an operator that rotates 90 degrees counterclockwise in the bird's-eye view plane; For the bird's-eye view plane Operators that perform mirror flipping along the axis; To achieve the aligned bird's-eye view layout after rotation and flipping; symbol Unless otherwise specified, Unified as .

3. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: Step two includes: establishing a semantic occupancy variational coding distribution and performing reparameter sampling, as follows: The first The sequence number is... The frame semantics occupies the truth input encoder, outputs the latent variable distribution parameters, obtains continuous latent variables through reparameterization, and the decoder recovers the class probabilities, as shown in the following formula: In the formula, The parameter is Variational encoder; For the first The sequence number is... The Occ3D semantics of a frame occupy truth value; The mean tensor of the Gaussian distribution of latent variables; For the latent variable log-variance tensor, i.e. ; It is the element-wise positive standard deviation tensor; It is an element-wise exponential function; Semantic occupancy is a continuous latent variable; To and A standard Gaussian random tensor of the same size; This represents a tensor with zero mean. Covariance is the identity matrix The standard Gaussian distribution; This is element-wise multiplication; The parameter is A three-dimensional voxel decoder; Let be the class probability tensor.

4. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: Step two includes: joint optimization of semantic reconstruction, region intersection and latent variable distribution, as follows: Voxel-level semantic reconstruction using category-weighted cross-entropy constraints: In the formula, Category-weighted cross-entropy loss; The total number of sample sequences; Number of consecutive frames; , and These represent the vertical dimension, horizontal dimension, and number of height layers of the voxel mesh. For sequence indexing; For frame time index; , and These are the vertical, horizontal, and height indices of the voxel, respectively; For true value In voxels Category number at the location; The reconstruction weight corresponding to this true value category; This represents the predicted probability of the ground truth class given by the decoder at the corresponding voxel position; It is the natural logarithm function; Using relative entropy to constrain the distribution of latent variables to approximate a standard Gaussian distribution: In the formula, The relative entropy regularization loss is the sum of the latent variable distribution and the standard Gaussian distribution. The index of the element after the hidden variable tensor is flattened; The number of hidden channels; and For implicit variable plane dimensions; For mean tensor In the The value at each element; For the log-variance tensor In the The value at each element; The corresponding variance; a constant. Unit variance term from a standard Gaussian distribution; The overall training objective of the variational autoencoder is as follows: In the formula, This represents the total training loss of the variational autoencoder. Let be the set of all predicted probability tensors and the complete set of semantic occupancy truth values Calculated Lovász-Softmax region loss; , and These are the weights of the corresponding loss terms; Freeze encoder parameters after training is complete and decoder parameters The upper limit of latent variable expression is determined by the reconstruction performance of the validation set.

5. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: Step three includes: freezing the variational autoencoder, scaling the semantic latent variables, and performing positive noise addition, as follows: The latent variables output by the frozen variational autoencoder are scaled by a fixed coefficient and then adjusted according to the diffusion time. Perform positive noise addition: In the formula, For the first The sequence number is... Latent variables in frame diffusion; The scaling factor is the implicit variable. Semantic occupancy is a continuous latent variable; For diffusion time index, The maximum number of steps for diffusion; For diffusion time Noisy latent variables; For diffusion time The cumulative signal holding coefficient; For the first Single-step signal preservation coefficient for each diffusion step; The diffusion step index used when calculating the cumulative product; Indicates and Independent standard Gaussian noise of the same size; , and Let represent the standard Gaussian distribution, the zero-mean tensor, and the unit covariance matrix, respectively; For sequence index, For frame time index.

6. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: Step three includes: forming temporal condition features using a complete multi-channel bird's-eye view layout, as follows: The first Continuous sequences All frames A bird's-eye view layout, channel, and time location coding. The set of input parameters is Condition encoder , obtained the Temporal bird's-eye view of the conditional features of each sequence ; The number of passageways is determined by an aerial view; Step three further includes: predicting noise using a time-series diffusion transformer, as follows: By inputting the noisy latent variables, diffusion time, and temporal bird's-eye view conditions into the prediction network, the noise estimate is obtained: In the formula, For the first The sequence at the diffusion time The predicted noise sequence; The parameter set is A time-series diffusion converter; It is a sequence of noisy latent variables in consecutive frames. ; For the first Optional low-dimensional scene structure conditions for each sequence.

7. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: Step four includes the following steps: The method for constructing a priori reference frame structure and restricting its usage is as follows: During the training phase, the latent variables of the sequence reference frame are copied to the time dimension, and random noise is added with small weights to obtain the structural prior noise: In the formula, The training noise sequence after incorporating the prior knowledge of the reference frame structure; The original Gaussian noise sequence, ; These are the prior coefficients for the reference frame structure; To copy the latent variables of a single frame to The time extension operator at each moment; For the first Diffusion latent variables of each sequence reference frame; For reference frame time index; This is a training sequence of noisy latent variables reconstructed using a reference prior; This is the cumulative signal hold factor; For a continuous frame spread latent variable sequence, ; For sequence index, Index for diffusion time; The method for directly supervising latent variables using the teacher loss is as follows: The diffusion time is deduced from the predicted noise. Corresponding latent variable estimation: In the formula, To be based on diffusion time The first noisy latent variable is obtained by back-reasoning Estimation of latent variables in a sequence; For the first The sequence at the diffusion time A noisy sequence of latent variables; To predict noisy sequences; This is the cumulative signal hold factor; For sequence indexing; Index for diffusion time; The average noise prediction loss and the average latent variable teacher loss are as follows: In the formula, For the first Average noise prediction loss for each sequence; This is the original Gaussian noise sequence added during forward diffusion; For the first The average latent variable teacher loss for each sequence; Represents the sum of squares of all elements; Number of consecutive frames; The number of hidden channels; and For implicit variable plane dimensions.

8. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: Step four includes the following steps: Sample weights are set based on sensor reliability and rare categories during the training period, as follows: The training reliability score is derived from the LiDAR observation density and image structure sharpness. In the formula, For the first Training reliability score of each sequence; The fusion weights are the reliability components of the lidar. The fusion weights for the image reliability components; For the first The effective point density of the lidar corresponding to each sequence; This is the normalization constant for the point density of the lidar; For the first Texture intensity of a sequence of front view images; This is the image structure intensity normalization constant; To truncate the input scalar to a closed interval The function; This serves as the source identifier for the lidar. Image source identification; images and point clouds are used only for calculating training weights and are not input into the prediction network. ; Class augmentation coefficients are constructed based on whether rare classes appear in the semantic occupancy ground truth values ​​of the Occ3D dataset: In the formula, For the first Rare class enhancement coefficients for each sequence; Increase the magnitude for rare categories; This is an indicator function; it takes the value 1 if the condition inside the parentheses is true, and 0 otherwise. This indicates that at least one voxel satisfies the condition; For frame time index, , and These are the vertical, horizontal, and height indices for voxels, respectively. The truth class number for the corresponding voxel; A pre-specified set of rare categories; For the entire set of semantic categories; This indicates that the former is a subset of the latter; Based on comprehensive noise prediction, latent variable teacher supervision, training reliability, and rare class reinforcement, the predicted total network loss is as follows: In the formula, The total training loss is used to predict the temporal diffusion prediction network. The total number of sample sequences; For sequence indexing; Weighted magnitude for training reliability; Predict loss for average noise; The teacher loss weight is a latent variable. The average latent variable is the teacher loss; , , and Each element controls the reference frame prior, rare class enhancement, training reliability weighting, and teacher supervision, with their meanings being distinct. The method for maintaining an exponential moving average replica of the predicted network parameters is as follows: Suppose the prediction network completes the first... After the secondary parameter update, its instantaneous parameters and exponential moving average parameters are updated as follows: In the formula, For the first The updated exponential moving average parameters; The parameters are the exponential moving average parameters from the previous update; For the first The instantaneous predicted network parameters are obtained through subgradient update; , The exponential moving average decay coefficient; The updated weights for the current real-time parameters; Update the index for the parameter; This represents the total number of parameter updates during the training phase.

9. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: In step five, the method for performing backdiffusion sampling based solely on the bird's-eye view layout is as follows: During inference, we start with standard Gaussian random latent variables and use an exponential moving average parameter network to predict noise and update the latent variables at each backdiffusion time step. In the formula, For the first The sequence at the maximum diffusion time The initial random latent variables; Indicates a standard Gaussian distribution; It is a zero-mean tensor; For unit covariance matrix; For the diffusion time Updated to the previous moment The subsequent sequence of latent variables; This represents the cumulative signal retention coefficient at the previous diffusion moment; For estimating latent variables; The noisy sequence is predicted using an exponential moving average parameter network. For the first The random strength of each backward sampling step, and satisfying ; Add noise to the reverse sampling; For sequence indexing; The maximum number of steps for diffusion; when Deterministic sampling is formed when Random sampling is generated at that time.

10. The temporal semantic occupancy prediction method based on bird's-eye view layout condition diffusion according to claim 1, characterized in that: In step five, the method for decoding semantic occupancy and saving the three-dimensional semantic occupancy prediction is as follows: Divide the final latent variable obtained from backsampling by the scaling factor and input it into the frozen decoder to select the class with the highest probability for each voxel: In the formula, For the first The sequence number is... The predicted semantics of a frame occupy a grid. The target voxel location, where , and These are indexes for vertical, horizontal, and height respectively; An operator that returns the category number corresponding to the highest probability; Index for candidate semantic categories; A set of semantic categories; The parameter set is Frozen 3D voxel decoder; After backsampling is completed, the first Latent variables in frame diffusion; The scaling factor is the implicit variable; the square brackets are the subscripts. Indicates the type of reading and voxel location The corresponding predicted probability; For sequence index, It serves as a frame time index; the output mesh size is consistent with the Occ3D semantic occupancy truth value, and is saved according to the scene name and sample identifier.

Images