A method, system and product for generating an occupancy grid scene

Abstract

The embodiment of the application provides a kind of generation method, system and product of occupancy grid scene, which comprises: embedding trajectory feature into the noise feature vector corresponding to long time sequence occupancy grid scene by embedding layer of target model, to obtain target noise feature vector;De-noising is carried out on target noise feature vector by de-noising layer of target model, to obtain to-be-generated feature vector;The to-be-generated feature vector is processed by occupancy grid decoder, to obtain the occupancy grid scene under the corresponding trajectory.The purpose is to efficiently generate long time sequence high-quality occupancy grid scene, and efficiently generate and model external environment.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a method, system, and product for generating a grid scene. Background Technology

[0002] Occupied grids are an environmental modeling method that uses sensors such as LiDAR, cameras, and millimeter-wave radar to acquire environmental information. They divide the 3D space of the environment into multiple 3D grids and mark each grid as either vacant or semantically specific. Generally, an occupied grid exists in 3D space, representing environmental information at a specific moment. 4D occupied grids introduce a temporal dimension, representing a sequence of occupied grids over a period of time, which can represent the vehicle's continuous observation of the environment during driving. For autonomous driving, occupied grids provide comprehensive and detailed environmental modeling, supporting applications such as path planning. For example, a vehicle can avoid occupied grids and plan an empty path.

[0003] World simulators are a way to deeply understand and model the environment. Most current world simulators use an autoregressive framework, which generates occupied grids by progressively predicting scene changes at each step. This method is inefficient when dealing with long time series and is prone to noise accumulation errors. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, system, and product for generating occupied grid scenes. The aim is to efficiently generate high-quality occupied grid scenes over long time sequences, and to efficiently generate and model the external environment.

[0005] The first aspect of this application provides a method for generating a grid scene, the method comprising:

[0006] By embedding the trajectory features into the noise feature vector corresponding to the long-term sequence occupying the grid scene through the embedding layer of the target model, the target noise feature vector is obtained.

[0007] The target noise feature vector is denoised by the denoising layer of the target model to obtain the feature vector to be generated;

[0008] The feature vector to be generated is processed by the occupancy grid decoder to obtain the occupancy grid scene under the corresponding trajectory.

[0009] Optionally, the initial model is trained to obtain the target model, including:

[0010] The feature vectors corresponding to the occupied grid data are discretized by the occupied grid encoder to obtain the corresponding discrete vectors.

[0011] The target discrete vector is obtained by embedding trajectory features into the discrete vector using an initial model;

[0012] The target discrete vector is subjected to noise diffusion processing to obtain a target noise discrete vector;

[0013] The target noise discrete vector is denoised by the denoising layer of the initial model to obtain a denoised discrete vector;

[0014] Based on the target discrete vector, the denoised discrete vector, and the loss function, the parameters of the initial model are updated until the noise distribution error of the initial model is minimized. At this point, the initial model is deemed to have passed training, and the target model is obtained.

[0015] Optionally, determine trajectory features, including:

[0016] The trajectory composed of the vehicle's coordinate positions at different times is encoded to obtain trajectory encoding information;

[0017] The trajectory encoding information is converted into trajectory features with the same feature size as the discrete vector using an MLP structure.

[0018] Optionally, the feature vector corresponding to the occupied raster data is discretized by an occupied raster encoder to obtain the corresponding discrete vector, including:

[0019] The feature vector corresponding to the occupied raster data is compared with each element in the codebook by the quantizer in the occupied raster encoder to obtain the comparison result;

[0020] Based on the comparison results, the feature vector corresponding to the occupied grid data is mapped to the codebook element closest to the feature vector to obtain the discrete vector corresponding to the feature vector.

[0021] Optionally, determine the feature vector corresponding to the occupied raster data, including:

[0022] By reshaping the shape, the height channel and semantic category channel of the original feature vector occupying the raster data are compressed to the feature dimension to obtain the first feature vector;

[0023] By using the downsampling 3D convolution module of the grid encoder, the first feature vector is compressed to obtain the feature vector corresponding to the grid data.

[0024] Optionally, after mapping the feature vector corresponding to the occupied raster data to the codebook element closest to the feature vector based on the comparison result to obtain the discrete vector corresponding to the feature vector, the method further includes:

[0025] The discrete vector is dequantized based on the closest codebook element to obtain the reconstructed features;

[0026] The codebook is updated based on the reconstructed features, the feature vectors corresponding to the occupied grid data, and the quantization loss function.

[0027] Optionally, the feature vector to be generated is processed by an occupancy grid decoder to obtain the occupancy grid scene under the corresponding trajectory, including:

[0028] The feature vector to be generated is extended by the upsampling 3D convolution module of the decoder to obtain the extended feature vector;

[0029] The extended feature vector is reshaped and restored using the reshaping module of the decoder to obtain the occupied grid scene under the corresponding trajectory.

[0030] Optionally, when the initial model is a DiT model, the step of embedding trajectory features into the discrete vector using the initial model to obtain the target discrete vector includes:

[0031] The discrete vector is positionally encoded using the position encoding layer of the initial model to obtain the first discrete vector.

[0032] The trajectory features are embedded into the first discrete vector through the embedding layer of the initial model to obtain the target discrete vector.

[0033] Optionally, before performing positional encoding on the discrete vector through the positional encoding layer of the initial model to obtain the first discrete vector, the method further includes:

[0034] The discrete vector is flattened to obtain a one-dimensional discrete vector.

[0035] The step of performing position encoding on the discrete vector through the position encoding layer of the initial model to obtain the first discrete vector includes: performing position encoding on the flattened one-dimensional discrete vector through the position encoding layer of the initial model to obtain the first discrete vector.

[0036] A second aspect of this application provides a system for generating a grid scene, the system comprising:

[0037] The embedding layer of the target model is used to embed the trajectory features into the noise feature vector corresponding to the long-term sequence occupying the grid scene, so as to obtain the target noise feature vector;

[0038] The denoising layer of the target model is used to denoise the target noise feature vector to obtain the feature vector to be generated;

[0039] The occupying grid decoder is used to process the feature vector to be generated to obtain the occupying grid scene under the corresponding trajectory.

[0040] A third aspect of this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, it implements the steps of a method for generating an occupied grid scene as described in the first aspect of this application.

[0041] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a method for generating an occupied grid scene as described in the first aspect of this application.

[0042] The method for generating a grid scene provided in this application has the following advantages:

[0043] This application provides a method for generating occupied grid scenes. First, trajectory features are embedded into a noise feature vector corresponding to a long-term occupied grid scene through an embedding layer of a target model, obtaining a target noise feature vector. Then, the target noise feature vector is denoised through a denoising layer of the target model to obtain a feature vector to be generated. Finally, the occupied grid decoder processes the feature vector to be generated to obtain the occupied grid scene under the corresponding trajectory. Therefore, this application, by using a target model and introducing trajectory features, can efficiently generate grid occupied scenes under corresponding trajectories for long-term sequences, enabling efficient generation and modeling of the external environment and solving the problem of low efficiency in existing autoregressive models. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a method for generating an occupying grid scene according to an embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating model training in a method for generating an occupied grid scene, as shown in one embodiment of this application;

[0047] Figure 3 Another flowchart illustrating a method for generating an occupying grid scene according to one embodiment of this application;

[0048] Figure 4 This is a schematic diagram illustrating an embodiment of the generation system for an occupied grid scene. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for generating an occupying grid scene according to one embodiment of this application. Figure 1 As shown, the method includes:

[0051] Step S1: Embed the trajectory features into the noise feature vector corresponding to the long-term sequence occupying the grid scene through the embedding layer of the target model to obtain the target noise feature vector.

[0052] In this embodiment, the vehicle trajectory corresponding to the long-term occupied grid scene to be generated is first determined. For example, if the desired trajectory is a vehicle that first goes straight at one intersection and then turns left at the next, the corresponding vehicle trajectory features are generated based on this trajectory. Then, these vehicle trajectory features are embedded into the noise feature vector corresponding to the long-term occupied grid scene through an embedding layer in a trained target model. This allows the vehicle trajectory features to guide the initial model in generating the long-term occupied grid scene based on this trajectory. After embedding, the corresponding target noise feature vector is obtained. This noise feature vector corresponding to the long-term occupied grid scene is the noise-laden grid data obtained after multiple noise additions to the long-term occupied grid data, resulting in a noisy grid data with a large amount of noise.

[0053] Step S2: Denoise the target noise feature vector through the denoising layer of the target model to obtain the feature vector to be generated.

[0054] In this embodiment, after obtaining the target noise feature vector embedded with trajectory features through step S1, the target noise feature vector is denoised by the denoising layer of the target model to remove noise from the feature vector. The target model will then output the corresponding feature vector to be generated. The target model includes, but is not limited to, the DiT (Scalable Diffusion Models with Transformers) model and the UNet model. When the target model is a DiT (Scalable Diffusion Models with Transformers) model, the denoising layer of the target model is the Transformer structure; when the target model is a UNet model, the denoising layer of the target model is the UNet structure.

[0055] Step S3: Process the feature vector to be generated by the occupancy grid decoder to obtain the occupancy grid scene under the corresponding trajectory.

[0056] In this embodiment, after denoising the target noise feature vector through the target model to obtain the corresponding feature vector to be generated, the obtained feature vector to be generated is processed by the occupied grid decoder to obtain the long-term sequence occupied grid scene under the corresponding trajectory.

[0057] This application provides a method for generating occupied grid scenes. First, trajectory features are embedded into the noise feature vector corresponding to a long-term occupied grid scene through the embedding layer of the target model, obtaining a target noise feature vector. Then, the target noise feature vector is denoised through the denoising layer of the target model to obtain a feature vector to be generated. Finally, the occupied grid decoder processes the feature vector to be generated to obtain the occupied grid scene under the corresponding trajectory. Therefore, this application can efficiently generate grid occupied scenes under corresponding trajectories in a long-term sequence by using a target model and introducing trajectory features, enabling efficient generation and modeling of the external environment and solving the problem of low efficiency in existing autoregressive models. Simultaneously, simulation testing provides a safer and more controllable environment for debugging control algorithms. Therefore, the development of autonomous driving algorithms heavily relies on simulation testing. However, due to actual physical limitations, it is difficult to collect a wide variety of scenarios for temporal occupied grid information, such as vehicles driving on collapsed roads or vehicles overturning and stopping in the middle of the road. Yet, multiple scenarios are needed for downstream algorithms to perform simulation testing during the simulation phase. The 4D occupancy grid data generation model of this application can generate an unlimited number of temporal occupancy grid scenes from noise and preset trajectories (which can be of any form). This can greatly enrich the occupancy grid scene data required for simulation testing, thereby avoiding the need to actually collect a large number of various occupancy grid scene data. This can improve the efficiency and speed of simulation testing and reduce the cost of simulation testing. In addition, as a high-precision and detailed modeling method for the physical world, 4D occupancy grids enable downstream planning and control algorithms to develop end-to-end models.

[0058] In conjunction with the above embodiments, in one implementation, this application also provides a method for generating an occupying grid scene. In this method for generating an occupying grid scene, an initial model is trained to obtain a target model, including steps S01 to S05:

[0059] Step S01: Discretize the feature vector corresponding to the occupied grid data by the occupied grid encoder to obtain the corresponding discrete vector.

[0060] In this embodiment, the main task of the 4D occupancy grid is to predict the semantic information of spatial voxels at a specific location and time. Compared to the 3D occupancy grid, the 4D occupancy grid has temporal dimension information. The original representation of the 4D occupancy grid is relatively redundant and computationally intensive; directly processing the original feature vectors corresponding to the 4D occupancy grid requires significant computational resources. Therefore, to reduce resource consumption during subsequent initial model training, this application discretizes the feature vectors corresponding to the occupancy grid data. Specifically, as shown... Figure 2As shown, the feature vectors corresponding to the occupied raster data of the long-term series are discretized by the occupied raster encoder (4DOccupancy Encoder) to obtain the corresponding discrete vectors.

[0061] Step S02: Embed trajectory features into the discrete vector using the initial model to obtain the target discrete vector.

[0062] In this embodiment, after discretizing the feature vectors corresponding to the occupied grid data of the long-term series in step S01 to obtain the corresponding discrete vectors, the initial model embeds trajectory features into these discrete vectors to obtain the target discrete vector. Since feature vectors are being constructed for subsequent model training, a large number of discrete vectors will be embedded with trajectory features. Different discrete vectors will contain different trajectories in their embedded trajectory features to ensure the diversity of training samples.

[0063] Step S03: Perform noise diffusion processing on the target discrete vector to obtain the target noise discrete vector.

[0064] In this embodiment, noise is gradually added to a large number of target discrete vectors obtained through step S02 to construct a target noise discrete vector with noise for training the initial model.

[0065] In this embodiment, the noise addition process, each step, follows a Gaussian distribution, expressed as follows:

[0066]

[0067] Where R represents the target discrete vector; q represents the noise distribution added to the target discrete vector; and g represents the trajectory characteristics at the corresponding time step. represents an empirical constant, and represents the weight of the noise accumulated to the current step; N represents a normal distribution; I represents the identity matrix.

[0068] Step S04: Denoise the target noise discrete vector through the denoising layer of the initial model to obtain a denoised discrete vector.

[0069] In this embodiment, after obtaining a large number of target noise discrete vectors that have undergone noise diffusion processing in step S03, the initial model is trained using these large number of target noise discrete vectors. Specifically, the initial model's denoising layer denoises the obtained target noise discrete vectors that have undergone noise diffusion processing, removing the noise added to the target noise discrete vectors to obtain the corresponding denoised discrete vectors. Thus, for any denoised discrete vector, there is a corresponding target noise discrete vector after noise diffusion processing, and also a target discrete vector without noise diffusion processing.

[0070] In this embodiment, the denoising process is the reverse of the noise addition process, and a model is used to learn the parameters, expressed as:

[0071]

[0072] Where, p θ Let θ represent the distribution of noise from the inverse transformation at time g to time g-1, where θ indicates that the distribution of this noise is a function of θ and is a learnable value. Σ represents the mean of the discrete vector of the target at time g; θ (R g ) represents the variance of the target discrete vector at time g.

[0073] Step S05: Based on the target discrete vector, the denoised discrete vector, and the loss function, update the parameters of the initial model until the noise distribution error of the initial model is minimized. Then, determine that the initial model is qualified for training and obtain the target model.

[0074] In this embodiment, a loss function is used to calculate the noise distribution error value. The noise distribution error value obtained by denoising the initial model's denoising layer is compared with the corresponding target discrete vector. If this noise distribution error value is smaller than the noise distribution error value of the initial model corresponding to the previous model parameters, the model parameters of the initial model are updated. Then, the denoising layer of the updated initial model is used to further denoise the obtained target noise discrete vector after noise diffusion processing, removing the noise added to the target noise discrete vector to obtain the corresponding denoised discrete vector.

[0075] Then, the loss function is used to calculate the noise distribution error value. If this noise distribution error value is larger than the noise distribution error value of the initial model corresponding to the previous round of model parameters, then the noise distribution error corresponding to the previous round of model parameters is determined to be the minimum, and the previous round of model parameters is set as the final model parameters of the initial model. This results in a successfully trained target model. Conversely, if the noise distribution error value is smaller than the noise distribution error value of the initial model corresponding to the previous round of model parameters, the initial model parameters are updated, and a new round of model training is performed until the minimum noise distribution error is found. The model parameters corresponding to this minimum noise distribution error are then set as the final model parameters of the initial model, resulting in a successfully trained target model.

[0076] In this embodiment, the expression for the loss function is:

[0077]

[0078] Where L represents the noise distribution error; This represents the denoised discrete vector after denoising the target discrete vector.

[0079] In conjunction with the above embodiments, in one implementation, this application also provides a method for generating an occupied grid scene. In this method for generating an occupied grid scene, determining trajectory features includes: encoding the trajectory composed of the coordinate positions of vehicles at different times to obtain trajectory encoding information; and converting the trajectory encoding information into trajectory features with the same feature size as the discrete vector using an MLP structure.

[0080] In this embodiment, one approach to determining trajectory features is as follows: the trajectory, composed of the coordinate positions of the vehicle at different times, is encoded to obtain trajectory encoding information, represented as B×T×2, where B represents the batch size, T represents time, and 2 represents the x and y coordinates of the location. Since the coordinates will be embedded into a discrete vector, this application further converts the obtained trajectory encoding information into trajectory features with the same feature size as the discrete vector using an MLP (Multi-Layer Perceptron) structure, represented as B×c×(hwt), where c represents the feature channel size, which is the same as the feature channel size of the discrete vector, and hwt represents the height, width, and time, respectively, with the height and width being the same as the height and width of the discrete vector.

[0081] In conjunction with the above embodiments, in one implementation, this application also provides a method for generating an occupying grid scene. In this method for generating an occupying grid scene, step S01 may include steps S011 to S012:

[0082] Step S011: The feature vector corresponding to the occupied grid data is compared with each codebook element in the codebook by the quantizer in the occupied grid encoder to obtain the comparison result.

[0083] In this embodiment, a codebook is pre-constructed for discretizing the feature vectors corresponding to the occupied grid data. The codebook has an N×D shape, where N is the defined number of codebook elements and D is the element dimension of the codebook. The quantizer in the occupied grid encoder compares the feature vectors corresponding to the occupied grid data with each element in the codebook to obtain the corresponding comparison results.

[0084] Step S012: Based on the comparison result, map the feature vector corresponding to the occupied grid data to the codebook element closest to the feature vector to obtain the discrete vector corresponding to the feature vector.

[0085] In this embodiment, based on the comparison results obtained in step S011, the codebook element closest to the feature vector corresponding to the occupied grid data is determined, and then the feature vector corresponding to the occupied grid data is mapped to the closest codebook element to obtain the discrete vector corresponding to the mapped feature vector.

[0086] In conjunction with the above embodiments, in one implementation, this application also provides a method for generating an occupied grid scene. In this method, determining the feature vector corresponding to the occupied grid data includes: compressing the height channel and semantic category channel of the original feature vector of the occupied grid data to the feature dimension through shape recombination to obtain a first feature vector; and compressing the first feature vector using the downsampling 3D convolution module of the occupied grid encoder to obtain the feature vector corresponding to the occupied grid data.

[0087] In this embodiment, to further save computing resources, one way to determine the feature vector corresponding to the grid data in this application is as follows: by shape recombination, the height channel and semantic category channel of the original feature vector occupying the grid data are compressed to the feature dimension to obtain the first feature vector. For example, if the original feature vector occupying the grid data is represented as B×D×T×H×W×C, then the first feature vector obtained after shape recombination is represented as B×(D×C)×T×H×W, where B represents the batch size, D represents the height of the grid, H represents the length of the grid, W represents the width of the grid, C represents the number of semantic categories, and (D×C) represents compressing the height channel and semantic category channel to the feature dimension. Based on the obtained first feature vector, the first feature vector is compressed by the downsampling 3D convolution module of the occupied grid encoder to obtain the feature vector corresponding to the occupied grid data. The feature vector is represented as B×(8×D×C)×T / 8×H / 8×W / 8, where 8 in (8×D×C) represents the specified feature dimension, and 8 in T / 8×H / 8×W / 8 represents the feature scale.

[0088] In conjunction with the above embodiments, in one implementation, this application also provides a method for generating an occupying grid scene. In this method for generating an occupying grid scene, after step S012, the method further includes steps S101 to S102:

[0089] Step S101: Dequantize the discrete vector based on the closest codebook element to obtain the reconstructed features.

[0090] In this embodiment, to reduce the error introduced by the codebook after discretizing the feature vector, this application continuously updates the codebook used to discretize the feature vector corresponding to the grid data. Specifically: after obtaining the discrete vector after discretization by the codebook, the discrete vector is dequantized based on the codebook element closest to it to obtain the reconstructed feature.

[0091] Step S102: Update the codebook based on the reconstructed features, the feature vector corresponding to the occupied grid data, and the quantization loss function.

[0092] In this embodiment, after obtaining the reconstructed features through step S101, the quantization loss function is used to calculate the quantization loss of the reconstructed features and the feature vectors corresponding to the reconstructed features. The codebook is updated based on the calculated quantization loss value so as to construct a codebook with better performance. The better the codebook performance, the smaller the error introduced to the feature vectors after discretizing the feature vectors through the codebook.

[0093] In conjunction with the above embodiments, in one implementation, this application also provides a method for generating an occupying grid scene. In this method for generating an occupying grid scene, step S3 may include steps S31 to S32:

[0094] Step S31: The feature vector to be generated is extended by the upsampling 3D convolution module of the decoder to obtain the extended feature vector.

[0095] In this embodiment, to improve the processing efficiency of the target model for target noise feature vectors, this application also performs shape reconstruction processing on the feature vectors of long-term occupied grid data corresponding to long-term occupied grid scenes. After shape reconstruction, the feature vectors are compressed using the downsampling 3D convolution module of the occupied grid encoder. Then, the compressed feature vectors are discretized using the quantizer and codebook in the occupied grid encoder to obtain discretized feature vectors. These discretized feature vectors are the noise feature vectors corresponding to long-term occupied grid scenes. In other words, the noise feature vectors corresponding to long-term occupied grid scenes processed by the target model are feature vectors after shape reconstruction, feature compression, and discretization, which improves the processing efficiency of the target model.

[0096] In this embodiment, after the target model processes the noise feature vector corresponding to the long-term sequence occupying the grid scene to obtain the corresponding feature vector to be generated, the occupying grid decoder performs a recovery process on the feature vector to be generated. First, the upsampling 3D convolution module of the decoder expands the feature vector to be generated to obtain the expanded feature vector. This process is a feature recovery process relative to feature compression, in order to restore the feature vector to be generated to the original size space, that is, to restore the feature vector to be generated of size B×(8×D×C)×T / 8×H / 8×W / 8 to the size space of B×(D×C)×T×H×W.

[0097] Step S32: The extended feature vector is reshaped and restored using the reshaping module of the decoder to obtain the occupied grid scene under the corresponding trajectory.

[0098] In this embodiment, the extended feature vector is then reshaped and restored by the reshape module of the decoder. This process is a feature restoration process relative to shape reconstruction, in order to restore the extended feature vector of the B×(D×C)×T×H×W size space to the feature vector of the B×D×H×W×T×C size space. Then, based on the feature vector, a long-term sequence occupying the grid scene under the corresponding trajectory is generated.

[0099] In this embodiment, as Figure 3 As shown, during model training, the original feature vectors corresponding to noise-free occupied grid data are reshaped. The encoder then downsamples and compresses the reshaped feature vectors. The compressed feature vectors are then discretized using a quantizer and codebook. Trajectory features and noise are added to the discretized feature vectors to obtain the initial feature vectors used for training the model. After successful model training, the target model is obtained. In practical applications, this target model can also reshape, compress, and discretize the original feature vectors corresponding to noisy occupied grid data. Trajectory features are then added, and the target model performs denoising. The result is then fed to the decoder for feature compression recovery and shape reconstruction recovery, yielding the reconstructed feature vectors corresponding to the noise-free occupied grid data. Based on these feature vectors, the occupied grid scene under a specific trajectory without noise can be obtained.

[0100] In conjunction with the above embodiments, in one implementation, this application also provides a method for generating an occupying grid scene. In this method for generating an occupying grid scene, when the initial model is a DiT model, step S02 may include steps S021 to S022:

[0101] Step S021: The discrete vector is positionally encoded through the position encoding layer of the initial model to obtain the first discrete vector.

[0102] In this embodiment, when the initial model to be trained is a DiT (Scalable Diffusion Models with Transformers) model, since the Transformer structure itself does not have the ability to process sequence order, this application adds positional information to each vector. This is typically achieved through a positional encoding layer, which uses a position-encoded discrete vector calculated based on sine and cosine functions to obtain the corresponding first discrete vector.

[0103] Step S022: Embed the trajectory features into the first discrete vector through the embedding layer of the initial model to obtain the target discrete vector.

[0104] In this embodiment, after obtaining the first discrete vector with position encoding through step S021, the trajectory features are embedded into the first discrete vector through the embedding layer of the initial model to obtain the corresponding target discrete vector.

[0105] In conjunction with the above embodiments, in one implementation, this application also provides a method for generating an occupied grid scene. In this method for generating an occupied grid scene, before step S021, the method further includes: flattening the discrete vector to obtain a one-dimensional discrete vector.

[0106] In this embodiment, before position encoding of the multidimensional discrete vector, the multidimensional discrete vector is flattened into a one-dimensional discrete vector by the embedding module of the initial model.

[0107] In this application, before step S021, the method further includes: flattening the discrete vector to obtain a one-dimensional discrete vector, step S021 includes: performing position encoding on the flattened one-dimensional discrete vector through the position encoding layer of the initial model to obtain a first discrete vector.

[0108] In this embodiment, after flattening the multidimensional discrete vector into a one-dimensional discrete vector, the trajectory features are embedded into the first discrete vector through the embedding layer of the initial model to obtain the corresponding target discrete vector.

[0109] Based on the same inventive concept, one embodiment of this application provides a system for generating a grid scene, such as... Figure 4 As shown, the system 400 includes:

[0110] The embedding layer 4011 of the target model 401 is used to embed the trajectory features into the noise feature vector corresponding to the long-term sequence occupying the grid scene to obtain the target noise feature vector.

[0111] The denoising layer 4012 of the target model 401 is used to denoise the target noise feature vector to obtain the feature vector to be generated;

[0112] Occupied grid decoder 402 is used to process the feature vector to be generated to obtain the occupied grid scene under the corresponding trajectory.

[0113] Optionally, the system 400 further includes a model training module for training an initial model to obtain a target model; specifically, it is used to discretize the feature vectors corresponding to the occupied grid data through an occupied grid encoder to obtain corresponding discrete vectors; specifically, it is used to embed trajectory features into the discrete vectors through the initial model to obtain a target discrete vector; specifically, it is used to perform noise diffusion processing on the target discrete vectors to obtain a target noise discrete vector; specifically, it is used to denoise the target noise discrete vectors through a denoising layer of the initial model to obtain a denoised discrete vector; specifically, it is used to update the parameters of the initial model based on the target discrete vector, the denoised discrete vector, and the loss function until the noise distribution error of the initial model is minimized, then the initial model is determined to be successfully trained, and the target model is obtained.

[0114] Optionally, the system 400 further includes a trajectory feature determination module for determining trajectory features; the trajectory feature determination module includes:

[0115] The encoding module is used to encode the trajectory composed of the coordinate positions of the vehicle at different times to obtain trajectory encoding information;

[0116] The feature conversion module is used to convert the trajectory encoding information into trajectory features with the same feature size as the discrete vector through an MLP structure.

[0117] Optionally, the model training module is further configured to compare the feature vector corresponding to the occupied grid data with each codebook element in the codebook through the quantizer in the occupied grid encoder to obtain a comparison result; and to map the feature vector corresponding to the occupied grid data to the codebook element closest to the feature vector according to the comparison result to obtain the discrete vector corresponding to the feature vector.

[0118] Optionally, the system 400 further includes a feature vector determination module, used to determine the feature vector corresponding to the occupied raster data; the feature vector determination module includes:

[0119] The reorganization module is used to compress the height channel and semantic category channel of the original feature vector occupying the raster data to the feature dimension through shape reorganization to obtain the first feature vector;

[0120] The feature compression module is used to compress the first feature vector by using the downsampling 3D convolution module of the grid encoder to obtain the feature vector corresponding to the grid data.

[0121] Optionally, the system 400 further includes:

[0122] The dequantization module is used to dequantize the discrete vector based on the nearest codebook element to obtain reconstructed features;

[0123] The codebook update module is used to update the codebook based on the reconstructed features, the feature vector corresponding to the occupied grid data, and the quantization loss function.

[0124] Optionally, the decoder's upsampling 3D convolution module is used to perform feature expansion on the feature vector to be generated to obtain an expanded feature vector;

[0125] The decoder's reshaping module is used to reshape and restore the extended feature vector to obtain the occupied grid scene under the corresponding trajectory.

[0126] Optionally, if the initial model is a DiT model, the model training module is further configured to perform position encoding on the discrete vector through the position encoding layer of the initial model to obtain a first discrete vector; and to embed trajectory features into the first discrete vector through the embedding layer of the initial model to obtain a target discrete vector.

[0127] Optionally, the model training module is further configured to flatten the discrete vector to obtain a one-dimensional discrete vector; and to perform position encoding on the flattened one-dimensional discrete vector through the position encoding layer of the initial model to obtain a first discrete vector.

[0128] Based on the same inventive concept, one embodiment of this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, it implements the steps in the method for generating an occupied grid scene as described in the first aspect of this application.

[0129] Based on the same inventive concept, one embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in a method for generating an occupied grid scene as described in the first aspect of this application.

[0130] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0131] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of this application.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0133] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0138] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0139] The above provides a detailed description of the method, system, and product for generating a grid scene provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating a grid scene, characterized in that, The method comprises: embedding trajectory features into a noise feature vector corresponding to an occupancy grid scene of a long time sequence through an embedding layer of a target model to obtain a target noise feature vector; de-noising the target noise feature vector through a de-noising layer of the target model to obtain a to-be-generated feature vector; processing the to-be-generated feature vector through an occupancy grid decoder to obtain an occupancy grid scene under a corresponding trajectory; training an initial model to obtain a target model, comprising: discretizing a feature vector corresponding to occupancy grid data through an occupancy grid encoder to obtain a corresponding discrete vector; embedding trajectory features into the discrete vector through the initial model to obtain a target discrete vector; performing noise diffusion processing on the target discrete vector to obtain a target noise discrete vector; de-noising the target noise discrete vector through a de-noising layer of the initial model to obtain a de-noised discrete vector; updating parameters of the initial model based on the target discrete vector, the de-noised discrete vector and a loss function until the noise distribution error of the initial model is minimized, determining that the initial model is qualified for training, and obtaining the target model; determining a feature vector corresponding to occupancy grid data, comprising: compressing a height channel and a semantic class channel of an original feature vector of the occupancy grid data to a feature dimension through shape reorganization to obtain a first feature vector; performing feature vector compression on the first feature vector through a down-sampling 3D convolution module of the occupancy grid encoder to obtain a feature vector corresponding to the occupancy grid data.

2. The method of claim 1, wherein, determining trajectory features, comprising: encoding a trajectory composed of coordinate positions of a vehicle at different time instants to obtain trajectory encoding information; converting the trajectory encoding information into trajectory features with the same feature size as the discrete vector through an MLP structure.

3. The method of claim 1, wherein, discretizing a feature vector corresponding to occupancy grid data through an occupancy grid encoder to obtain a corresponding discrete vector, comprising: comparing the feature vector corresponding to the occupancy grid data with each codebook element in a codebook through a quantizer in the occupancy grid encoder to obtain a comparison result; mapping the feature vector corresponding to the occupancy grid data to a codebook element closest to the feature vector according to the comparison result to obtain a discrete vector corresponding to the feature vector.

4. The method of claim 3, wherein, After mapping the feature vector corresponding to the occupancy grid data to the codebook element closest to the feature vector according to the comparison result to obtain the discrete vector corresponding to the feature vector, the method further comprises: dequantizing the discrete vector based on the closest codebook element to obtain a reconstructed feature; updating the codebook according to the reconstructed feature, the feature vector corresponding to the occupancy grid data and a quantization loss function.

5. The method of claim 1, wherein, processing the to-be-generated feature vector through an occupancy grid decoder to obtain an occupancy grid scene under a corresponding trajectory, comprising: performing feature expansion on the to-be-generated feature vector through an up-sampling 3D convolution module of the decoder to obtain an expanded feature vector; performing reshaping recovery processing on the expanded feature vector through a reshaping module of the decoder to obtain an occupancy grid scene under a corresponding trajectory.

6. The method of claim 1, wherein, In the case that the initial model is a DiT model, the trajectory feature embedding of the discrete vector by the initial model comprises: position encoding of the discrete vector by a position encoding layer of the initial model to obtain a first discrete vector; embedding of the trajectory feature into the first discrete vector by an embedding layer of the initial model to obtain a target discrete vector.

7. The method of claim 6, wherein, Before the position encoding of the discrete vector by the position encoding layer of the initial model to obtain the first discrete vector, the method further comprises: flattening the discrete vector to obtain a one-dimensional discrete vector; the position encoding of the discrete vector by the position encoding layer of the initial model to obtain the first discrete vector comprises: position encoding of the flattened one-dimensional discrete vector by the position encoding layer of the initial model to obtain the first discrete vector.

8. A system for generating an occupancy grid map of a scene, the system comprising: The system comprises: an embedding layer of the target model, configured to embed a trajectory feature into a noise feature vector corresponding to an occupancy grid scene of a long time sequence to obtain a target noise feature vector; a denoising layer of the target model, configured to denoise the target noise feature vector to obtain a to-be-generated feature vector; an occupancy grid decoder, configured to process the to-be-generated feature vector to obtain an occupancy grid scene under a corresponding trajectory. The system further comprises a model training module, configured to train an initial model to obtain a target model. The model training module is specifically configured to: discretize a feature vector corresponding to occupancy grid data by an occupancy grid encoder to obtain a corresponding discrete vector; embed a trajectory feature into the discrete vector by the initial model to obtain a target discrete vector; perform noise diffusion processing on the target discrete vector to obtain a target noise discrete vector; denoise the target noise discrete vector by the denoising layer of the initial model to obtain a denoised discrete vector; and update parameters of the initial model based on the target discrete vector, the denoised discrete vector and a loss function until a noise distribution error of the initial model is minimized, so as to determine that the initial model is qualified for training and obtain the target model. The system further comprises a feature vector determination module, configured to determine a feature vector corresponding to occupancy grid data. The feature vector determination module comprises: a recombination module, configured to compress a height channel and a semantic class channel of an original feature vector of the occupancy grid data to a feature dimension by shape recombination to obtain a first feature vector; a feature compression module, configured to compress the first feature vector to a feature vector corresponding to the occupancy grid data by a down-sampling 3D convolution module of the occupancy grid encoder.

9. An electronic device, comprising: comprise: a processor, a memory, and a computer program stored on the memory and running on the processor, the computer program being executed by the processor to implement the steps of the method for generating an occupancy grid scene according to any one of claims 1 to 7.

10. A computer readable storage medium characterized by, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps in the generation method of the occupancy grid scene according to any one of claims 1 to 7.

Images