Three-dimensional occupancy prediction method and three-dimensional occupancy prediction device

Abstract

The invention provides a three-dimensional occupancy prediction method and a three-dimensional occupancy prediction device. The method comprises the following steps: acquiring a multi-view image of a surrounding environment of a vehicle by using a sensor, processing the multi-view image by using a basic occupancy prediction model to obtain a three-dimensional grid network of the multi-view image, performing feature extraction on the three-dimensional grid network to obtain voxel features of the multi-view image, and determining the voxel features of the multi-view image according to the voxel features of the multi-view image. The voxel features serve as conditional variables to guide generation of a diffusion model, and the diffusion model is used for conducting three-dimensional occupancy prediction on the surrounding environment of the vehicle. According to the three-dimensional occupancy prediction method, the voxel features of the three-dimensional grid network of the known scene serve as conditional variables to guide generation of the diffusion model, the diffusion model can be helped to improve the accuracy of occupancy grid generation, missing data caused by shielding and other reasons are filled, a complete occupancy map is generated, and the prediction efficiency is improved. And the generated map is ensured to accord with the actual situation.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a three-dimensional occupancy prediction method and a three-dimensional occupancy prediction device. Background Art

[0002] 3D occupancy prediction can generate a 3D representation of the surrounding environment based on multiple sensor data. The core is to construct a 3D voxel grid that describes the location of environmental objects and their occupancy through algorithmic processing and analysis of sensor input. 3D occupancy prediction can provide a relatively complete and accurate environmental model, thereby supporting higher-level autonomous navigation functions, including but not limited to path planning, obstacle avoidance, target tracking, and future traffic situation prediction.

[0003] Existing 3D occupancy prediction methods directly map from images to occupancy grids, ignoring possible correlations and structural information between adjacent voxels, and are unable to make accurate predictions for unseen areas, resulting in defects in the generated occupancy maps. Summary of the invention

[0004] A first aspect of the present application provides a three-dimensional occupancy prediction method, the method comprising:

[0005] Using sensors to obtain multi-view images of the vehicle's surroundings;

[0006] Processing the multi-view image using a basic occupancy prediction model to obtain a three-dimensional grid network of the multi-view image, performing feature extraction on the three-dimensional grid network to obtain voxel features of the multi-view image, and using the voxel features as conditional variables to guide generation of a diffusion model;

[0007] The diffusion model is used to perform three-dimensional occupancy prediction of the vehicle surrounding environment.

[0008] In one embodiment, the three-dimensional occupancy prediction method further includes: inputting a known original occupancy grid into the diffusion model to train the diffusion model.

[0009] In one embodiment, the training method for the diffusion model includes: a forward process of adding noise to the original occupied grid using the diffusion model to obtain a noisy occupied grid, and a reverse process of denoising the noisy occupied grid using a denoiser using the diffusion model, and obtaining a training prediction result after the reverse process is completed.

[0010] In one embodiment, noise addition is performed in the original occupancy grid using a transfer matrix.

[0011] In one embodiment, the uniform transfer matrix is ​​described using a uniform resampling method.

[0012] In one embodiment, a cosine schedule is used to control the rate at which noise is added in the original occupancy grid.

[0013] In one embodiment, a learnable embedding layer is used to map the discrete labels of the noisy occupancy grid into a high-dimensional continuous feature space and then serve as the input of the denoiser.

[0014] In one embodiment, in the backward process, the denoised variables are predicted using a reparameterization technique.

[0015] In one embodiment, a joint loss function of the diffusion model is constructed according to the KL divergence between the forward process and the reverse process and the auxiliary loss between the original occupied grid and the training prediction result, and the diffusion model is evaluated using the joint loss function.

[0016] A second aspect of the present application provides a three-dimensional occupancy prediction device for executing the above three-dimensional occupancy prediction method, wherein the three-dimensional occupancy prediction device includes an image acquisition unit, a processing unit, and a prediction unit;

[0017] The image acquisition unit is used to acquire multi-view images of the vehicle's surroundings;

[0018] The processing unit is used to process the multi-view image acquired by the image acquisition unit to obtain a three-dimensional grid network of the multi-view image, and perform feature extraction on the three-dimensional grid network to obtain voxel features of the multi-view image;

[0019] The prediction unit is used to perform three-dimensional occupancy prediction on the vehicle surrounding environment based on the guidance of the voxel features of the multi-view images.

[0020] The three-dimensional occupancy prediction method provided in the embodiment of the present application uses the voxel features of the three-dimensional grid network of the known scene as conditional variables to guide the generation of the diffusion model, which can help the diffusion model improve the accuracy of occupancy grid generation, fill in the missing data caused by occlusion and other reasons, generate a complete occupancy map, and ensure that the generated map conforms to the actual situation.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] Figure 1 A flowchart of a three-dimensional occupancy prediction method provided in one embodiment of the present application;

[0024] Figure 2 This is a functional structure diagram of a three-dimensional occupancy prediction device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0025] Here, the technical solutions in the embodiments (or "implementations") of the present application will be described clearly and completely in conjunction with the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0026] If there are terms involving directional indications or positional relationships in the embodiments of the present application (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings); if the specific posture changes, the directional indication or positional relationship will also change accordingly. In addition, the terms "first", "second", etc. involved in the embodiments of the present application are only used for the purpose of convenience of description and cannot be understood as indicating or implying relative importance.

[0027] In conjunction with the accompanying drawings, a three-dimensional occupancy prediction method and a three-dimensional occupancy prediction device according to an embodiment of the present application are described in detail below. In the absence of conflict, the features of the following embodiments and implementations may complement or be combined with each other.

[0028] The present application embodiment provides a three-dimensional occupancy prediction method, such as Figure 1 As shown, the method includes:

[0029] Step 100: Use sensors to acquire multi-view images of the vehicle's surroundings.

[0030] Step 200: Process the multi-view image to obtain a three-dimensional grid network of the multi-view image, use a basic occupancy prediction model to perform feature extraction on the three-dimensional grid network of the multi-view image to obtain voxel features of the multi-view image, and use the voxel features as conditional variables to guide the generation of a diffusion model.

[0031] Step 300: Use the diffusion model to perform three-dimensional occupancy prediction on the vehicle surroundings.

[0032] Multi-view images are image data based on different angles of the vehicle. In the autonomous driving scenario, multi-view images can include images from different viewpoints of the camera, lidar point cloud data or other sensor data. The basic occupancy prediction model can be any bird's-eye view model, which can fuse multi-view images into a unified top view image to generate a three-dimensional grid network. The basic occupancy prediction model extracts features from the three-dimensional grid network to obtain voxel features of the multi-view image, and uses the voxel features as conditional variables to guide the generation of the diffusion model. In the denoising process of three-dimensional occupancy prediction, the diffusion model refers to the information of the voxel features at each denoising step. If the voxel features show that a certain area is more likely to be occupied, the diffusion model will consider this information during denoising and tend to mark the area as occupied. If the voxel features show that a certain area is free, the diffusion model infers that the state of the area after denoising should be unoccupied based on this information.

[0033] The three-dimensional occupancy prediction method of the present application uses the voxel features of the three-dimensional grid network of the known scene as conditional variables to guide the generation of the diffusion model, which can help the diffusion model improve the accuracy of occupancy grid generation, fill in the missing data caused by occlusion and other reasons, generate a complete occupancy map, and ensure that the generated map conforms to the actual situation.

[0034] In one embodiment, the three-dimensional occupancy prediction method further includes: inputting a known original occupancy grid into the diffusion model to train the diffusion model. The original occupancy grid may be a real occupancy situation of the vehicle surrounding environment obtained by a sensor, and may be used as raw data in the training process, and the raw data may include various environments and situations so that the diffusion model can learn a wide range of rules.

[0035] In one embodiment, the training method for the diffusion model also includes: a forward process of adding noise to the original occupied grid using the diffusion model to obtain a noisy occupied grid, and a reverse process of denoising the noisy occupied grid using a denoiser using the diffusion model, and obtaining a training prediction result after the reverse process is completed.

[0036] In the forward process of adding noise to the original occupancy grid using the diffusion model and obtaining the noisy occupancy grid, the voxel features of the original occupancy grid are used as the original data x0. Stepwise addition of noise to the original data x0. For time step t=1, generate x1 from x0, ..., and for time step t=T, generate x T , that is, the original data x0 is gradually covered by noise and becomes a series of increasingly noisy random variables x 1:T =x1,x2,…,x T , and finally get the noise data x T .

[0037] In the reverse process of denoising the noisy occupancy grid using the denoiser using the diffusion model, the noise data x T After a series of denoising steps, we finally get a sample that is close to the original data x0. The reverse process can be expressed as For time step t = T, from x T Generate x T-1 , ..., until time step t = 1, the training prediction results are obtained

[0038] In one embodiment, the denoiser is a variant of U-Net, and the denoiser performs denoising using a parameterised method of x0. The denoiser of the diffusion model does not directly predict the random variable x at the next time step at each time step. t-1 Instead of using the denoiser to predict the original data x0 in the initial noise-free state, the denoiser is used to calculate the random variable x0 based on the predicted x0 and the current time step. t Derive the random variable x t-1 Compared to directly predicting x t-1 ,This method can better control the noise removal and enable the diffusion model to more accurately restore the original data x0.

[0039] In one embodiment, the noise is added to the original occupancy grid using a transfer matrix. In the diffusion model, the forward process is to gradually add noise to the original data x0, so that it is transformed from real data to fully noisy noise data x T For a random variable x t-1 and x t , their value range is discrete, x t-1 ,x t ∈{1,2,…,K}, that is, the random variable x t-1 and x t There are K different categories.

[0040] For discrete random variables, the noise addition process can be obtained by the transfer matrix Q t To describe, for the random variable x corresponding to time step t-1 t-1 , at time step t, the random variable x t The state of is obtained by transferring from the state of the previous moment, and its forward transfer probability is [Q t ] ij =q(x t =j|x t-1 =i), at time step t-1, the random variable x t-1 The value of is i, and at time step t, the probability of it transferring to j is [Q t ] ij. Representing the random variable as a one-hot version of the row vector x, the forward transition probability is q(x t |x t-1 )=Cat(x t ; p = x t-1 Q t ). Cat(x t ; p) means that at time t, the random variable x t According to the probability p, it is distributed in each category of K. t-1 Q t Represents time x t-1 At x t-1 The random variables of the state undergo uniform transfer matrix Q t The calculated vector p.

[0041] In the diffusion model, from the original data x0 to the random variable x t The t-step marginal distribution of The transfer matrix represents the transfer operation after multiple time steps, This means that the random variable x t The distribution of is based on the original data x0 through the transfer matrix After t steps of diffusion, the marginal distribution q(x t |x0) can get the random variable x t The distribution of .

[0042] In one embodiment, the uniform resampling method is used to describe the uniform transfer matrix, and the forward transfer probability is 1-β t The probability that the random variable x t Keep the state of the previous time, so that the data retains some of the original information; t The probability that the random variable x t Randomly and uniformly resample from K possible categories to gradually randomize the data.

[0043] In one embodiment, cosine scheduling is used to control the rate at which noise is added to the original occupancy grid. The diffusion model needs to gradually add noise to the original data x0 at each time step. If the noise is added too quickly, the diffusion model may be exposed to highly noisy data at earlier time steps, making it difficult for the diffusion model to learn how to recover the original data from slight noise. Cosine scheduling ensures that the intensity of the noise addition is small at the beginning and large at the end, thereby matching the gradual noise addition process of the diffusion model.

[0044] In one embodiment, in the reverse process, the denoised variable is predicted using the reparameterization technique. For the original data x0 and the random variable x t , the random variable x at time t-1 t-1 The conditional probability distribution of where q(x t-1 |x t ,x0) refers to the given original data x0 and random variable x t-1 After that, the random variable x t The conditional probability of q(x t-1 |x0) refers to the original data x0 to the random variable x t-1 The marginal probability of q(x t |x0) refers to the original data x0 to the random variable x t In the reverse process, the reparameterization technique is used to predict the denoised variable During the training process, the random variable x is adjusted by continuously optimizing θ t-1 to denoise the noisy occupancy grid.

[0045] In one embodiment, a learnable embedding layer is used to map the discrete labels of the noisy occupancy grid to a high-dimensional continuous feature space as the input of the denoiser. In the forward process, the diffusion model applies noise to the category label of each voxel in the original occupancy grid, so that these discrete category labels are gradually randomized. In order to enable the diffusion model to better handle the noisy discrete labels, the discrete labels are converted into high-dimensional continuous feature vectors through a learnable embedding layer during the directional denoising process. The vector can not only represent the category of the voxel, but also contain information such as similarity with other categories and position in the feature space, so that the diffusion model can use this information to recover the noisy data.

[0046] In one embodiment, the Gumbel technique is used to enable the diffusion model to be optimized by gradient descent during training. When using standard discrete sampling, the neural network cannot optimize parameters by gradient propagation because discrete sampling is not differentiable. The Gumbel technique allows the model to sample from a discrete distribution during training while still performing effective gradient updates by introducing differentiable noise.

[0047] In one embodiment, the CFG (Classifier-Free Guidance) method is used to perform the conditional generation task. According to the formula l = (s+1) l c -sl u, the diffusion model can combine conditional and unconditional generation processes to generate more reasonable occupancy grids or three-dimensional scenes. l is the final guided logit, which controls the behavior of the model when generating data; l c is the logit of the conditional model, i.e., the output of the diffusion model when it is generated under given specific conditions; l u is the logit of the unconditional model, that is, the output of the model when there is no conditional input; s is the bootstrap scale, which is used to control the balance between conditional and unconditional generation. The output of the diffusion model can be controlled by controlling the bootstrap scale s.

[0048] In one embodiment, a joint loss function of the diffusion model is constructed according to the KL divergence between the forward process and the reverse process and the auxiliary loss between the original occupied grid and the training prediction result, and the diffusion model is evaluated using the joint loss function.

[0049] Joint loss function It consists of two parts, where the KL divergence between the forward process and the reverse process is D KL (q(x t-1 |x t ,x0)||p θ (x t-1 |x t ), describes the distribution q(x t-1 |x t ,x0) and the distribution of the reverse process p θ (x t-1 |x t ), by minimizing this difference, the diffusion model can learn how to gradually restore the noisy data to the original data during the denoising process. The auxiliary loss of the original occupancy grid and the occupancy grid of the predicted result is Describes the original data x0 and the predicted results By minimizing the auxiliary loss, the diffusion model can better generate results consistent with the original data.

[0050] The present application also provides a three-dimensional occupancy prediction device, such as Figure 2 As shown, the three-dimensional occupancy prediction device includes an image acquisition unit 10, a processing unit 20 and a prediction unit 30. The image acquisition unit 10 is used to acquire a multi-view image of the vehicle's surroundings. The processing unit 20 is used to process the multi-view image acquired by the image acquisition unit 10 to obtain a three-dimensional grid network of the multi-view image, and perform feature extraction on the three-dimensional grid network to obtain voxel features of the multi-view image. The prediction unit 30 is used to perform three-dimensional occupancy prediction on the vehicle's surroundings based on the guidance of the voxel features of the multi-view image.

[0051] The processing unit 20 can extract voxel features from the multi-view image. The voxel features may include information such as the geometric structure, object distribution and category in the scene, and use this information to guide the prediction unit to help the prediction unit 30 make accurate predictions.

[0052] In one embodiment, the three-dimensional occupancy prediction device includes a first input terminal 40 and an output terminal 50, and the prediction unit 30 includes a noise adding unit 31 and a noise removing unit 32. The noise adding unit 31 is used to receive the original data input by the first input terminal 40 and add noise to the original data, and then transmit the noise added data to the noise removing unit 32 for noise removal and generating a training prediction result, and then output the training prediction result through the output terminal 50. In this way, the prediction unit 30 is trained to improve the prediction accuracy of the prediction unit 30.

[0053] In one embodiment, the three-dimensional occupancy prediction device includes a second input terminal 60, and the prediction unit 30 is used to receive data of the vehicle's surrounding environment input by the second input terminal 60, that is, a completely random noise image, and use the denoising unit 32 to denoise the noise image to obtain a prediction result that conforms to the actual scene distribution, and output the prediction result through the output terminal 50.

[0054] It should be noted that the technical solutions or technical features described in the above embodiments can be combined or supplemented with each other without causing conflicts. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the drawings; all modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A three-dimensional occupancy prediction method, characterized in that: The method comprises: Using sensors to obtain multi-view images of the vehicle's surroundings; Processing the multi-view image using a basic occupancy prediction model to obtain a three-dimensional grid network of the multi-view image, performing feature extraction on the three-dimensional grid network to obtain voxel features of the multi-view image, and using the voxel features as conditional variables to guide generation of a diffusion model; The diffusion model is used to perform three-dimensional occupancy prediction of the vehicle surrounding environment.

2. The three-dimensional occupancy prediction method according to claim 1, characterized in that: The three-dimensional occupancy prediction method further includes: inputting a known original occupancy grid into the diffusion model to train the diffusion model.

3. The three-dimensional occupancy prediction method according to claim 2, characterized in that: The training method for the diffusion model includes: a forward process of adding noise to the original occupied grid using the diffusion model to obtain a noisy occupied grid, and a reverse process of denoising the noisy occupied grid using a denoiser using the diffusion model, and obtaining a training prediction result after the reverse process is completed.

4. The three-dimensional occupancy prediction method according to claim 3, characterized in that: Noise is added in the original occupancy grid using the transfer matrix.

5. The three-dimensional occupancy prediction method according to claim 4, characterized in that: The uniform transfer matrix is ​​described using a uniform resampling method.

6. The three-dimensional occupancy prediction method according to claim 3, characterized in that: A cosine schedule is used to control the rate at which noise is added in the original occupancy grid.

7. The three-dimensional occupancy prediction method according to claim 3, characterized in that: The discrete labels of the noisy occupancy grid are mapped to a high-dimensional continuous feature space using a learnable embedding layer as input to the denoiser.

8. The three-dimensional occupancy prediction method according to claim 3, characterized in that: In the backward process, the denoised variables are predicted using the reparameterization technique.

9. The three-dimensional occupancy prediction method according to claim 4, characterized in that: A joint loss function of the diffusion model is constructed according to the KL divergence between the forward process and the reverse process and the auxiliary loss between the original occupied grid and the training prediction result, and the diffusion model is evaluated using the joint loss function.

10. A three-dimensional occupancy prediction device, used to execute the three-dimensional occupancy prediction method according to any one of claims 1 to 9, characterized in that: The three-dimensional occupancy prediction device includes an image acquisition unit, a processing unit and a prediction unit; The image acquisition unit is used to acquire multi-view images of the vehicle's surroundings; The processing unit is used to process the multi-view image acquired by the image acquisition unit to obtain a three-dimensional grid network of the multi-view image, and perform feature extraction on the three-dimensional grid network to obtain voxel features of the multi-view image; The prediction unit is used to perform three-dimensional occupancy prediction on the vehicle surrounding environment based on the guidance of the voxel features of the multi-view images.

Images