An autonomous driving occupancy grid prediction method using multi-frame and rendering supervision

Abstract

The application discloses an automatic driving occupancy grid prediction method using multiple frames and rendering supervision, comprising the following steps: S1, inputting data of a main perspective image into a processing procedure of a GaussianFormer model to obtain a predicted spatial main perspective semantic Gaussian point cloud; then, the processing procedure of the GaussianFormer model is used to obtain a main perspective occupancy grid prediction loss; S2, using three-dimensional dynamic object labeling to convert the main perspective semantic Gaussian point cloud to obtain an auxiliary perspective semantic Gaussian point cloud; S3, using the processing procedure of the GaussianFormer model to obtain an auxiliary perspective occupancy grid prediction loss; S4, using an image semantic extraction network to obtain a main perspective semantic graph and an auxiliary perspective semantic graph; S5, obtaining a main perspective semantic rendering graph and an auxiliary perspective semantic rendering graph through semantic rendering; and S6, jointly using the main perspective semantic graph and the main perspective semantic rendering graph and the auxiliary perspective semantic graph and the auxiliary perspective semantic rendering graph to obtain a semantic rendering loss.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and in particular relates to an autonomous driving occupancy prediction method using multiple frames and rendering supervision. Background Art

[0002] In the field of autonomous driving, traditional bird's-eye view (BEV) target detection methods have always been key tasks, aiming to identify and locate objects of predefined categories in the environment. BEVFormer is a pure visual perception method based on the Transformer model, focusing on generating bird's-eye view features from multi-camera images, effectively querying and fusing image features through spatial cross-attention and temporal self-attention, and supporting tasks such as 3D object detection and semantic map segmentation. With the development of technology, occupancy prediction has gradually become mainstream, which requires not only precise geometric perception, but also semantic classification of objects in the scene. The occupancy prediction task aims to predict the semantic occupancy of each voxel in three-dimensional space. By predicting the occupancy state of each voxel, the autonomous driving system can better understand and navigate complex environments, thereby improving driving safety and stability.

[0003] GaussianFormer is a 3D semantic occupancy prediction model based on a 3D Gaussian distribution, primarily used for 3D scene reconstruction and semantic understanding in autonomous driving scenarios. However, current methods like GaussianFormer do not fully utilize annotated data, particularly the temporal information between previous and next frames, to supervise the training process. Acquiring annotated data is very costly, as it requires significant manpower and time to manually annotate each frame with object and scene information. Furthermore, the annotation process requires a high level of expertise to ensure data accuracy and consistency. Leveraging annotated data for supervision not only improves the model's prediction accuracy but also enhances its robustness across diverse scenarios and conditions. Therefore, leveraging this costly annotated data, particularly the temporal information between previous and next frames, to effectively supervise the entire prediction process is a valuable and pressing issue. This will not only help improve the performance of existing models but also provide a more solid foundation for the future development of autonomous driving technology. Summary of the Invention

[0004] To solve the above problems, the present invention proposes an autonomous driving occupancy prediction method using multiple frames and rendering supervision. This method converts the prediction result of one frame to another frame through the labeled information and the positions of the vehicles in the previous and next frames, and supervises the prediction of this frame through the labeled data of the other frame. This method can ensure better prediction of occupancy in the autonomous driving scene space even when the resolution of the labeled data is low.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for autonomous driving occupancy prediction using multi-frame and rendering supervision, comprising the following steps:

[0007] S1. Input the data of the main view image into the processing flow of the GaussianFormer model to obtain the predicted main view semantic Gaussian point cloud in the space; then use the processing flow of the GaussianFormer model to use the main view space occupancy grid prediction and the main view space occupancy grid annotation to obtain the main view occupancy grid prediction loss;

[0008] S2. The main view semantic Gaussian point cloud is annotated with a 3D dynamic object and converted into an auxiliary view semantic Gaussian point cloud through a point cloud conversion process;

[0009] S3. Using the processing flow of the GaussianFormer model, using the auxiliary view space occupancy grid prediction and the auxiliary view space occupancy grid annotation, obtains the auxiliary view space occupancy grid prediction loss;

[0010] S4. Using an image semantic extraction network, extract semantic information from the primary view image and the auxiliary view image respectively to obtain the primary view semantic intent and the auxiliary view semantic intent;

[0011] S5. Rendering the main view semantic Gaussian point cloud and the auxiliary view semantic Gaussian point cloud at the positions of the main view image and the auxiliary view image respectively through semantic rendering to obtain the main view semantic rendering image and the auxiliary view semantic rendering image;

[0012] S6. Combine the main perspective semantic intent and the main perspective semantic rendering image with the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering image to obtain the semantic rendering loss.

[0013] Preferably, the data in step S1 are multi-perspective, multi-frame images and their camera parameters at two adjacent moments of the autonomous driving scene.

[0014] Preferably, the specific process of step S2 is:

[0015] S21, using the three-dimensional dynamic object annotation to determine whether the point in the main view semantic Gaussian point cloud is within the annotation;

[0016] S22. Transform the position and rotation quadruple of the Gaussian point cloud of the point located within the annotation to obtain an auxiliary perspective semantic Gaussian point cloud.

[0017] Preferably, the three-dimensional dynamic object annotation in step S2 includes the position and category of each occupied grid.

[0018] Preferably, the specific process of step S6 is: calculating L2Loss between the main perspective semantic intent and the main perspective semantic rendering graph to obtain the main perspective semantic rendering loss; calculating L2Loss between the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering graph to obtain the auxiliary perspective semantic rendering loss; then adding the main perspective semantic rendering loss and the auxiliary perspective semantic rendering loss to obtain the global semantic rendering loss.

[0019] By adopting the above technical solution, the present invention has the following beneficial effects: the present invention transfers the prediction results of one frame to another frame using the annotated information and the vehicle positions of the two preceding and succeeding frames, and uses the annotated data of the other frame to supervise the prediction of the first frame. This method can ensure better prediction of occupancy in the autonomous driving scene space when the annotated data resolution is low. At the same time, the combination of 2D images and 3D space can improve the robustness and generalization ability of the model. The multi-view 2D image data can more comprehensively capture the details and changes in the scene, thereby improving the accuracy of 3D reconstruction and object detection. In autonomous driving perception, using 2D information to supervise the 3D space can effectively improve the performance and reliability of the model. This method provides the autonomous driving system with more comprehensive and accurate environmental perception capabilities, thereby improving driving safety and stability. Therefore, in the form of a GaussianFormer model, the present invention uses predicted Gaussian points to render the semantics of a certain perspective, and compares the rendered result with the original 2D semantic prediction to ensure fine-grained supervision of the Gaussian points of the semantics in the predicted space. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of the present invention;

[0021] Figure 2 This is a schematic diagram of point cloud conversion according to the present invention;

[0022] Figure 3 It is a schematic diagram of the process framework of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] like Figures 1 to 3 As shown, a method for autonomous driving occupancy prediction using multiple frames and rendering supervision includes the following steps:

[0025] S1. Input the data of the main view image into the processing flow of the GaussianFormer model to obtain the predicted main view semantic Gaussian point cloud in the space; then use the processing flow of the GaussianFormer model to use the main view space occupancy grid prediction and the main view space occupancy grid annotation to obtain the main view occupancy grid prediction loss;

[0026] The data in step S1 is multi-view, multi-frame images and their camera parameters at two adjacent moments of the autonomous driving scene;

[0027] S2. The main view semantic Gaussian point cloud is annotated with a 3D dynamic object and converted into an auxiliary view semantic Gaussian point cloud through a point cloud conversion process;

[0028] The specific process of step S2 is:

[0029] S21, using the three-dimensional dynamic object annotation to determine whether the point in the main view semantic Gaussian point cloud is within the annotation;

[0030] S22, transforming the position and rotation quadruple of the Gaussian point cloud of the point located within the annotation to obtain an auxiliary perspective semantic Gaussian point cloud;

[0031] The three-dimensional dynamic object annotation in step S2 includes the position and category of each occupied grid;

[0032] S3. Using the processing flow of the GaussianFormer model, using the auxiliary view space occupancy grid prediction and the auxiliary view space occupancy grid annotation, obtains the auxiliary view space occupancy grid prediction loss;

[0033] S4. Using an image semantic extraction network, extract semantic information from the primary view image and the auxiliary view image respectively to obtain the primary view semantic intent and the auxiliary view semantic intent;

[0034] S5. Rendering the main view semantic Gaussian point cloud and the auxiliary view semantic Gaussian point cloud at the positions of the main view image and the auxiliary view image respectively through semantic rendering to obtain the main view semantic rendering image and the auxiliary view semantic rendering image;

[0035] S6. Combine the main perspective language intent and the main perspective semantic rendering image with the auxiliary perspective language intent and the auxiliary perspective semantic rendering image to obtain the semantic rendering loss;

[0036] The specific process of step S6 is as follows: the L2 Loss is calculated between the main perspective semantic intent and the main perspective semantic rendering image to obtain the main perspective semantic rendering loss; the L2 Loss is calculated between the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering image to obtain the auxiliary perspective semantic rendering loss; and the main perspective semantic rendering loss and the auxiliary perspective semantic rendering loss are added together to obtain the global semantic rendering loss.

[0037] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for autonomous driving occupancy prediction using multi-frame and rendering supervision, characterized in that: The following steps are involved: S1. Input the data of the main view image into the processing flow of the GaussianFormer model to obtain the predicted main view semantic Gaussian point cloud in the space; then use the processing flow of the GaussianFormer model to use the main view space occupancy grid prediction and the main view space occupancy grid annotation to obtain the main view occupancy grid prediction loss; The data in step S1 is multi-view, multi-frame images and their camera parameters at two adjacent moments of the autonomous driving scene; S2. The main view semantic Gaussian point cloud is annotated with a 3D dynamic object and converted into an auxiliary view semantic Gaussian point cloud through a point cloud conversion process; S3. Using the processing flow of the GaussianFormer model, using the auxiliary view space occupancy grid prediction and the auxiliary view space occupancy grid annotation, obtains the auxiliary view space occupancy grid prediction loss; S4. Using an image semantic extraction network, extract semantic information from the primary view image and the auxiliary view image respectively to obtain the primary view semantic intent and the auxiliary view semantic intent; S5. Rendering the main view semantic Gaussian point cloud and the auxiliary view semantic Gaussian point cloud at the positions of the main view image and the auxiliary view image respectively through semantic rendering to obtain the main view semantic rendering image and the auxiliary view semantic rendering image; S6. Combine the main perspective language intent and the main perspective semantic rendering image with the auxiliary perspective language intent and the auxiliary perspective semantic rendering image to obtain the semantic rendering loss; The specific process of step S6 is as follows: calculating the L2 Loss between the main perspective semantic intent and the main perspective semantic rendering to obtain the main perspective semantic rendering loss; calculating the L2 Loss between the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering to obtain the auxiliary perspective semantic rendering loss; Then add the main perspective semantic rendering loss and the auxiliary perspective semantic rendering loss to obtain the global semantic rendering loss.

2. The method for autonomous driving occupancy prediction using multiple frames and rendering supervision according to claim 1, wherein: The specific process of step S2 is: S21, using the three-dimensional dynamic object annotation to determine whether the point in the main view semantic Gaussian point cloud is within the annotation; S22. Transform the position and rotation quadruple of the Gaussian point cloud of the point located within the annotation to obtain an auxiliary perspective semantic Gaussian point cloud.

3. The method for autonomous driving occupancy prediction using multiple frames and rendering supervision according to claim 1, wherein: The three-dimensional dynamic object annotation in step S2 includes the position and category of each occupied grid.

Images