Automatic driving occupancy grid prediction method using multiple frames and rendering supervision

Abstract

The invention discloses an automatic driving occupancy grid prediction method using multiple frames and rendering supervision, and the method comprises the following steps: S1, inputting the data of a main visual angle image into a processing flow of a Gaussian Former model, and obtaining a predicted main visual angle semantic Gaussian point cloud in a space; adopting a processing flow of a Gaussian Former model to obtain a main view angle occupancy lattice prediction loss; s2, labeling the semantic Gaussian point cloud of the main view angle by using a three-dimensional dynamic object, and converting to obtain a semantic Gaussian point cloud of an auxiliary view angle; s3, adopting a processing flow of a Gaussian Former model to obtain the prediction loss of the occupancy grid of the auxiliary view angle; s4, adopting an image semantic extraction network to obtain a main view language intention and an auxiliary view language intention; s5, through semantic rendering, obtaining a main visual angle semantic rendering graph and an auxiliary visual angle semantic rendering graph; and S6, jointly using the main view language intention and the main view semantic rendering graph as well as the auxiliary view language intention and the auxiliary view semantic rendering graph to obtain semantic rendering loss.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to an autonomous driving occupancy prediction method using multiple frames and rendering supervision. Background Art

[0002] In the field of autonomous driving, traditional bird's-eye view (BEV) target detection methods have always been a key task, aiming to identify and locate objects of predefined categories from the environment. BEVFormer is a pure visual perception method based on the Transformer model, focusing on generating bird's-eye view features from multi-camera images, effectively querying and fusing image features through spatial cross-attention and temporal self-attention, and supporting tasks such as 3D object detection and semantic map segmentation. With the development of technology, occupancy prediction has gradually become mainstream, which requires not only precise geometric perception, but also semantic classification of objects in the scene. The occupancy prediction task aims to predict the semantic occupancy of each voxel in three-dimensional space. By predicting the occupancy state of each voxel, the autonomous driving system can better understand and navigate complex environments, thereby improving driving safety and stability.

[0003] GaussianFormer is a 3D semantic occupancy prediction model based on 3D Gaussian distribution, which is mainly used for 3D scene reconstruction and semantic understanding in autonomous driving scenarios. However, current methods such as GaussianFormer do not make full use of annotated data, especially the annotated data between the previous and next frames to supervise the training process. The acquisition cost of annotated data is very high because a lot of manpower and time are required to manually annotate the objects and scene information in each frame of the image. In addition, the annotation process also requires a high level of expertise to ensure the accuracy and consistency of the data. Making full use of annotated data for supervision can not only improve the prediction accuracy of the model, but also enhance the robustness of the model in different scenarios and conditions. Therefore, how to make full use of these high-cost annotated data, especially the time information between the previous and next frames, to effectively supervise the entire prediction process is a very valuable and urgent problem to be solved. This will not only help improve the performance of existing models, but also provide a more solid foundation for the future development of autonomous driving technology. Summary of the invention

[0004] To solve the above problems, the present invention proposes an autonomous driving occupancy grid prediction method using multiple frames and rendering supervision. The method converts the prediction result of one frame to another frame through the labeled information and the positions of the vehicles in the previous and next two frames, and supervises the prediction of this frame through the labeled data of another frame. This method can ensure better prediction of occupancy in the autonomous driving scene space when the resolution of the labeled data is low.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for autonomous driving occupancy prediction using multiple frames and rendering supervision, comprising the following steps:

[0007] S1. Input the data of the main view image into the processing flow of the GaussianFormer model to obtain the predicted Gaussian point cloud of the main view semantics in the space; then adopt the processing flow of the GaussianFormer model, use the main view space occupancy grid prediction and the main view space occupancy grid annotation, and obtain the main view occupancy grid prediction loss;

[0008] S2, the main perspective semantic Gaussian point cloud is annotated with a three-dimensional dynamic object, and converted into an auxiliary perspective semantic Gaussian point cloud through a point cloud conversion process;

[0009] S3, using the processing flow of the GaussianFormer model, using the auxiliary view space occupancy grid prediction and the auxiliary view space occupancy grid annotation, to obtain the auxiliary view occupancy grid prediction loss;

[0010] S4, using an image semantic extraction network to extract semantic information from the main view image and the auxiliary view image, respectively, to obtain the main view semantic intent and the auxiliary view semantic intent;

[0011] S5. By semantic rendering, the main view semantic Gaussian point cloud and the auxiliary view semantic Gaussian point cloud are rendered at the positions of the main view image and the auxiliary view image respectively to obtain the main view semantic rendering image and the auxiliary view semantic rendering image;

[0012] S6. Combine the main perspective semantic intent and the main perspective semantic rendering image with the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering image to obtain the semantic rendering loss.

[0013] Preferably, the data in step S1 are multi-perspective, multi-frame images and their camera parameters at two adjacent moments of the autonomous driving scene.

[0014] Preferably, the specific process of step S2 is:

[0015] S21, using the three-dimensional dynamic object annotation to determine whether a point in the main view semantic Gaussian point cloud is within the annotation;

[0016] S22. Transform the position and rotation quadruple of the Gaussian point cloud of the point located within the annotation to obtain an auxiliary perspective semantic Gaussian point cloud.

[0017] Preferably, the three-dimensional dynamic object annotation in step S2 includes the position and category of each occupied grid.

[0018] Preferably, the specific process of step S6 is: calculating L2Loss between the main perspective semantic intent and the main perspective semantic rendering graph to obtain the main perspective semantic rendering loss; calculating L2Loss between the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering graph to obtain the auxiliary perspective semantic rendering loss; and then adding the main perspective semantic rendering loss and the auxiliary perspective semantic rendering loss to obtain the global semantic rendering loss.

[0019] After adopting the above technical solution, the present invention has the following beneficial effects: the present invention converts the prediction result of one frame to another frame through the annotated information and the position of the vehicle in the previous and next two frames, and supervises the prediction of this frame through the annotated data of another frame. This method can ensure that when the resolution of the annotated data is low, there is a better prediction of the occupancy in the autonomous driving scene space. At the same time, the combination of 2D images and three-dimensional space can improve the robustness and generalization ability of the model. Through multi-view 2D image data, the details and changes in the scene can be more comprehensively captured, thereby improving the accuracy of three-dimensional reconstruction and object detection; in autonomous driving perception, using 2D information to supervise the three-dimensional space can effectively improve the performance and reliability of the model. This method provides the autonomous driving system with a more comprehensive and accurate environmental perception capability, thereby improving the safety and stability of driving. Therefore, in the form of the GaussianFormer model, the present invention uses the predicted Gaussian points to render the semantics of a certain perspective, and compares the rendering result with the original 2D semantic prediction to ensure that the Gaussian points of the semantics in the predicted space are fine-grained. Supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of the present invention;

[0021] Figure 2 It is a schematic diagram of point cloud conversion of the present invention;

[0022] Figure 3 It is a schematic diagram of the process framework of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] like Figures 1 to 3 As shown, a method for autonomous driving occupancy prediction using multiple frames and rendering supervision includes the following steps:

[0025] S1. Input the data of the main view image into the processing flow of the GaussianFormer model to obtain the predicted Gaussian point cloud of the main view semantics in the space; then adopt the processing flow of the GaussianFormer model, use the main view space occupancy grid prediction and the main view space occupancy grid annotation, and obtain the main view occupancy grid prediction loss;

[0026] The data in step S1 are multi-view, multi-frame images and their camera parameters at two adjacent moments of the autonomous driving scene;

[0027] S2, the main perspective semantic Gaussian point cloud is annotated with a three-dimensional dynamic object, and converted into an auxiliary perspective semantic Gaussian point cloud through a point cloud conversion process;

[0028] The specific process of step S2 is:

[0029] S21, using the three-dimensional dynamic object annotation to determine whether a point in the main view semantic Gaussian point cloud is within the annotation;

[0030] S22, transforming the position and rotation quadruple of the Gaussian point cloud of the point located in the annotation to obtain the auxiliary perspective semantic Gaussian point cloud;

[0031] The three-dimensional dynamic object annotation in step S2 includes the position and category of each occupied grid;

[0032] S3, using the processing flow of the GaussianFormer model, using the auxiliary view space occupancy grid prediction and the auxiliary view space occupancy grid annotation, to obtain the auxiliary view occupancy grid prediction loss;

[0033] S4, using an image semantic extraction network to extract semantic information from the main view image and the auxiliary view image, respectively, to obtain the main view semantic intent and the auxiliary view semantic intent;

[0034] S5. By semantic rendering, the main view semantic Gaussian point cloud and the auxiliary view semantic Gaussian point cloud are rendered at the positions of the main view image and the auxiliary view image respectively to obtain the main view semantic rendering image and the auxiliary view semantic rendering image;

[0035] S6, jointly using the main perspective language intent and the main perspective semantic rendering graph as well as the auxiliary perspective language intent and the auxiliary perspective semantic rendering graph to obtain the semantic rendering loss;

[0036] The specific process of step S6 is: calculating L2 Loss between the main perspective semantic intent and the main perspective semantic rendering graph to obtain the main perspective semantic rendering loss; calculating L2 Loss between the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering graph to obtain the auxiliary perspective semantic rendering loss; and then adding the main perspective semantic rendering loss and the auxiliary perspective semantic rendering loss to obtain the global semantic rendering loss.

[0037] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for autonomous driving occupancy prediction using multiple frames and rendering supervision, characterized in that: The following steps are involved: S1. Input the data of the main view image into the processing flow of the GaussianFormer model to obtain the predicted Gaussian point cloud of the main view semantics in the space; then adopt the processing flow of the GaussianFormer model, use the main view space occupancy grid prediction and the main view space occupancy grid annotation, and obtain the main view occupancy grid prediction loss; S2, the main perspective semantic Gaussian point cloud is annotated with a three-dimensional dynamic object, and converted into an auxiliary perspective semantic Gaussian point cloud through a point cloud conversion process; S3, using the processing flow of the GaussianFormer model, using the auxiliary view space occupancy grid prediction and the auxiliary view space occupancy grid annotation, to obtain the auxiliary view occupancy grid prediction loss; S4, using an image semantic extraction network to extract semantic information from the main view image and the auxiliary view image, respectively, to obtain the main view semantic intent and the auxiliary view semantic intent; S5. By semantic rendering, the main view semantic Gaussian point cloud and the auxiliary view semantic Gaussian point cloud are rendered at the positions of the main view image and the auxiliary view image respectively to obtain the main view semantic rendering image and the auxiliary view semantic rendering image; S6. Combine the main perspective semantic intent and the main perspective semantic rendering image with the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering image to obtain the semantic rendering loss.

2. The method for autonomous driving occupancy prediction using multiple frames and rendering supervision as claimed in claim 1, characterized in that: The data in step S1 are multi-perspective, multi-frame images and their camera parameters at two adjacent moments of the autonomous driving scene.

3. The method for autonomous driving occupancy prediction using multiple frames and rendering supervision as claimed in claim 1, characterized in that: The specific process of step S2 is: S21, using the three-dimensional dynamic object annotation to determine whether a point in the main view semantic Gaussian point cloud is within the annotation; S22. Transform the position and rotation quadruple of the Gaussian point cloud of the point located within the annotation to obtain an auxiliary perspective semantic Gaussian point cloud.

4. The method for autonomous driving occupancy prediction using multiple frames and rendering supervision as claimed in claim 1, characterized in that: The three-dimensional dynamic object annotation in step S2 includes the position and category of each occupied grid.

5. The method for autonomous driving occupancy prediction using multiple frames and rendering supervision as claimed in claim 1, characterized in that: The specific process of step S6 is: calculating L2 Loss between the main perspective semantic intent and the main perspective semantic rendering graph to obtain the main perspective semantic rendering loss; calculating L2 Loss between the auxiliary perspective semantic intent and the auxiliary perspective semantic rendering graph to obtain the auxiliary perspective semantic rendering loss; Then add the main perspective semantic rendering loss and the auxiliary perspective semantic rendering loss to get the global semantic rendering loss.

Images