Self-adaptive single object three-dimensional reconstruction and image point cloud synthesis method for automatic driving scene

Abstract

The invention discloses an automatic driving scene adaptive single object three-dimensional reconstruction and image point cloud synthesis method, and relates to the technical field of automatic driving scene data production. The method comprises the following steps: respectively projecting each vertex of a box of a target object in a three-dimensional space to each frame, then carrying out image segmentation, setting the opaqueness of a part except the segmented object to be 0, taking the part as training input, and supervising the object by using the opaqueness; rendering is carried out in combination with the parameters of Scaffold-GS and the adaptive parameters of the camera to obtain a rendered image, and training loss is calculated to update the Gaussian point cloud; the camera poses of the first frames of the training scene and the target scene are aligned to determine the placement position of the target object, then the target object is rendered to obtain the multi-view object image, and the multi-view object image can be directly pasted with the target scene image. According to the method, the single object three-dimensional representation without edge blur is generated, and the object is rendered in the target scene.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving scene data production, and in particular to an autonomous driving scene adaptive single-object three-dimensional reconstruction and image point cloud synthesis method. Background Art

[0002] In the field of autonomous driving, object detection is a key task that aims to identify and locate objects of predefined categories in the environment. Existing detection methods for autonomous driving scenarios (such as BEVFormer, OccFormer, TPVFormer, etc.) rely on a large amount of data. For many boundary cases, there is usually not enough data for training, which leads to poor recognition and detection of these boundary cases, such as cones, triangles, etc. Therefore, how to synthesize data for these boundary cases that is close to real pictures is a very valuable problem.

[0003] At present, a better method is to map a picture of a single object onto a new image to identify it. However, this method is only a process from two dimensions to two dimensions, so it is not continuous for multiple frames of data and it is difficult to ensure the consistency of the map in three-dimensional space. At the same time, the current perception method of autonomous driving uses point clouds as input, so it is necessary not only to render and synthesize RGB images of new scenes and new perspectives, but also to synthesize a point cloud that conforms to the actual scan to input into the perception network. Therefore, using only maps cannot meet this requirement.

[0004] In order to solve the point cloud input problem, a three-dimensional reconstruction method can be used to reconstruct the object and project the reconstructed three-dimensional object into two dimensions. This method can not only ensure the consistency of the positions of multiple frames in three-dimensional space, but also synthesize a point cloud in three-dimensional space. This method can be very conveniently completed when using 3D-GS. 3D-GS explicitly uses a set of Gaussian ellipsoids to model the scene, including information such as position, color, and opacity. 3D-GS is suitable for processing large-scale point cloud data, such as terrain reconstruction and three-dimensional model generation. However, the use of 3D-GS is not very good for synthesizing the real point cloud of a single object. At the same time, in the process of reconstructing a single object, it is usually necessary to use the mask of the object as a guide to make the background transparent so as not to affect the rendering of the object. However, this step often requires a lot of manual annotation.

[0005] In addition, in autonomous driving scenarios, since the time between cameras cannot be accurately synchronized, the slight errors in the multi-camera poses obtained by existing methods are easily magnified in the rendering process, resulting in poor reconstruction effects. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for adaptive single-object three-dimensional reconstruction and image point cloud synthesis in an autonomous driving scene, which uses the box of the point cloud in the three-dimensional space as an annotation and opacity as a guide to gradually obtain better segmentation of the object during the reconstruction process, thereby generating a single-object three-dimensional representation without edge blur, thereby realizing the rendering of the object in the RGB image of the target scene.

[0007] The present invention provides a method for adaptive single-object 3D reconstruction and image point cloud synthesis in an autonomous driving scene, comprising:

[0008] Gaussian point cloud reconstruction process: obtain multi-frame multi-view images of visible target objects in the autonomous driving scene and their camera parameters; project each vertex of the box of the target object in three-dimensional space onto each frame, and then perform image segmentation to obtain the first segmentation result, set the opacity of the part outside the segmented object to 0, and obtain the first training image; use the first training image as the input of the training, and use opacity to supervise the object;

[0009] Initialize the point cloud obtained by Scaffold-GS through a certain frame to obtain the original Gaussian point cloud, and render it in combination with the parameters of Scaffold-GS and the adaptive parameters of the camera to obtain a rendered image; calculate the training loss based on the rendered image and the first training image, update the Gaussian point cloud, and obtain the trained Gaussian point cloud after training;

[0010] Rendering process: Align the camera poses of the first frame of the training scene and the target scene to determine the position of the target object, and then render the target object to obtain a multi-view object image, which can be directly pasted with the target scene image to achieve the rendering of the object in the target scene image.

[0011] Furthermore, the Gaussian point cloud reconstruction process also includes a training data update process: using the rendered image at stage t according to a set interval, selecting a specified number of pixels with opacity greater than a certain threshold as prompts, using the prompts to perform segmentation on the initial multi-view image, obtaining a second segmentation result, setting the opacity of the part outside the segmented object to 0, and obtaining a second training image for updating the first training image at stage t+1.

[0012] Furthermore, the training data update process specifically adopts the following formula:

[0013]

[0014] in, Represents the coordinates of pixels in the rendered image at stage t whose opacity is greater than the threshold. represents the original input image, Segment represents image segmentation, (u, v) represents the coordinates in terms of pixel length and width, and α represents opacity. Represents the rendered image, α thresh Represents the opacity threshold.

[0015] Furthermore, during the Gaussian point cloud reconstruction process, the parameters of Scaffold-GS and the adaptive parameters of the camera are combined for rendering to obtain a rendered image, specifically:

[0016] I out = Render(G′)

[0017] G′=transform(norm(θ(C i )+I4), G)

[0018] Among them, I out is the rendered image, Render is the process of Gaussian point cloud rendering, G′ is the transformed Gaussian point cloud, G is the original Gaussian point cloud, transform is the transformation process of the point cloud, norm is the matrix normalization process, I4 is the unit matrix with a dimension of 4, Θ(C i ) is the adaptive parameter of the camera, C i The camera ID.

[0019] Furthermore, during the rendering process, the camera poses of the first frames of the training scene and the target scene are aligned, specifically using the following formula:

[0020]

[0021] p target =T trans ×((t train0 -p train )

[0022] Among them, p target is the three-dimensional position of the object in the target scene, K train0 is the rotation matrix of the camera pose in the first frame of the training scene, Ktarget0 is the rotation matrix of the first frame of the target scene, T trans is the rotation matrix of the rendered object, t train0 is the translation matrix of the camera pose in the first frame of the training scene, P train are the 3D coordinates of the objects in the training scene.

[0023] Furthermore, after obtaining the multi-view object image, the xyz of the Gaussian point cloud is directly converted according to the relative P targetThe position of the object is directly pasted into the point cloud of the target scene to achieve the placement of a new object in both the image and the point cloud.

[0024] Preferably, efficientSAM is used to segment the image.

[0025] The technical solution provided in the embodiments of the present invention has at least the following technical effects:

[0026] In the process of Gaussian point cloud reconstruction, the box of the point cloud in the three-dimensional space is used as an annotation, and opacity is used as a guide. In the process of reconstruction, a better segmentation of the object is gradually obtained, thereby generating a single object three-dimensional representation without edge blur, and realizing the synthesis of the real point cloud of the specified single object without manual annotation. In the rendering process, adaptive adjustment of the camera's posture is added to ensure that the reconstructed single object can remain consistent under multiple camera perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0028] Figure 1 It is a schematic diagram of the overall process of the method of the embodiment of the present invention;

[0029] Figure 2 A flowchart of a training process of a method according to an embodiment of the present invention;

[0030] Figure 3 The figure is a flowchart of the rendering process of the method according to the embodiment of the present invention. DETAILED DESCRIPTION

[0031] The embodiment of the present invention provides an adaptive single-object 3D reconstruction and image point cloud synthesis method for an autonomous driving scene, uses the box of the point cloud in the three-dimensional space as a annotation, uses opacity as a guide, and gradually obtains better segmentation of the object during the reconstruction process, thereby generating a single-object 3D representation without edge blur, and realizing the rendering of the object in the RGB image of the target scene.

[0032] Before introducing the embodiments of the present invention, the existing three-dimensional reconstruction technology used in the embodiments of the present invention is first introduced. 3D Gaussian Splatting (3D-GS) is a three-dimensional reconstruction method based on point cloud data, which uses Gaussian distribution to reconstruct the surface. Unlike NeRF (Neural Radiance Field), 3D-GS explicitly uses a set of Gaussian ellipsoids to model the scene, which contains information such as position, color, and opacity. 3D-GS is suitable for processing large-scale point cloud data, such as terrain reconstruction and three-dimensional model generation. Scaffold-GS is a variant of 3D-GS designed to improve rendering speed and memory efficiency. It uses the underlying scene structure to help prune over-extended Gaussian spheres, so that it is memory efficient while maintaining rendering quality and speed. Scaffold-GS has potential in real-time rendering and large-scale scene reconstruction.

[0033] In addition, Segment Anything Model (SAM) is a state-of-the-art image segmentation model that aims to change the field of computer vision. Based on the foundational model that has had a significant impact on natural language processing (NLP), SAM focuses on the prompt segmentation task and uses prompt engineering to adapt to different downstream segmentation problems. SAM contains an image encoder, a prompt encoder, and a mask decoder to generate accurate segmentation masks. SAM can accurately segment and recognize any object and has strong zero-sample generalization capabilities. It uses prompts as input to adapt to different segmentation tasks.

[0034] The technical solution in the embodiment of the present invention has the following general idea:

[0035] In order to synthesize realistic RGB images, it is usually necessary to increase the number of point clouds. Therefore, when using the form of Scaffold-GS, the present invention uses anchor points as the final synthesized point cloud, and can also use the features of the anchor points to render very high-quality RGB images. The anchor points are built-in in scaffold-gs. In the process of synthesizing the final point cloud, the anchor points of scaffoldgs are directly used to obtain the point cloud in space. The method of the present invention only requires the box of the point cloud in the three-dimensional space as an annotation, and uses opacity as a guide to gradually obtain a better segmentation of the object during the reconstruction process, thereby generating a single object three-dimensional representation without edge blur. In addition, in the autonomous driving scenario, since the time between cameras cannot be accurately synchronized, the present invention adds adaptive adjustment of the camera's posture in the process of synthesizing a single object to ensure that the reconstructed single object can also maintain consistency under multiple camera perspectives.

[0036] By projecting each vertex of the box of the target object in three-dimensional space onto each frame, and then performing image segmentation, the point processing of the target single object can be realized, the opacity of the part outside the segmented object is set to 0 as the input of training, and the opacity is used to supervise the object, saving the cost of manual labeling; rendering is performed in combination with the parameters of Scaffold-GS and the adaptive parameters of the camera to obtain a rendered image, and the training loss is calculated to update the Gaussian point cloud; the camera poses of the first frame of the training scene and the target scene are aligned to determine the position of the target object, and then the target object is rendered to obtain a multi-view object image, which can be directly pasted with the target scene image. The present invention generates a three-dimensional representation of a single object without edge blur, and realizes the rendering of the object in the target scene.

[0037] This embodiment provides a method for adaptive single-object 3D reconstruction and image point cloud synthesis in an autonomous driving scene. Figure 1 As shown, it includes: Gaussian point cloud reconstruction process and rendering process;

[0038] Gaussian point cloud reconstruction process Figure 2 As shown, in this embodiment, the Gaussian point cloud reconstruction process includes a training process and a training data update process;

[0039] Training process: obtain multi-frame multi-view images of visible target objects in the autonomous driving scene and their camera parameters; project each vertex of the box of the target object in three-dimensional space (expressed as (x, y, z, w, h, l), where x, y, z are the center coordinates of the box, and w, h, l represent the width, height and length of the box) onto each frame, and then perform image segmentation (for example, efficient SAM can be used) to obtain the first segmentation result, set the opacity of the part outside the segmented object to 0, and obtain the first training image (RGBA image); use the first training image as the input of the training, and use opacity to supervise the object;

[0040] Initialize the point cloud obtained by Scaffold-GS through a certain frame to obtain the original Gaussian point cloud (the training data itself has point clouds, and the effect of Scaffold-GS will be better after initialization with point clouds), and render it with the parameters of Scaffold-GS and the adaptive parameters of the camera to obtain a rendered image; calculate the training loss based on the rendered image and the first training image, update the Gaussian point cloud, and complete the training to obtain the trained Gaussian point cloud.

[0041] The rendering is performed by combining the parameters of Scaffold-GS and the adaptive parameters of the camera to obtain a rendered image, specifically:

[0042] Iout = Render(G′)

[0043] G'=transform(norm(Θ(C i )+I4),G)

[0044] Among them, I out is the rendered image, Render is the process of Gaussian point cloud rendering, G′ is the transformed Gaussian point cloud, G is the original Gaussian point cloud, transform is the transformation process of the point cloud, norm is the matrix normalization process, I4 is the unit matrix with a dimension of 4, Θ(C i ) is the adaptive parameter of the camera, C i is the camera number. The conversion process is for each position (x, y, z) of the Gaussian point cloud T Expand to (x,y,z,1) T , and use the transformation matrix to perform matrix multiplication with the position of each point cloud.

[0045] The training data updating process is used to segment the training image according to the opacity and then update the training image, including: using the rendered image in the t stage according to the set interval, randomly selecting a specified number of pixels with opacity greater than a certain threshold as prompts, using the prompts to segment the initial multi-view image (efficientSAM can also be used), obtaining a second segmentation result, setting the opacity of the part outside the segmented object to 0, obtaining a second training image, and using it to update the first training image in the t+1 stage.

[0046] The training data update process specifically uses the following formula:

[0047]

[0048] in, Represents the coordinates of pixels in the rendered image at stage t whose opacity is greater than the threshold. represents the original input image, Segment represents image segmentation, (u,v) represents the coordinates in terms of pixel length and width, and α represents opacity. Represents the rendered image, α thresh Represents the opacity threshold.

[0049] The interval between each training data update process can be set to train 2000 iterations.

[0050] The training data update process extracts hints from the rendered images to segment the initial images, which saves labor costs and improves training results.

[0051] The rendering process is as follows Figure 3As shown: the camera poses of the first frame of the training scene and the target scene are aligned to determine the position of the target object, and then the target object is rendered to obtain a multi-view object image, which can be directly pasted with the target scene image to achieve the rendering of the object in the target scene image.

[0052] Among them, the camera poses of the first frame of the training scene and the target scene are aligned, and the specific formula is as follows:

[0053]

[0054]

[0055] Among them, P target is the three-dimensional position of the object in the target scene, K train0 is the rotation matrix of the camera pose in the first frame of the training scene, K target0 is the rotation matrix of the first frame of the target scene, T trans is the rotation matrix of the rendered object, t train0 is the translation matrix of the camera pose in the first frame of the training scene, P train are the 3D coordinates of the objects in the training scene.

[0056] Finally, after obtaining the multi-view object image, the xyz of the Gaussian point cloud can be directly converted according to the relative P target The position of the object is directly pasted into the point cloud of the target scene to achieve the placement of a new object in both the image and the point cloud.

[0057] The present invention utilizes the box of the point cloud in the three-dimensional space as a label and opacity as a guide in the process of Gaussian point cloud reconstruction, gradually obtains better segmentation of the object in the process of reconstruction, thereby generating a three-dimensional representation of a single object without edge blur, and realizes the synthesis of the real point cloud of a specified single object without manual labeling; in the rendering process, adaptive adjustment of the camera's posture is added to ensure that the reconstructed single object can remain consistent under multiple camera perspectives.

[0058] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0060] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0062] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for adaptive single-object 3D reconstruction and image point cloud synthesis in an autonomous driving scene, characterized in that: include: Gaussian point cloud reconstruction process: obtain multi-frame multi-view images and camera parameters of visible target objects in the autonomous driving scene; Project each vertex of the box of the target object in the three-dimensional space onto each frame, and then perform image segmentation to obtain the first segmentation result. Set the opacity of the part outside the segmented object to 0 to obtain the first training image. The first training image is used as training input, and the opacity is used to supervise the object; Initialize the point cloud obtained by Scaffold-GS through a certain frame to obtain the original Gaussian point cloud, and render it in combination with the parameters of Scaffold-GS and the adaptive parameters of the camera to obtain a rendered image; calculate the training loss based on the rendered image and the first training image, update the Gaussian point cloud, and obtain the trained Gaussian point cloud after training; Rendering process: Align the camera poses of the first frame of the training scene and the target scene to determine the position of the target object, and then render the target object to obtain a multi-view object image, which can be directly pasted with the target scene image to achieve the rendering of the object in the target scene image.

2. The method according to claim 1, characterized in that: The Gaussian point cloud reconstruction process also includes a training data update process: using the rendered image at stage t according to a set interval, selecting a specified number of pixels with opacity greater than a certain threshold as prompts, using the prompts to perform segmentation on the initial multi-view image to obtain a second segmentation result, setting the opacity of the part outside the segmented object to 0, and obtaining a second training image for updating the first training image at stage t+1.

3. The method according to claim 2, characterized in that: The training data update process specifically uses the following formula: in, Represents the coordinates of pixels in the rendered image at stage t whose opacity is greater than the threshold. represents the original input image, Segment represents image segmentation, (u,v) represents the coordinates in terms of pixel length and width, and α represents opacity. Represents the rendered image, α thresh Represents the opacity threshold.

4. The method according to claim 1 or 2, characterized in that: During the Gaussian point cloud reconstruction process, the parameters of Scaffold-GS and the adaptive parameters of the camera are combined for rendering to obtain a rendered image, specifically: I out =Render(G′) G′=transform(norm(Θ(C i )+I4),G) Among them, I out is the rendered image, Render is the process of Gaussian point cloud rendering, G′ is the transformed Gaussian point cloud, G is the original Gaussian point cloud, transform is the transformation process of the point cloud, norm is the matrix normalization process, I4 is the unit matrix with a dimension of 4, Θ(C i ) is the adaptive parameter of the camera, C i The camera ID.

5. The method according to claim 1, characterized in that: During the rendering process, the camera poses of the first frame of the training scene and the target scene are aligned, specifically using the following formula: Among them, p target is the three-dimensional position of the object in the target scene, K train0 is the rotation matrix of the camera pose in the first frame of the training scene (superscript -1 represents the inverse matrix), K target0 is the rotation matrix of the first frame of the target scene, T trans is the rotation matrix of the rendered object, t train0 is the translation matrix of the camera pose in the first frame of the training scene, P train are the 3D coordinates of the objects in the training scene.

6. The method according to claim 5, characterized in that: After obtaining the multi-view object image, directly convert the xyz of the Gaussian point cloud according to the relative p target The position of the object is directly pasted into the point cloud of the target scene to achieve the placement of a new object in both the image and the point cloud.

7. The method according to claim 1, characterized in that: Use efficient SAM to segment the image.

Images