Three-dimensional Gaussian rendering method and device based on orbslam3 initialization

By adopting the initialization method of orbslam3 in three-dimensional reconstruction, the colmap method has solved the problem of long optimization time and sparse point cloud model when processing a large number of images, achieving more efficient three-dimensional reconstruction and better map output.

CN119027599APending Publication Date: 2024-11-26SOUTHERN UNIVERSITY OF SCIENCE AND TECHNOLOGY
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410966832.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing colmap-based three-dimensional reconstruction method has a long optimization time when processing a large number of images, and the output point cloud models are sparse and irregular, resulting in complex and time-consuming post-processing, affecting the efficiency of downstream tasks.

Method used

Using the initialization method based on orbslam3, the internal parameter matrix and distortion coefficient are obtained by calibration of the camera, image distortion correction is performed, feature points are extracted, maps are constructed and 3D Gaussian body are initialized, and the renderer is used to render and output, and the three-dimensional Gaussian body density is controlled. Optimize the three-dimensional reconstruction process.

Benefits of technology

It significantly reduces the optimization time of 3D reconstruction, improves the quality and density of 3D maps, simplifies the post-processing process, and improves the operating efficiency of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027599A_ABST
    Figure CN119027599A_ABST
Patent Text Reader

Abstract

The invention provides a three-dimensional Gaussian rendering method and device based on orbslam3 initialization, and the method comprises the steps: carrying out the processing of sensor data collection through an ROS system, enabling the sensor data collection to be used in an orbslam3 system, obtaining the camera pose and three-dimensional point cloud data for training three-dimensional Gaussian through orbslam3, carrying out the training of a three-dimensional Gaussian body in a scene through a three-dimensional Gaussian rendering training algorithm, and obtaining the three-dimensional point cloud data through the three-dimensional point cloud data; and rendering and outputting the three-dimensional Gaussian body in the scene through a renderer to obtain a final three-dimensional reconstruction result. Compared with the existing method for performing three-dimensional Gaussian rendering by using coomap, the method disclosed by the invention has the advantages that GPU resources can be reasonably utilized, the optimization time can be saved and higher three-dimensional reconstruction quality can be realized within reasonable time under the condition of not adding hardware equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a three-dimensional Gaussian rendering method and device based on orbslam3 initialization. Background Art

[0002] 3D reconstruction refers to the process of reconstructing a 3D model of an object from a single 2D image or multiple 2D images and performing texture mapping on the 3D model. With the development of society and science and technology, the demand for 3D reconstruction technology is increasing. 3D reconstruction can obtain 3D models with color textures observed from any perspective, and is also an important research direction in the field of computer vision. 3D Gaussian rendering is a scene reconstruction and rendering technology based on 3D Gaussian volumes. This technology uses Gaussian distribution to represent the point cloud data in the scene and converts these point cloud data into Gaussian volumes during the rendering process. The characteristics of the Gaussian volume enable it to efficiently simulate the propagation and scattering of light in the scene, thereby achieving efficient real-time rendering.

[0003] Existing technologies generally use the colmap method to generate an initial point cloud and calculate the accurate camera pose. The initial point cloud and the solved camera pose are then input into a 3D Gaussian rendering system for forward and back propagation operations of the 3D Gaussian rendering.

[0004] Although the 3D reconstruction method based on colmap has high reconstruction accuracy and good effect, it has the following two disadvantages:

[0005] 1) The optimization time of the colmap-based 3D reconstruction method increases exponentially with the number of image sequences, and the system cannot process a large number of image inputs within an acceptable time range.

[0006] 2) The final output map result of the colmap-based 3D reconstruction system is a point cloud model composed of many 3D point clouds. First, due to the sparsity and irregularity of point cloud data, post-processing (such as point cloud segmentation, denoising, surface reconstruction, etc.) becomes more complicated and time-consuming, which is not conducive to downstream tasks.

[0007] Therefore, the current optimization method for the three-dimensional Gaussian rendering method still needs to be further improved. Summary of the Invention

[0008] In view of this, the present invention proposes a three-dimensional Gaussian rendering method and device based on orbslam3 initialization, which solves the technical problem in the prior art that colmap initialization optimization takes a long time and thus leads to slow three-dimensional reconstruction speed.

[0009] The technical solution of the present invention is achieved as follows:

[0010] In one aspect, the present invention provides a three-dimensional Gaussian rendering method based on orbslam3 initialization, comprising the following steps:

[0011] Calibrate the camera to obtain the camera's intrinsic parameter matrix and distortion coefficients, and use the intrinsic parameter matrix and distortion coefficients to correct the distortion of the image captured by the camera, publish a new node to obtain the corrected image, and obtain image information;

[0012] Build a map, modify the configuration file, input the intrinsic parameter matrix and the distortion coefficient, extract new feature points, and obtain the 3D point cloud and camera pose of the map;

[0013] Perform three-dimensional Gaussian rendering, initialize the map, obtain a three-dimensional Gaussian body, control the density of the three-dimensional Gaussian body by copying or pruning, render and output the map using a renderer, and obtain a three-dimensional reconstruction result.

[0014] On the basis of this technical solution, it is further preferred that the constructing of the map, the changing of the configuration file, and the inputting of the corrected internal parameter matrix and the distortion coefficient specifically include the following steps:

[0015] Modify the file processing part, save the txt file of the camera pose and 3D point cloud, change the configuration file, pass in the corrected intrinsic parameter matrix and the distortion coefficient, and load the updated configuration file.

[0016] On the basis of this technical solution, it is further preferred that the step of loading and updating the configuration file further includes the following steps:

[0017] Through the receiving node of the default image, the image information is passed in, the script is started, all the passed in images are received, and a new folder is created to change the storage path of the image information.

[0018] On the basis of this technical solution, further preferably, initializing the three-dimensional Gaussian map further includes the following steps:

[0019] According to the configuration file of the camera, the txt file required for the operation of the three-dimensional Gaussian system is generated, and the storage path of the file is changed. The changed storage path contains all the images related to the object to be reconstructed, the txt file of the camera required for initialization, and the txt file of the 3D point cloud. These files are passed into the three-dimensional Gaussian system to train all the files.

[0020] On the basis of this technical solution, it is further preferred that the camera is calibrated to obtain the camera's intrinsic parameter matrix and distortion coefficient, specifically including the following steps:

[0021] Connect the camera, obtain the camera image, and push it to the original camera node, record the original camera's image stream, and save it in the corresponding original camera folder in 3D point cloud data storage format;

[0022] Until the image of the calibration plate appears, keep moving the camera to capture the image to ensure that the entire calibration plate can be displayed in the image and that all pixel areas are evenly covered. Then use the toolbox to calibrate the original camera folder to obtain the camera's intrinsic parameter matrix and distortion coefficients.

[0023] On the basis of this technical solution, it is further preferred that the image acquired by the camera is subjected to distortion correction by using the intrinsic parameter matrix and the distortion coefficient, and a new node is released to obtain the corrected image, which specifically includes the following steps:

[0024] Receive the video stream information of the original camera node through a new script, use the intrinsic parameter matrix and distortion coefficient to perform distortion correction on the image obtained by the original camera, and publish a new node to push the image corrected to the ideal pinhole camera model;

[0025] The node is used to move the camera around the reconstructed target, and image information is acquired at 30 frames per second to ensure that the captured images cover all viewing angles of the reconstructed target. The acquired video stream is recorded and saved.

[0026] On the basis of this technical solution, it is further preferred that three-dimensional Gaussian rendering, initializing a three-dimensional Gaussian map, obtaining a three-dimensional Gaussian volume, and controlling the density of the three-dimensional Gaussian volume by copying or pruning specifically include the following steps:

[0027] Initialize the three-dimensional Gaussian map, obtain the three-dimensional Gaussian body, and project the map composed of the three-dimensional Gaussian body according to the camera pose information contained in the folder, copy or prune the three-dimensional Gaussian body, and control the density of the three-dimensional Gaussian body.

[0028] On the basis of this technical solution, it is further preferred that a renderer is used to render and output the three-dimensional Gaussian map, which specifically includes the following steps:

[0029] By superimposing the rendering process of the three-dimensional Gaussian body, the predicted two-dimensional image in the current state is output. According to the loss function of the difference between the predicted two-dimensional image and the actual captured two-dimensional image, the three-dimensional Gaussian body is back-propagated and trained 30,000 times. Finally, a three-dimensional Gaussian map is obtained. The obtained three-dimensional Gaussian map is rendered and output using a renderer to obtain a three-dimensional reconstruction result.

[0030] On the basis of this technical solution, further preferably, the rendering process of the three-dimensional Gaussian volume is alpha rendering.

[0031] In a second aspect, the present invention further provides a three-dimensional Gaussian rendering device based on orbslam3 initialization, comprising the following:

[0032] The calibration module is used to calibrate the camera, obtain the camera's intrinsic parameter matrix and distortion coefficients, and perform distortion correction on the image obtained by the camera using the intrinsic parameter matrix and distortion coefficients, publish a new node to obtain the corrected image, and obtain image information;

[0033] An extraction module is used to construct a map, modify a configuration file, input the intrinsic parameter matrix and the distortion coefficient, extract new feature points, and obtain the 3D point cloud and camera pose of the map;

[0034] The rendering module performs three-dimensional Gaussian rendering, initializes the map, obtains a three-dimensional Gaussian volume, controls the density of the three-dimensional Gaussian volume by copying or pruning, and uses a renderer to render and output the map to obtain a three-dimensional reconstruction result.

[0035] In a third aspect, the present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the three-dimensional Gaussian rendering method as described in any one of the above.

[0036] The three-dimensional Gaussian rendering method based on orbslam3 initialization described in the present invention has the following beneficial effects compared with the prior art:

[0037] Compared with existing colmap-based 3D reconstruction methods, its optimization time increases exponentially with the number of image sequences, and the system cannot process a large number of image inputs within an acceptable time frame. At the same time, the point cloud model of the map is composed of many 3D point clouds. Due to the sparsity and irregularity of point cloud data, post-processing processes such as point cloud segmentation, denoising, and surface reconstruction become more complex and time-consuming, making it unsuitable for downstream tasks.

[0038] During operation, the orbslam3 system of the present invention first performs key frame extraction processing on the incoming image frames, excludes some frames that have little impact on reconstruction, and only retains frames that have a greater impact on system reconstruction, have a large number of observable spatial points, and have a certain amount of camera posture movement; the feature points used by the orbslam3 system are ORB feature points, which have a significant advantage in extraction speed compared to the SIFT feature points extracted by the colmap system.

[0039] In existing technologies, global BA optimization is a very time-consuming step in 3D reconstruction systems. The orbslam3 system only performs BA optimization on the global map once loop closure is detected, whereas the colmap system requires global BA optimization after the 3D map has grown a certain amount. This makes full use of GPU resources in the rendering system of the present invention. Training with camera poses and map point cloud information from the orbslam3 system takes only 5-10 minutes, resulting in a higher-quality 3D map than the point cloud, without significantly impacting system runtime. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of the three-dimensional Gaussian rendering method based on orbslam3 initialization according to the present invention;

[0042] Figure 2 Schematic diagram of a rendering device for the three-dimensional Gaussian rendering method based on orbslam3 initialization according to the present invention;

[0043] Figure 3 A schematic diagram of a computer-readable storage medium according to the present invention;

[0044] Figure 4 This is a software flow chart of the three-dimensional Gaussian rendering method based on orbslam3 initialization described in the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] Example 1

[0047] This embodiment provides a three-dimensional Gaussian rendering method based on orbslam3 initialization, the flow chart is as follows Figure 1 As shown, the following steps are included:

[0048] S1. Calibrate the camera to obtain the camera's intrinsic parameter matrix and distortion coefficients, perform distortion correction on the image acquired by the camera using the intrinsic parameter matrix and distortion coefficients, publish a new node to obtain the corrected image, and obtain image information;

[0049] S2. Build a map, modify the configuration file, input the intrinsic parameter matrix and the distortion coefficient, extract new feature points, and obtain the 3D point cloud and camera pose of the map;

[0050] S3. Perform three-dimensional Gaussian rendering to initialize the map, obtain a three-dimensional Gaussian volume, control the density of the three-dimensional Gaussian volume by copying or pruning, and use a renderer to render and output the map to obtain a three-dimensional reconstruction result.

[0051] Specifically, the construction of the map, modification of the configuration file, and input of the corrected internal parameter matrix and the distortion coefficient include the following steps:

[0052] Modify the file processing part, save the txt file of the camera pose and 3D point cloud, change the configuration file, pass in the corrected intrinsic parameter matrix and the distortion coefficient, and load the updated configuration file.

[0053] Specifically, loading and updating the configuration file also includes the following steps:

[0054] Through the receiving node of the default image, the image information is passed in, the script is started, all the passed in images are received, and a new folder is created to change the storage path of the image information.

[0055] Specifically, initializing the three-dimensional Gaussian map also includes the following steps:

[0056] According to the configuration file of the camera, the txt file required for the operation of the three-dimensional Gaussian system is generated, and the storage path of the file is changed. The changed storage path contains all the images related to the object to be reconstructed, the txt file of the camera required for initialization, and the txt file of the 3D point cloud. These files are passed into the three-dimensional Gaussian system to train all the files.

[0057] Specifically, the camera calibration is performed to obtain the camera's intrinsic parameter matrix and distortion coefficients, and specifically includes the following steps:

[0058] Connect the camera, obtain the camera image, and push it to the original camera node, record the original camera's image stream, and save it in the corresponding original camera folder in 3D point cloud data storage format;

[0059] Until the image of the calibration plate appears, keep moving the camera to capture the image to ensure that the entire calibration plate can be displayed in the image and that all pixel areas are evenly covered. Then use the toolbox to calibrate the original camera folder to obtain the camera's intrinsic parameter matrix and distortion coefficients.

[0060] Specifically, the image acquired by the camera is subjected to distortion correction using the intrinsic parameter matrix and the distortion coefficient, and a new node is released to obtain the corrected image, which specifically includes the following steps:

[0061] Receive the video stream information of the original camera node through a new script, use the intrinsic parameter matrix and distortion coefficient to perform distortion correction on the image obtained by the original camera, and publish a new node to push the image corrected to the ideal pinhole camera model;

[0062] The node is used to move the camera around the reconstructed target, and image information is acquired at 30 frames per second to ensure that the captured images cover all viewing angles of the reconstructed target. The acquired video stream is recorded and saved.

[0063] Specifically, the three-dimensional Gaussian rendering, initializing the three-dimensional Gaussian map, obtaining the three-dimensional Gaussian volume, and controlling the density of the three-dimensional Gaussian volume by copying or pruning, specifically includes the following steps:

[0064] Initialize the three-dimensional Gaussian map, obtain the three-dimensional Gaussian body, and project the map composed of the three-dimensional Gaussian body according to the camera pose information contained in the folder, copy or prune the three-dimensional Gaussian body, and control the density of the three-dimensional Gaussian body.

[0065] Specifically, the three-dimensional Gaussian map is rendered and output using a renderer, which specifically includes the following steps:

[0066] By superimposing the rendering process of the three-dimensional Gaussian body, the predicted two-dimensional image in the current state is output. According to the loss function of the difference between the predicted two-dimensional image and the actual captured two-dimensional image, the three-dimensional Gaussian body is back-propagated and trained 30,000 times. Finally, a three-dimensional Gaussian map is obtained. The obtained three-dimensional Gaussian map is rendered and output using a renderer to obtain a three-dimensional reconstruction result.

[0067] Specifically, the rendering process of the three-dimensional Gaussian volume is alpha rendering.

[0068] Example 2

[0069] This embodiment provides a three-dimensional Gaussian rendering method based on orbslam3 initialization, comprising the following steps:

[0070] S1. Use the ROS system to calibrate the camera, obtain the camera's intrinsic parameter matrix and distortion coefficients, and use the intrinsic parameter matrix and distortion coefficients to perform distortion correction on the image obtained by the camera, and publish a new ROS node to obtain the corrected image;

[0071] Specifically, this step includes the following:

[0072] Connect your own camera to the computer, obtain the camera image through the ROS system and push it to the / camera_raw node of ROS, use the rosbag record tool in the ROS system to record the image stream of / camera_raw and save it to the corresponding camera_raw.bag file of the ROS system; display a standard kalibr / AprilGrid calibration plate image on the computer screen, and capture the image with the calibration plate by continuously moving the camera to ensure that the entire calibration plate can be displayed in the image and that all pixel areas are evenly covered; then use the kalibr toolbox to calibrate the camera_raw.bag file to obtain all the intrinsic parameters and distortion coefficients of the camera;

[0073] A new ROS script receives the video stream from the / camera_raw node and uses the intrinsic parameters and distortion coefficients obtained in step 1 to correct the distortion of the image acquired through / camera_raw. A new ROS node, / camera_cali, is published to push images corrected to the ideal pinhole camera model. This node is used to move the camera around the reconstruction target at 30 frames per second to capture image information, ensuring that the captured images cover every viewpoint of the reconstruction target. The video stream is recorded using the rosbag record tool and saved as object.bag.

[0074] S2. Install orbslam3 and build a map, modify the configuration file of orbslam3, input the corrected intrinsic parameter matrix and the distortion coefficient, start modifying the orbslam3 monocular mode under ROS, extract new feature points, and obtain the 3D point cloud and camera pose of the map;

[0075] Specifically, this step includes the following:

[0076] Install and compile the ROS version of orbslam3, and save the images.txt file containing all camera poses and all locations in a format similar to colmap by modifying the orbslam3 file processing part. Figure 3 points3D.txt, change the configuration file in orbslam3, and pass in the internal parameters and distortion coefficients of your own camera.

[0077] Open the orbslam3 system through ROS, load the default bag-of-words model and the configuration file for the custom camera. Pass image information through the default image sink node / camera / image_raw in orbslam3, i.e., / camera_cali:= / camera / image_raw. Start a ROS script to receive all images passed in by / camera_cali, create a new folder object, and create two folders within it called images and sparse. Create a new folder 0 within the sparse folder. Save all images to the images folder. Wait for the rosbag file to finish playing and for the map and camera pose in ORBSLAM3 to be calculated. Generate the images.txt and points3D.txt files and save them to the 0 folder.

[0078] Furthermore, the format of the 0 folder is as follows:

[0079] <object>

[0080] |---images

[0081]

[0082] Generate the cameras.txt file required for running the 3D Gaussian system based on the camera configuration file in orbslam3 and place it in the / object / sparse / 0 folder. Transfer the prepared object folder, containing all the images of the object to be reconstructed, as well as the cameras.txt, images.txt, and points.txt files required for initialization, to the 3D Gaussian system. Train all files in this folder using Python code.

[0083] S3, 3D Gaussian rendering, initializing the 3D Gaussian map, obtaining a 3D Gaussian volume, controlling the density of the 3D Gaussian volume by copying or pruning, rendering and outputting the 3D Gaussian map using a renderer, and obtaining a 3D reconstruction result.

[0084] Specifically, this step includes the following:

[0085] After the map initialization and Gaussian initialization stages, the 3D Gaussian map is initialized, and the map composed of 3D Gaussian bodies is projected according to the camera pose information contained in the images.txt file. By judging the difficulty between the current 3D Gaussian body and the rendered image, the 3D Gaussian body is copied or pruned to adaptively control the 3D Gaussian body density.

[0086] For two-dimensional images, the alpha rendering process of superimposing a three-dimensional Gaussian body outputs the predicted two-dimensional image in the current state; based on the loss function that measures the difference between the predicted two-dimensional image and the actual captured two-dimensional image, all parameters related to the three-dimensional Gaussian body are back-propagated and trained, and finally iterated 30,000 times to obtain the final three-dimensional Gaussian map after training. The three-dimensional Gaussian map is rendered and output using a three-dimensional Gaussian map renderer to obtain the final three-dimensional reconstruction result.

[0087] Existing Colmap 3D reconstruction requires extensive bundle adjustment to optimize parameters such as camera pose and spatial points, which takes considerable time. This optimization process is particularly time-consuming during system expansion, when multiple global bundle adjustments are required. The sheer number of poses and spatial points that require optimization can be significant, leading to significant overall system time consumption.

[0088] For example, in robotic navigation and virtual reality interaction, point cloud models primarily focus on geometric structure, while texture and color information require additional processing and storage. Even if some point clouds contain color information, this information is sparse and incomplete, making it difficult to use for high-quality texture mapping and visual effects. Third, point cloud models typically consist of a large number of 3D points, resulting in a very large data volume. This places a burden on storage and transmission, especially when dealing with high-resolution and large-scale scenes.

[0089] In a preferred embodiment, a three-dimensional Gaussian rendering method based on orbslam3 initialization is also provided, comprising the following steps:

[0090] S1. Use the ROS system to calibrate the camera, obtain the camera's intrinsic parameter matrix and distortion coefficients, and use the intrinsic parameter matrix and distortion coefficients to perform distortion correction on the image obtained by the camera, and publish a new ROS node to obtain the corrected image;

[0091] S2. Install orbslam3 and build a map, modify the configuration file of orbslam3, input the corrected intrinsic parameter matrix and the distortion coefficient, start modifying the orbslam3 monocular mode under ROS, extract new feature points, and obtain the 3D point cloud and camera pose of the map;

[0092] S3, 3D Gaussian rendering, initializing the 3D Gaussian map, obtaining a 3D Gaussian volume, controlling the density of the 3D Gaussian volume by copying or pruning, rendering and outputting the 3D Gaussian map using a renderer, and obtaining a 3D reconstruction result.

[0093] The installation of orbslam3 and construction of a map, modification of the orbslam3 configuration file, and the input of the corrected internal parameter matrix and the distortion coefficient specifically include the following steps:

[0094] Install and compile the ROS version of orbslam3, modify the file processing part of orbslam3, save the txt files of the camera pose and 3D point cloud, change the configuration file of orbslam3, pass in the corrected intrinsic parameter matrix and the distortion coefficient, open the orbslam3 system through ROS, and load the updated configuration file.

[0095] Specifically, starting to modify the orbslam3 monocular mode under ROS includes the following steps:

[0096] Start the monocular initialization mode under ROS. After loop closure detection, only one global BA optimization is performed to optimize all camera poses and 3D point clouds.

[0097] The advantage of this approach is that existing colmap systems don't extract input image sequences, forcing the optimization process to optimize the entire input image set. This significantly increases the optimization time required for large numbers of images. This approach helps reduce optimization time and achieve higher-quality 3D reconstruction within a reasonable timeframe.

[0098] Specifically, opening the orbslam3 system through ROS and loading the updated configuration file also includes the following steps:

[0099] Open the orbslam3 system through ROS, load the orbslam3 default bag-of-words model and the configuration file corresponding to the self-used camera;

[0100] Through the receiving node of the default image of orbslam3, the image information is passed in, and a ROS script is started to receive all the passed in images. A new folder is created, and a new images folder and sparse folder are created under the folder. A new folder is created in the sparse folder, and all images are saved in the images folder.

[0101] Specifically, initializing the three-dimensional Gaussian map also includes the following steps:

[0102] Generate the camera txt file required for the operation of the 3D Gaussian system according to the camera configuration file in orbslam3, and put it into a newly created folder. The prepared folder contains all the images related to the object to be reconstructed, the txt file of the camera required for initialization, and the txt file of the 3D point cloud. Pass it into the 3D Gaussian system, and train all the files in the folder through Python code.

[0103] Specifically, the camera calibration using the ROS system to obtain the camera's intrinsic parameter matrix and distortion coefficients specifically includes the following steps:

[0104] Connect your own camera to the computer, obtain the camera image through the ROS system, and push it to the ROS original camera node. Use the tools in the ROS system to record the original camera's image stream and save it to the bag file corresponding to the original camera in the ROS system.

[0105] Display a calibration plate image on the computer screen, and continuously move the camera to capture the image with the calibration plate to ensure that the entire calibration plate can be displayed in the image and that all pixel areas are evenly covered. Then, use the toolbox to calibrate the bag file of the original camera to obtain the intrinsic parameter matrix and distortion coefficient of the camera.

[0106] Specifically, the image acquired by the camera is subjected to distortion correction using the intrinsic parameter matrix and the distortion coefficient, and a new ROS node is released to obtain the corrected image, which specifically includes the following steps:

[0107] Receive the video stream information of the original camera node through a new ROS script, use the intrinsic parameter matrix and distortion coefficient to perform distortion correction on the image obtained by the original camera, and publish a new ROS node to push the image corrected to the ideal pinhole camera model;

[0108] Use this node to move the camera around the reconstructed object, acquiring image information at 30 frames per second to ensure that the captured images cover all viewpoints of the reconstructed object. Use the tool to record the video stream and save it as object.bag.

[0109] Specifically, the three-dimensional Gaussian rendering, initializing the three-dimensional Gaussian map, obtaining the three-dimensional Gaussian volume, and controlling the density of the three-dimensional Gaussian volume by copying or pruning, specifically includes the following steps:

[0110] Initialize the three-dimensional Gaussian map, obtain the three-dimensional Gaussian body, and project the map composed of the three-dimensional Gaussian body according to the camera pose information contained in the images folder, copy or prune the three-dimensional Gaussian body, and control the density of the three-dimensional Gaussian body.

[0111] Specifically, the three-dimensional Gaussian map is rendered and output using a renderer, which specifically includes the following steps:

[0112] By superimposing the rendering process of the three-dimensional Gaussian body, the predicted two-dimensional image in the current state is output. According to the loss function of the difference between the predicted two-dimensional image and the actual captured two-dimensional image, the three-dimensional Gaussian body is back-propagated and trained 30,000 times. Finally, a three-dimensional Gaussian map is obtained. The obtained three-dimensional Gaussian map is rendered and output using a renderer to obtain a three-dimensional reconstruction result.

[0113] Specifically, the rendering process of the three-dimensional Gaussian volume is alpha rendering.

[0114] In a preferred embodiment, Figure 2 This embodiment further provides a three-dimensional Gaussian rendering device based on orbslam3 initialization, comprising:

[0115] The calibration module 10 is used to calibrate the camera, obtain the camera's intrinsic parameter matrix and distortion coefficients, perform distortion correction on the image acquired by the camera using the intrinsic parameter matrix and distortion coefficients, publish a new node to obtain the corrected image, and obtain image information;

[0116] Extraction module 20, used to build a map, modify the configuration file, input the intrinsic parameter matrix and the distortion coefficient, extract new feature points, and obtain the 3D point cloud and camera pose of the map;

[0117] The rendering module 30 performs 3D Gaussian rendering, initializes the map, obtains a 3D Gaussian volume, controls the density of the 3D Gaussian volume by copying or pruning, and renders and outputs the map using a renderer to obtain a 3D reconstruction result.

[0118] In addition, in a preferred embodiment, this embodiment also provides a computer-readable storage medium, such as Figure 3 As shown, the computer storage medium 400 has a computer program 41. When the computer program 41 is executed, it is used to implement the three-dimensional Gaussian rendering method based on orbslam3 initialization described above. It can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. The software flow chart is shown in FIG. Figure 4 shown.

[0119] As a computer-readable storage medium, it can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor executes the software programs, instructions, or modules stored in the memory to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0120] During operation, the present invention first performs key frame extraction on the incoming image frames, excludes some frames that have little impact on reconstruction, and leaves only frames that have a greater impact on system reconstruction, have more observable spatial points, and have a certain amount of camera posture movement; the feature points used by the orbslam3 system are ORB feature points, which have obvious advantages in extraction speed compared to the SIFT feature points extracted by the colmap system.

[0121] In summary, the present invention provides a 3D Gaussian rendering method based on orbslam3 initialization. It uses the ROS system to process sensor data acquisition for use in the orbslam3 system, and then uses orbslam3 to obtain camera poses and 3D point cloud data for training 3D Gaussian. It can achieve higher quality 3D reconstruction in a reasonable time without adding hardware equipment.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / object>

Claims

1. A three-dimensional Gaussian rendering method based on orbslam3 initialization, characterized in that: The steps include: Calibrate the camera, obtain the camera's intrinsic parameter matrix and distortion coefficient, and use the intrinsic parameter matrix and distortion coefficient to correct the distortion of the image acquired by the camera, publish a new node to obtain the corrected image, and obtain image information. Build a map, change the configuration file, input the intrinsic parameter matrix and the distortion coefficient, extract new feature points, and obtain the 3D point cloud and camera pose of the map; Perform three-dimensional Gaussian rendering, initialize the map, obtain a three-dimensional Gaussian body, control the density of the three-dimensional Gaussian body by copying or pruning, render and output the map using a renderer, and obtain a three-dimensional reconstruction result.

2. The three-dimensional Gaussian rendering method based on orbslam3 initialization as claimed in claim 1, characterized in that: The construction of the map, the modification of the configuration file, and the input of the corrected internal parameter matrix and the distortion coefficient specifically include the following steps: Modify the file processing part, save the txt file of the camera pose and 3D point cloud, change the configuration file, pass in the corrected intrinsic parameter matrix and the distortion coefficient, and load and update the configuration file.

3. The three-dimensional Gaussian rendering method based on orbslam3 initialization as claimed in claim 2, characterized in that: Loading the update configuration file also includes the following steps: Through the receiving node of the default image, the image information is passed in, the script is started, all the passed in images are received, and a new folder is created to change the storage path of the image information.

4. The three-dimensional Gaussian rendering method based on orbslam3 initialization as claimed in claim 1, characterized in that: Initializing the 3D Gaussian map also includes the following steps: According to the configuration file of the camera, the txt file required for the operation of the three-dimensional Gaussian system is generated, and the storage path of the file is changed. The changed storage path contains all the images related to the object to be reconstructed, the txt file of the camera required for initialization, and the txt file of the 3D point cloud, which are passed into the three-dimensional Gaussian system to train all files.

5. The three-dimensional Gaussian rendering method based on orbslam3 initialization as claimed in claim 1, characterized in that: The camera is calibrated to obtain the camera's intrinsic parameter matrix and distortion coefficient, specifically including the following steps: Connect the camera, obtain the camera image, and push it to the node of the original camera, record the image stream of the original camera, and save it in the corresponding folder of the original camera in the 3D point cloud data storage format; Until the image of the calibration plate appears, keep moving the camera to capture the image to ensure that the entire calibration plate can be displayed in the image and that all pixel areas are evenly covered. Then use the toolbox to calibrate the folder of the original camera to obtain the intrinsic parameter matrix and distortion coefficient of the camera.

6. The three-dimensional Gaussian rendering method based on orbslam3 initialization as claimed in claim 5, characterized in that: The image acquired by the camera is subjected to distortion correction by using the internal parameter matrix and the distortion coefficient, and a new node is released to obtain the corrected image, which specifically includes the following steps: Receive the video stream information of the node of the original camera through a new script, use the internal parameter matrix and the distortion coefficient to perform distortion correction on the image acquired by the original camera, and publish a new node to push the image corrected to the ideal pinhole camera model; The node is used to move the camera around the reconstructed target, and image information is acquired at 30 frames per second to ensure that the captured images cover all viewing angles of the reconstructed target. The acquired video stream is recorded and saved.

7. The three-dimensional Gaussian rendering method based on orbslam3 initialization as claimed in claim 5, characterized in that: Performing three-dimensional Gaussian rendering, initializing the map, obtaining a three-dimensional Gaussian body, and controlling the density of the three-dimensional Gaussian body by copying or pruning, specifically includes the following steps: Initialize the three-dimensional Gaussian map, obtain the three-dimensional Gaussian body, and project the map composed of the three-dimensional Gaussian body according to the camera posture information contained in the folder, copy or prune the three-dimensional Gaussian body, and control the density of the three-dimensional Gaussian body.

8. The three-dimensional Gaussian rendering method based on orbslam3 initialization as claimed in claim 1, characterized in that: The three-dimensional Gaussian map is rendered and output using a renderer, which specifically includes the following steps: By superimposing the rendering process of the three-dimensional Gaussian body, the predicted two-dimensional image in the current state is output, and the three-dimensional Gaussian body is back-propagated and trained according to the loss function of the difference between the predicted two-dimensional image and the actual two-dimensional image. The three-dimensional Gaussian body is iterated 30,000 times to finally obtain a three-dimensional Gaussian map. The obtained three-dimensional Gaussian map is rendered and output by a renderer to obtain a three-dimensional reconstruction result.

9. A three-dimensional Gaussian rendering device based on orbslam3 initialization, characterized in that: These include: The calibration module is used to calibrate the camera, obtain the camera's internal parameter matrix and distortion coefficient, and perform distortion correction on the image obtained by the camera through the internal parameter matrix and distortion coefficient, publish a new node to obtain the corrected image, and obtain image information; An extraction module is used to construct a map, change a configuration file, input the intrinsic parameter matrix and the distortion coefficient, extract new feature points, and obtain the 3D point cloud and camera pose of the map; The rendering module performs three-dimensional Gaussian rendering, initializes the map, obtains a three-dimensional Gaussian body, controls the density of the three-dimensional Gaussian body by copying or pruning, renders and outputs the map using a renderer, and obtains a three-dimensional reconstruction result.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the three-dimensional Gaussian rendering method according to any one of claims 1-8.

Citation Information

Cited By

  • Industrial product three-dimensional diagram rendering method and system based on virtual reality

    CN119693520A

  • Method and system for rendering three-dimensional images of industrial products based on virtual reality

    CN119693520B