Three-dimensional scene reconstruction and real-time positioning method and system based on 3D Gaussian spattering

Abstract

The invention provides a three-dimensional scene reconstruction and real-time positioning method and system.Three-dimensional scene data are collected through a monocular camera, a binocular camera or an RGB-D camera, feature point cloud is constructed through the ORB SLAM3 technology, rotation, zooming and other information are obtained, after data are collected, three-dimensional scene reconstruction is achieved through processing of the 3D Gaussian spattering technology, and real-time positioning of the three-dimensional scene is achieved. In the positioning process, feature points collected by ORB SLAM3 are compared, the optimal feature point position in the current scene is matched and fed back, and real-time positioning is achieved. The system supports high-simulation visual display of the local map and real-time positioning, is suitable for unmanned vehicle cruising, unmanned aerial vehicle inspection and intelligent robot positioning, and has the advantages of being high in practicability and wide in application range. And high-precision three-dimensional scene reconstruction and real-time positioning functions are provided.

Description

Technical Field

[0001] The present invention relates to a method and system for three-dimensional scene reconstruction and real-time positioning based on 3D Gaussian splashing, belonging to the technical field of three-dimensional scene reconstruction and real-time positioning. Background Art

[0002] With the rapid development of modern technology and the growing demand for intelligent applications, three-dimensional scene reconstruction and real-time positioning technology have become hot topics in research and industry. With the breakthroughs in machine learning, artificial intelligence, and computer vision technology, more accurate and efficient scene understanding capabilities are particularly critical for various automation systems. This paper proposes a new three-dimensional scene reconstruction and real-time positioning method based on 3D Gaussian Splatting technology and ORB SLAM3 algorithm, which realizes high-precision scene reconstruction and real-time feedback positioning system, and has broad practical application prospects.

[0003] Traditional 3D scene reconstruction methods often rely on complex hardware systems and limited sensor data processing capabilities, which limits their application scope and efficiency. With the advancement of monocular, binocular and RGB-D camera technology and the optimization of data processing algorithms, such as the development of ORB SLAM3 technology, we can extract useful information from large amounts of data more quickly and accurately, and achieve real-time environmental perception and decision-making response. The core of this study is to integrate advanced visual processing technology and mathematical modeling methods, and optimize the data processing process through the 3D Gaussian Splatting algorithm, so as to achieve higher reconstruction accuracy and positioning response speed.

[0004] The implementation of the present invention can not only improve the accuracy of scene reconstruction, but also greatly optimize the efficiency of real-time positioning, making it widely applicable to multiple fields. For example, in unmanned vehicles, accurate three-dimensional maps and real-time location updates can ensure safe and efficient navigation of vehicles in complex environments. In addition, when conducting inspections or search and rescue missions, drones rely on accurate three-dimensional environmental data and fast positioning technology to optimize flight paths and mission strategies. Intelligent robots also need accurate scene understanding capabilities to classify, move and store items in warehouse management and logistics operations.

[0005] The technical implementation of this system is based on several key steps: First, data is collected from the environment through various types of cameras, and ORB SLAM3 technology is used to construct a preliminary feature point cloud, while recording key rotation, scaling and other morphological information. Subsequently, the collected data is deeply processed using 3D Gaussian Splatting technology, which can effectively integrate point cloud data and transformation information to optimize the quality and details of the reconstructed 3D scene. After the scene reconstruction is completed, the system stores the map data locally to support real-time positioning and subsequent processing. In the positioning stage, the system quickly and accurately determines the target position by comparing the real-time collected feature points with the stored map data, and provides intuitive navigation and position feedback through visualization tools.

[0006] This paper proposes an efficient and reliable three-dimensional scene reconstruction and real-time positioning method, which significantly improves the operating efficiency and practicality of the system by integrating advanced visual processing technology and algorithm optimization. The development of this technology not only provides strong technical support for the advancement of machine vision and automation technology, but also opens up new application possibilities for the development of related industries.

[0007] The above problems are those that should be considered and solved in the process of 3D scene reconstruction and real-time positioning. Summary of the invention

[0008] The purpose of the present invention is to provide a three-dimensional scene reconstruction and real-time positioning method and system based on 3D Gaussian Splatting and ORB SLAM3 technology, aiming to solve the problems of low scene reconstruction accuracy and slow positioning response speed in the prior art. By integrating advanced image acquisition technology and precise data processing algorithms, the present invention can efficiently generate high-quality three-dimensional environment maps and realize real-time and accurate positioning of dynamic objects, thereby improving the navigation and operation capabilities of the automation system in complex environments.

[0009] The technical solution of the present invention is:

[0010] A method and system for 3D scene reconstruction and real-time positioning based on 3D Gaussian Splatting, comprising the following steps:

[0011] S1, collect data of three-dimensional scenes through monocular, binocular or RGB-D cameras;

[0012] S2. Apply ORB SLAM3 technology to construct feature point cloud of the collected data, and obtain additional information such as rotation, scaling, spherical harmonics, etc. in addition to the ORB feature point cloud;

[0013] S3. After completing data collection and analysis, the 3D Gaussian Splatting technology is used to process the data to achieve the reconstruction of the three-dimensional scene. After the three-dimensional scene is constructed, the map is stored locally;

[0014] S4. Use visualization tools to visualize local maps;

[0015] S5. In the real-time positioning process, the camera carried by the positioning subject collects real-time images, uses ORB SLAM3 technology to analyze, obtains ORB features and compares feature points, matches the position most similar to the feature points in the current scene, and provides feedback, thereby achieving real-time positioning of the object;

[0016] S6. Realize real-time positioning visualization based on local maps through visualization tools.

[0017] Furthermore, in step S2, ORB SLAM3 technology is applied to construct a feature point cloud for the collected data, and additional information such as rotation, scaling, spherical harmonics, etc. other than the ORB feature point cloud is obtained, specifically:

[0018] S21, processing the three-dimensional scene data collected by the monocular, binocular or RGB-D camera, extracting feature point information through ORBSLAM3 technology and saving it through point cloud;

[0019] S22. At the same time, by calculating the rotation matrix, scaling matrix and spherical harmonic function of each key frame starting from the initial frame, data is collected for the application of 3D Gaussian Splatting technology.

[0020] Furthermore, in step S3, after completing data collection and analysis, the data is processed using 3D Gaussian Splatting technology to achieve reconstruction of the three-dimensional scene. After the three-dimensional scene is constructed, the map is stored locally, specifically:

[0021] S31. Process the rotation matrix, scaling matrix and spherical harmonic function calculated from the data in the previous step by using the 3D Gaussian Splatting technique.

[0022] S32, calculate each point in the three-dimensional space point cloud with the rotation matrix, scaling matrix and spherical harmonic function to complete the reconstruction of the three-dimensional scene and save it.

[0023] Furthermore, in step S4, a visualization tool is used to realize a visualization display of the local map, specifically:

[0024] S41. Visualize the reconstructed 3D scene map by installing the Viewer visualization tool officially provided by 3D Gaussian Splatting.

[0025] Furthermore, in step S5, in the real-time positioning process, the real-time picture is collected by the camera carried by the positioning subject, and the ORB SLAM3 technology is used for analysis, the ORB features are obtained and the feature points are compared, the position most similar to the feature points in the current scene is matched, and feedback is given, so as to realize the real-time positioning of the object, specifically:

[0026] S51. When performing a real-time positioning task, it is necessary to collect data in real time through a camera carried by the positioning subject;

[0027] S52, using ORB SLAM3 technology to process the collected data, extract feature points, and perform feature point matching to find the camera position with the highest similarity, i.e., the target position;

[0028] S53, feedback the position and add it to the point cloud to achieve real-time positioning of the object.

[0029] Furthermore, in step S6, an incremental high-simulation display of real-time positioning based on a local map is realized through a visualization tool, specifically:

[0030] S61. Perform three-dimensional reconstruction of the positioning subject through ORB SLAM technology, build a point cloud model of the positioning subject and save it locally;

[0031] S62, combining the real-time positioning information added to the point cloud with the locally saved subject point cloud model, and displaying it through a visualization tool to achieve a high-simulation display of real-time positioning based on a local map.

[0032] The beneficial effects of the present invention are:

[0033] The method and system for 3D scene reconstruction and real-time positioning proposed in the present invention realize an advanced 3D scene reconstruction and real-time positioning by integrating 3D Gaussian Splatting and ORB SLAM3 technology. This system not only provides unparalleled accuracy and response speed, but also ensures a high degree of applicability and user-friendly operation experience. By finely reconstructing the 3D scene and matching environmental feature points in real time, the system provides accurate navigation data for automated equipment such as unmanned vehicles, drones and intelligent robots in various complex environments. In addition, the design of the system fully considers the convenience of operation and the reusability of data, greatly reduces the difficulty of operation, and provides rich data resources for future path planning and analysis. This invention not only optimizes the performance of the automation system, but also broadens its application field, bringing innovative solutions and significant progress to the industry, and has the following advantages:

[0034] Accurate 3D scene reconstruction: By using 3D Gaussian Splatting technology, the present invention can effectively integrate complex data collected from monocular, binocular or RGB-D cameras. This method not only improves the accuracy of point cloud generation, but also optimizes the geometric expression of the scene, so that the 3D map reaches a very high standard in terms of precision and realism. Accurate scene reconstruction provides reliable environmental data for various types of automation equipment, which is the key to improving their efficiency and safety in executing tasks.

[0035] Real-time positioning and rapid response: Combined with ORB SLAM3 technology, this system can identify and match feature points in the environment in a very short time, achieving fast and accurate positioning. This real-time feedback mechanism is crucial for application scenarios that require rapid decision-making and response (such as unmanned driving and emergency rescue operations), greatly improving the reliability and efficiency of operations.

[0036] Wide adaptability and flexibility: The system design takes into account various operating environments and equipment requirements, and can be widely used in unmanned vehicles, drones, intelligent robots and other platforms. This not only improves the market adaptability of the system, but also enables the invention to play an important role in complex and changeable practical application scenarios.

[0037] Ease of operation and user experience: By providing intuitive visualization tools and a user-friendly interface, operators can easily monitor and manage the 3D reconstruction and real-time positioning process. This design lowers the technical threshold, allowing non-professional users to effectively use the system to perform advanced tasks.

[0038] Data storage and reusability: After completing the construction of the three-dimensional scene, the present invention will store the data locally, which not only ensures the security of the data, but also supports subsequent data reuse and analysis. The stored high-precision map can be used for subsequent path planning, simulation training or further analysis, providing valuable resources for research and development. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a method and system for three-dimensional scene reconstruction and real-time positioning according to an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram illustrating a method for three-dimensional scene reconstruction according to an embodiment;

[0041] Figure 3 This is a schematic diagram of an embodiment of a method for real-time positioning function module.

[0042] Figure 4 This is a schematic diagram illustrating a method for real-time positioning of a high-simulation display function module in an embodiment. DETAILED DESCRIPTION

[0043] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary.

[0044] The embodiment provides a method and system for three-dimensional scene reconstruction and real-time positioning, such as Figure 1 , including the following steps:

[0045] S1, collect data of three-dimensional scenes through monocular, binocular or RGB-D cameras;

[0046] S2. Apply ORB SLAM3 technology to construct feature point cloud of the collected data, and obtain additional information such as rotation, scaling, spherical harmonics, etc. in addition to the ORB feature point cloud;

[0047] S3. After completing data collection and analysis, the 3D Gaussian Splatting technology is used to process the data to achieve the reconstruction of the three-dimensional scene. After the three-dimensional scene is constructed, the map is stored locally;

[0048] S4. Use visualization tools to visualize local maps;

[0049] S5. In the real-time positioning process, the camera carried by the positioning subject collects real-time images, uses ORB SLAM3 technology to analyze, obtains ORB features and compares feature points, matches the position most similar to the feature points in the current scene, and provides feedback, thereby achieving real-time positioning of the object;

[0050] S6. Through visualization tools, incremental high-simulation display based on real-time positioning on local maps is realized.

[0051] In step S2, ORB SLAM3 technology is applied to construct feature point clouds for the collected data, and additional information such as rotation, scaling, spherical harmonics, etc. is obtained in addition to the ORB feature point cloud. Figure 2 , specifically:

[0052] S21, processing the three-dimensional scene data collected by the monocular, binocular or RGB-D camera, extracting feature point information through ORBSLAM3 technology and saving it through point cloud;

[0053] S22. At the same time, by calculating the rotation matrix, scaling matrix and spherical harmonic function of each key frame starting from the initial frame, data is collected for the application of 3D Gaussian Splatting technology.

[0054] In step S3, after completing data collection and analysis, the data is processed using 3D Gaussian Splatting technology to achieve reconstruction of the three-dimensional scene. After the three-dimensional scene is constructed, the map is stored locally, such as Figure 2 , specifically:

[0055] S31. Process the rotation matrix, scaling matrix and spherical harmonic function calculated from the data in the previous step by using the 3D Gaussian Splatting technique.

[0056] S32, calculate each point in the three-dimensional space point cloud with the rotation matrix, scaling matrix and spherical harmonic function to complete the reconstruction of the three-dimensional scene and save it.

[0057] In step S5, during the real-time positioning process, the real-time image is collected by the camera carried by the positioning subject, and the ORBSLAM3 technology is used for analysis to obtain the ORB feature and perform feature point comparison, match the position most similar to the feature point in the current scene, and provide feedback, thereby achieving real-time positioning of the object, such as Figure 3 , specifically:

[0058] S51. When performing a real-time positioning task, it is necessary to collect data in real time through a camera carried by the positioning subject;

[0059] S52, using ORB SLAM3 technology to process the collected data, extract feature points, and perform feature point matching to find the camera position with the highest similarity, i.e., the target position;

[0060] S53, feedback the position and add it to the point cloud to achieve real-time positioning of the object.

[0061] In step S6, a visualization tool is used to realize an incremental high-simulation display of real-time positioning based on a local map, such as Figure 4 , specifically:

[0062] S61. Perform three-dimensional reconstruction of the positioning subject through ORB SLAM technology, build a point cloud model of the positioning subject and save it locally;

[0063] S62, combining the real-time positioning information added to the point cloud with the locally saved subject point cloud model, and displaying it through a visualization tool to achieve a high-simulation display of real-time positioning based on a local map.

[0064] The method and system for 3D scene reconstruction and real-time positioning proposed in the present invention mainly include the following aspects: ORB SLAM3 extracts ORB feature points and constructs feature point clouds to provide basic data for subsequent map reconstruction; 3D Gaussian Splatting technology optimizes the data fusion process by efficiently integrating data from different cameras (monocular, binocular, RGB-D); during the positioning process, the system uses ORB SLAM3 technology to quickly match the ORB feature points in the real-time captured image data, and by comparing with the feature points in the pre-established map, the system can accurately determine the current position of the device; through this position information, a high-simulation display of the positioning subject in the current 3D environment can be achieved.

[0065] The system implementation process of the present invention is as follows:

[0066] First, for the 3D scene reconstruction that combines 3D Gaussian Splatting and ORB SLAM3 in the present invention, it is necessary to first configure the environment, including OpenCV, Eigen, Pangolin, etc. Since the source code of 3D Gaussian Splatting and ORB SLAM3 is written in Python and C++ respectively, the operating environments of the two need to be installed and configured separately. After the corresponding environment is installed, the mapping task can be carried out.

[0067] For areas that need to be inspected and mapped, data collection must be performed on the area first. Use a monocular, binocular or RGB-D camera to collect video information of the area. It is best to form a closed loop for the data collection range, that is, the first frame and the last frame need to be shot at the same location. After the data collection is completed, the mapping task can be started. The system needs to extract feature points from the data set first, then calculate the descriptor, filter the key frames, and calculate the corresponding rotation matrix, scaling matrix and spherical harmonic function based on the filtered key frames. At the same time, the extracted ORB feature data is saved in the form of a point cloud. After completing the data calculation and processing to obtain the required new data, use 3D Gaussian Splatting to calculate the covariance matrix, color and opacity of each 3D Gaussian ellipsoid, and finally perform rasterization to complete the mapping. After completing the reconstruction of the 3D scene map, you can use the Viewer to view the 3D scene map saved locally.

[0068] After completing the mapping task, you can use the positioning function in this area. The device body that needs to be positioned needs to be equipped with a monocular camera. After starting the positioning, the camera needs to continuously read the current picture and select the picture with higher definition for calculation. The screened pictures are subjected to corner detection and descriptor calculation, and then feature-based frame-map matching (Scan-Map) is performed with the map saved locally. The pose information with the most feature points is the position information of the positioning subject currently equipped with the camera, and the positioning of the current frame is completed. Repeat the comparison of the current frame with the local map to complete the above process to complete the real-time positioning. After obtaining the real-time positioning information, it is combined with the point cloud model of the positioning subject, and the high-simulation display of real-time positioning is realized through visualization tools.

[0069] This method and system for 3D scene reconstruction and real-time positioning realizes an advanced, efficient and high-precision 3D scene reconstruction and real-time positioning platform by deeply integrating 3D Gaussian Splatting technology and ORB SLAM3 algorithm. This system not only provides high accuracy and high response speed, but also ensures a wide range of applicable scenarios and an intuitive and easy-to-use operation interface.

[0070] This method and system for 3D scene reconstruction and real-time positioning provides a solid and reliable navigation information foundation for automated equipment such as unmanned vehicles, drones and intelligent robots in complex and changing environments through refined 3D scene modeling and real-time environmental feature point matching mechanism.

[0071] This method and system for 3D scene reconstruction and real-time positioning reduces the user's learning curve and operation complexity by focusing on optimizing the convenience of operation and data reusability in system design, while building a rich data ecosystem to support subsequent advanced path planning, environmental analysis and performance optimization. This invention not only optimizes the performance of the automation system, but also broadens its application areas, bringing innovative solutions and significant progress to the industry.

[0072] The above description is only an implementation mode of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method and system for 3D scene reconstruction and real-time positioning based on 3D Gaussian splashing, characterized in that: The following steps are included: S1, collect data of three-dimensional scenes through monocular, binocular or RGB-D cameras; S2. Apply ORB SLAM3 technology to construct feature point cloud of the collected data, and obtain additional information such as rotation, scaling, spherical harmonics, etc. in addition to the ORB feature point cloud; S3. After completing data collection and analysis, the data is processed using 3D Gaussian splashing technology to achieve reconstruction of the three-dimensional scene. After the three-dimensional scene is constructed, the map is stored locally; S4. Use visualization tools to visualize local maps; S5. In the real-time positioning process, the camera carried by the positioning subject collects real-time images, uses ORB SLAM3 technology to analyze, obtains ORB features and compares feature points, matches the position most similar to the feature points in the current scene, and provides feedback, thereby achieving real-time positioning of the object; S6. Through visualization tools, incremental high-simulation display of real-time positioning based on local maps is realized.

2. The method and system for three-dimensional scene reconstruction and real-time positioning according to claim 1, characterized in that: In step S2, ORB SLAM3 technology is applied to construct a feature point cloud for the collected data, and additional information such as rotation, scaling, spherical harmonics, etc. is obtained in addition to the ORB feature point cloud. Specifically, S21, processing the three-dimensional scene data collected by the monocular, binocular or RGB-D camera, extracting feature point information through ORBSLAM3 technology and saving it through point cloud; S22. At the same time, by calculating the rotation matrix, scaling matrix and spherical harmonic function of each key frame starting from the initial frame, data is collected for the application of the 3D Gaussian splashing technology.

3. The method and system for three-dimensional scene reconstruction and real-time positioning according to any one of claims 1 to 2, characterized in that: In step S3, after completing data collection and analysis, the data is processed using 3D Gaussian splashing technology to achieve reconstruction of the three-dimensional scene. After the three-dimensional scene is constructed, the map is stored locally. Specifically, S31, processing the rotation matrix, scaling matrix and spherical harmonic function calculated from the data in the previous step by using the 3D Gaussian splashing technique; S32, calculating each point in the three-dimensional space point cloud with the rotation matrix, the scaling matrix and the spherical harmonic function to complete the reconstruction of the three-dimensional scene and save it.

4. The method and system for three-dimensional scene reconstruction and real-time positioning according to any one of claims 1 to 3, characterized in that: In step S4, a visualization tool is used to visualize the local map, specifically, S41. Visualize the reconstructed 3D scene map by installing the Viewer visualization tool officially provided by the 3D Gaussian Splashing technology.

5. The method and system for three-dimensional scene reconstruction and real-time positioning according to any one of claims 1 to 4, characterized in that: In step S5, during the real-time positioning process, the real-time image is collected by the camera carried by the positioning subject, and the ORBSLAM3 technology is used for analysis to obtain the ORB feature and perform feature point comparison, match the position most similar to the feature point in the current scene, and provide feedback, thereby achieving real-time positioning of the object, specifically, S51. When performing a real-time positioning task, it is necessary to collect data in real time through a camera carried by the positioning subject; S52, using ORB SLAM3 technology to process the collected data, extract feature points, and perform feature point matching to find the camera position with the highest similarity, i.e., the target position; S53, feeding back the position and adding it to the point cloud to achieve real-time positioning of the object.

6. The method and system for three-dimensional scene reconstruction and real-time positioning according to any one of claims 1 to 5, characterized in that: In step S6, a visualization tool is used to realize an incremental high-simulation display of real-time positioning based on a local map, specifically, S61. Perform three-dimensional reconstruction of the positioning subject through ORB SLAM technology, build a point cloud model of the positioning subject and save it locally; S62, combining the real-time positioning information added to the point cloud with the locally stored subject point cloud model, and displaying it through a visualization tool to achieve a high-simulation display of the real-time positioning based on the local map.

Images