Three-dimensional scene online reconstruction method and device, equipment and storage medium

Abstract

The invention relates to the field of three-dimensional reconstruction, and discloses a three-dimensional scene online reconstruction method, device and equipment and a storage medium, and the method is used for realizing the online reconstruction of a three-dimensional scene. The method comprises the following steps: acquiring and processing multiple frames of ordered scene RGB images to obtain sparse point clouds of each frame of scene RGB image under a camera coordinate system, and converting the sparse point clouds into sparse point clouds under a world coordinate system; initializing a sparse point cloud under a world coordinate system to obtain a local Gaussian surface element of each frame of scene RGB image; based on a fusion mechanism of a gating loop unit, reading global features of sparse point clouds of each frame of scene RGB image in a world coordinate system frame by frame from the first frame, and fusing the global features of the scene RGB image of the previous frame with the local Gaussian surface element of the scene RGB image of the current frame to generate a global Gaussian surface element of the scene RGB image of the current frame; and rendering the global Gaussian surface element of each frame of scene RGB image from a plurality of visual angles, and generating a scene reconstruction surface by using a Poisson reconstruction algorithm.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional scene online reconstruction method, device, equipment and storage medium. Background Art

[0002] Photorealistic reconstruction and rendering of large indoor scenes is a fundamental challenge in computer vision and graphics, with wide applications in virtual reality, robotics, and immersive simulation. Traditional methods, such as simultaneous localization and mapping (SLAM) systems, can achieve real-time 3D reconstruction, but they usually represent the scene using solid surfaces with vertex colors or texture maps. These representations lack the fine-grained appearance details required for high-quality photorealistic rendering, limiting their effectiveness in scenarios that require a realistic visual experience.

[0003] Recent advances in neural rendering, especially Neural Radiance Fields (NeRF), have achieved remarkable success in novel view synthesis by learning implicit representations of scenes. However, these methods usually require extensive optimization for each scene and are slow to render, especially when scaling to large environments. In addition, they usually require offline processing of all input images before optimization, which hinders the applicability of existing methods in real-time or interactive applications.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The present invention provides a three-dimensional scene online reconstruction method, device, equipment and storage medium, which are used to realize the online reconstruction of the three-dimensional scene.

[0006] The first aspect of the present invention provides a three-dimensional scene online reconstruction method, which includes: acquiring multiple frames of ordered scene RGB images, processing each frame of the scene RGB image, and obtaining a sparse point cloud of each frame of the scene RGB image in a camera coordinate system; converting the sparse point cloud in the camera coordinate system to a world coordinate system to obtain a sparse point cloud in the world coordinate system; initializing the sparse point cloud in the world coordinate system to obtain local Gaussian facets of each frame of the scene RGB image; based on a fusion mechanism of a gated recurrent unit, reading the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system frame by frame starting from the first frame, and fusing the global features of the scene RGB image of the previous frame with the local Gaussian facets of the scene RGB image of the current frame to generate the global Gaussian facets of the scene RGB image of the current frame; rendering the global Gaussian facets of each frame of the scene RGB image from multiple perspectives to obtain 2D images of each frame of the scene RGB image at different perspectives; and generating a scene reconstruction surface based on all 2D images using a Poisson reconstruction algorithm.

[0007] Preferably, the method of acquiring multiple frames of ordered scene RGB images, processing each frame of the scene RGB images, and obtaining a sparse point cloud of each frame of the scene RGB images in a camera coordinate system includes: acquiring multiple frames of ordered scene RGB images and a camera pose corresponding to each frame of the scene RGB images; preprocessing each frame of the scene RGB images; and constructing a sparse point cloud from the scene RGB images using a depth estimation method based on the camera pose corresponding to each frame of the scene RGB images, so as to obtain a sparse point cloud of each frame of the scene RGB images in the camera coordinate system.

[0008] Preferably, the converting of the sparse point cloud in the camera coordinate system to the world coordinate system to obtain the sparse point cloud in the world coordinate system includes: using the camera's position vector and rotation matrix to construct a 4x4 homogeneous transformation matrix; multiplying the sparse point cloud in the camera coordinate system by the transformation matrix to obtain new coordinates of the sparse point cloud in the world coordinate system; and using the new coordinates to update the sparse point cloud to obtain the sparse point cloud in the world coordinate system.

[0009] Preferably, the sparse point cloud in the initialized world coordinate system is used to obtain local Gaussian facets of each frame of the scene RGB image, including: constructing an initialized Gaussian facet based on a Gaussian function, wherein the parameters of the initialized Gaussian facets include center coordinates, rotation matrix, scaling vector, opacity and spherical harmonic coding; randomly initializing the parameters of the initialized Gaussian facets to obtain initialization parameters; using a feature extraction algorithm to detect key points in each frame of the scene RGB image, and calculating the parameters of the key points; using the parameters of the key points to update the parameters of the initialized Gaussian facets, and generating local Gaussian facets of each frame of the scene RGB image based on the updated initialization parameters.

[0010] Preferably, the fusion mechanism based on the gated recurrent unit reads the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system frame by frame starting from the first frame, and fuses the global features of the scene RGB image of the previous frame with the local Gaussian surface elements of the scene RGB image of the current frame to generate the global Gaussian surface elements of the scene RGB image of the current frame, including: reading the global features of the sparse point cloud of the scene RGB image of the previous frame in the world coordinate system based on the preset GRU model; reading the local features of the scene RGB image of the current frame based on the local Gaussian surface elements of the scene RGB image of the current frame; fusing the global features of the sparse point cloud of the scene RGB image of the previous frame in the world coordinate system with the local features of the scene RGB image of the current frame to obtain fused features; inputting the fused features into the preset GRU model to obtain the fused features of the current frame scene RGB image output by the preset GRU model; combining the fused features of the current frame scene RGB image and the sparse point cloud of the current frame scene RGB image in the world coordinate system to generate the global Gaussian surface elements of the current frame scene RGB image.

[0011] Preferably, the rendering of the global Gaussian surface elements of each frame of the scene RGB image from multiple perspectives to obtain the 2D image of each frame of the scene RGB image at different perspectives includes: equally dividing each frame of the scene RGB image into a plurality of image blocks; performing depth sorting on the global Gaussian surface elements detected in each image block; traversing the global Gaussian surface elements in each image block in the order of depth sorting, performing alpha blending calculation on the color of each intersecting global Gaussian surface element along the direction of the camera light to obtain the pixel color of each image block; and rendering and integrating each frame of the scene RGB image after block processing based on the pixel color to obtain the 2D image of each frame of the scene RGB image at different perspectives.

[0012] Preferably, the method of generating a scene reconstruction surface based on all 2D images using a Poisson reconstruction algorithm includes: processing all 2D images to obtain sparse point clouds and normal vectors thereof corresponding to the 2D images; reconstructing a Poisson surface based on the sparse point clouds and normal vectors thereof corresponding to the 2D images using a Poisson reconstruction algorithm to obtain an isosurface; and smoothing and denoising the isosurface to obtain a scene reconstruction surface.

[0013] The second aspect of the present invention provides an online reconstruction device for a three-dimensional scene, comprising: an acquisition module, used to acquire multiple frames of ordered scene RGB images, process each frame of the scene RGB image, and obtain a sparse point cloud of each frame of the scene RGB image in a camera coordinate system; a conversion module, used to convert the sparse point cloud in the camera coordinate system to the world coordinate system, and obtain a sparse point cloud in the world coordinate system; an initialization module, used to initialize the sparse point cloud in the world coordinate system, and obtain local Gaussian facets of each frame of the scene RGB image; a fusion module, used to read the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system frame by frame based on a fusion mechanism of a gated recurrent unit starting from the first frame, and fuse the global features of the scene RGB image of the previous frame with the local Gaussian facets of the scene RGB image of the current frame to generate the global Gaussian facets of the scene RGB image of the current frame; a rendering module, used to render the global Gaussian facets of each frame of the scene RGB image from multiple perspectives, and obtain 2D images of each frame of the scene RGB image at different perspectives; and a reconstruction module, used to generate a scene reconstruction surface based on all 2D images using a Poisson reconstruction algorithm.

[0014] A third aspect of the present invention provides a three-dimensional scene online reconstruction device, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the computer-readable instructions in the memory so that the three-dimensional scene online reconstruction device performs each step of the three-dimensional scene online reconstruction method as described above.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the various steps of the three-dimensional scene online reconstruction method as described above.

[0016] In the technical solution provided by the present invention, by acquiring multiple frames of ordered scene RGB images and corresponding camera poses, a sparse point cloud of each frame of image in the camera coordinate system can be accurately generated, and then converted to a unified world coordinate system. This process ensures the accuracy and consistency of the data; then, the sparse point cloud is initialized using local Gaussian surfaces, and the global features of the previous frame and the local Gaussian surfaces of the current frame are effectively fused through a fusion mechanism based on a gated recurrent unit, thereby generating a more complete and accurate global Gaussian surface. This fusion strategy not only improves the accuracy of reconstruction, but also enhances the coherence and integrity of the scene; in addition, rendering the global Gaussian surface from multiple perspectives can capture more details and features in the scene, providing rich data support for the subsequent Poisson reconstruction algorithm. Finally, the Poisson reconstruction algorithm can generate a high-quality scene reconstruction surface, which not only retains the high precision and detail information in the original scene, but also has good smoothness and continuity, providing a solid foundation for scene analysis, visualization and interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a three-dimensional scene online reconstruction method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a three-dimensional scene online reconstruction device provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a three-dimensional scene online reconstruction device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A first embodiment of a three-dimensional scene online reconstruction method in an embodiment of the present invention includes: S101, acquiring multiple frames of ordered scene RGB images, processing each frame of the scene RGB image, and obtaining a sparse point cloud of each frame of the scene RGB image in a camera coordinate system; S102, converting the sparse point cloud in the camera coordinate system to the world coordinate system to obtain a sparse point cloud in the world coordinate system; S103, initializing a sparse point cloud in a world coordinate system to obtain a local Gaussian surface element of each frame of the scene RGB image; S104, based on the fusion mechanism of the gated recurrent unit, reading the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system frame by frame starting from the first frame, and fusing the global features of the scene RGB image of the previous frame with the local Gaussian surface element of the scene RGB image of the current frame to generate the global Gaussian surface element of the scene RGB image of the current frame; S105, rendering the global Gaussian surface element of each frame of the scene RGB image from multiple perspectives to obtain a 2D image of each frame of the scene RGB image at different perspectives; S106 . Based on all the 2D images, generate a scene reconstruction surface using a Poisson reconstruction algorithm.

[0020] It is understandable that the execution subject of the present invention may be a 3D scene online reconstruction device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0021] In this embodiment, in step S101, multiple frames of ordered scene RGB images and a camera pose corresponding to each frame of the scene RGB image are obtained, and each frame of the scene RGB image is processed to obtain a sparse point cloud of each frame of the scene RGB image in the camera coordinate system, specifically including: obtaining multiple frames of ordered scene RGB images and a camera pose corresponding to each frame of the scene RGB image; preprocessing each frame of the scene RGB image; based on the camera pose corresponding to each frame of the scene RGB image, using a depth estimation method to construct a sparse point cloud from the scene RGB image to obtain a sparse point cloud of each frame of the scene RGB image in the camera coordinate system.

[0022] In this embodiment, multiple frames of ordered scene RGB images can be captured by a camera or a camera array, and the camera pose (including position and direction) corresponding to each frame of the scene RGB image can be recorded. Multiple frames of ordered scene RGB images and the camera pose corresponding to the scene RGB images can also be obtained from the captured video. The camera pose corresponding to the scene RGB image is obtained through camera calibration and pose estimation technology. Camera calibration is used to determine the internal parameters of the camera, such as focal length, distortion coefficient, etc.; pose estimation is used to clarify the position and orientation of the camera in the external world.

[0023] In this embodiment, each frame of the scene RGB image is preprocessed, including denoising, color correction, feature extraction, etc.

[0024] In this embodiment, a sparse point cloud is generated from a scene RGB image using a depth estimation method to obtain a sparse point cloud of each frame of the scene RGB image in a camera coordinate system.

[0025] Depth estimation refers to the process of inferring the three-dimensional position (i.e., depth) of objects in a scene from a two-dimensional image. Depth estimation methods include stereo vision, structured light, lightfield cameras, and deep learning techniques (such as convolutional neural networks (CNNs). Different depth estimation methods require specific processing based on their principles when combined with camera parameters. For example, the stereo vision method obtains disparity information through a binocular camera, and calculates the three-dimensional coordinates of objects in the scene by combining the camera's internal parameters and posture; deep learning technology uses a trained model to predict the depth value of each pixel in the image, and then converts it into three-dimensional coordinates by combining the camera parameters.

[0026] After obtaining the depth information of each frame of the scene RGB image, it can be combined with the internal and external parameters of the camera (i.e. the results of camera calibration and pose estimation) to generate sparse point clouds. These point clouds are represented in the camera coordinate system, and each point contains its position information (x, y, z) in three-dimensional space.

[0027] In this embodiment, in step S102, the sparse point cloud in the camera coordinate system is converted to the world coordinate system to obtain a sparse point cloud in the world coordinate system, specifically including: using the camera's position vector and rotation matrix to construct a 4x4 homogeneous transformation matrix; multiplying the sparse point cloud in the camera coordinate system with the transformation matrix to obtain new coordinates of the sparse point cloud in the world coordinate system; and using the new coordinates to update the sparse point cloud to obtain a sparse point cloud in the world coordinate system.

[0028] In this embodiment, the position vector describes the position of the camera in the world coordinate system, and the rotation matrix describes the orientation of the camera. In this embodiment, the sparse point cloud is represented in the form of homogeneous coordinates (ie, each point is a 4x1 vector, the last element of which is 1).

[0029] In this embodiment, in step S103, a sparse point cloud in a world coordinate system is initialized to obtain local Gaussian facets of each frame of the scene RGB image, specifically including: constructing an initialized Gaussian facet based on a Gaussian function, wherein the parameters of the initialized Gaussian facets include center coordinates, rotation matrix, scaling vector, opacity, and spherical harmonic coding; randomly initializing the parameters of the initialized Gaussian facets to obtain initialization parameters; using a feature extraction algorithm to detect key points in each frame of the scene RGB image, and calculating the parameters of the key points; using the parameters of the key points to update the parameters of the initialized Gaussian facets, and generating local Gaussian facets of each frame of the scene RGB image based on the updated initialization parameters.

[0030] In this embodiment, the feature extraction algorithm is such as SIFT, SURF or ORB algorithm.

[0031] In this embodiment, key points are generally points in the image with significant features (such as corner points, edges, or texture changes), which are crucial for subsequent updating of surface element parameters.

[0032] In this embodiment, the parameters of the key point include center coordinates, rotation matrix, scaling vector, opacity and spherical harmonic coding.

[0033] The rotation matrix can be estimated by analyzing the gradient or orientation of the pixels around the keypoint. The scaling vector can be estimated based on the size of the region around the keypoint or the scale of the feature point. Opacity may need to be estimated based on the brightness, color, or other visual features of the region around the keypoint. Spherical harmonic coding is often used to represent lighting and surface reflectance properties.

[0034] In this embodiment, in step S104, based on the fusion mechanism of the gated recurrent unit, the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system are read frame by frame starting from the first frame, and the global features of the scene RGB image of the previous frame are fused with the local Gaussian surface elements of the scene RGB image of the current frame to generate the global Gaussian surface elements of the scene RGB image of the current frame, specifically including: reading the global features of the sparse point cloud of the scene RGB image of the previous frame in the world coordinate system based on the preset GRU model; reading the local features of the scene RGB image of the current frame based on the local Gaussian surface elements of the scene RGB image of the current frame; fusing the global features of the sparse point cloud of the scene RGB image of the previous frame in the world coordinate system with the local features of the scene RGB image of the current frame to obtain the fused features; inputting the fused features into the preset GRU model to obtain the fused features of the current frame scene RGB image output by the preset GRU model; combining the fused features of the scene RGB image of the current frame and the sparse point cloud of the scene RGB image of the current frame in the world coordinate system to generate the global Gaussian surface elements of the scene RGB image of the current frame.

[0035] It should be noted that the GRU model is a recurrent neural network (RNN) model used to process sequence data. The core structure of the GRU model includes two gating mechanisms: Update Gate and Reset Gate.

[0036] The update gate determines whether the new input should be retained and how much information from the previous time step should be passed to the current time step. The update gate controls the flow of information by calculating a value between 0 and 1. When the value of the update gate is close to 1, it means that the information from the previous time step should be mostly retained; when the value is close to 0, it means that the information from the previous time step should be ignored and more attention should be paid to the current input.

[0037] The reset gate determines whether the hidden state of the previous time step should be ignored. The reset gate also controls the flow of information by calculating a value between 0 and 1. When the value of the reset gate is close to 1, it means that the hidden state of the previous time step should be retained; when the value is close to 0, it means that the hidden state of the previous time step should be ignored, and more attention should be paid to the relationship between the current input and the input of the previous time step.

[0038] The GRU model uses the current input and the hidden state of the previous time step for linear transformation, and obtains the values ​​of the reset gate and update gate through the sigmoid activation function.

[0039] The GRU model uses a reset gate to control the degree of retention of the hidden state of the previous time step, and concatenates it with the current input, then performs a linear transformation and tanh activation to obtain a candidate hidden state.

[0040] The GRU model uses an update gate to control the degree of retention of the hidden state of the previous time step and the candidate hidden state, and obtains the hidden state of the current time step through weighted summation.

[0041] In this embodiment, the preset GRU model will be used to process time series data and store and transmit the global feature information of the previous frame. The global features of the sparse point cloud of the previous frame scene RGB image in the world coordinate system can be read from the hidden state of the preset GRU model. These global features may include information describing the overall properties of the point cloud, such as shape, distribution, and density.

[0042] In this embodiment, the local Gaussian surface element of the current frame scene RGB image is converted into a local feature vector so as to be fused with the global feature of the previous frame.

[0043] In this embodiment, the global features of the sparse point cloud of the previous frame scene RGB image in the world coordinate system are spliced ​​or weighted summed with the local features of the current frame scene RGB image to obtain fused features. The purpose of this step is to combine the global information of the previous frame with the local information of the current frame to form a richer feature representation.

[0044] In this embodiment, the fused features are used as input and passed to the preset GRU model. The GRU model updates its hidden state and output state according to the input fused features. In this process, the update gate and reset gate of the GRU will control the flow and memory of information to generate the fused feature representation of the current frame scene RGB image.

[0045] In this embodiment, the fusion features of the current frame are combined with the sparse point cloud of the current frame scene RGB image in the world coordinate system. The purpose of this step is to attach the fused feature information to the sparse point cloud to generate a global Gaussian surface element with richer information. Based on the combined sparse point cloud and fusion features, the global Gaussian surface element of the current frame scene RGB image is generated. The global Gaussian surface element represents the shape, texture, brightness and other attributes of each part in the current frame scene, has global consistency and local details, and is modeled in the form of Gaussian distribution.

[0046] In this embodiment, taking the global Gaussian surface element acquisition process of the scene RGB image of the tth frame as an example, the calculation process can be described as:

[0047] in, and Respectively represent The global Gaussian bins and local Gaussian bins of the scene RGB image of the frame, Indicates The global Gaussian bins of the scene RGB image of the frame, is a gated recurrent unit.

[0048] In this embodiment, in step S105, the global Gaussian surface element of each frame of the scene RGB image is rendered from multiple perspectives to obtain a 2D image of each frame of the scene RGB image at different perspectives, specifically including: Each frame of the scene RGB image is equally divided into several image blocks; the global Gaussian surface elements detected in each image block are depth-sorted; the global Gaussian surface elements in each image block are traversed in the order of depth sorting, and the color of each intersecting global Gaussian surface element is alpha blended along the direction of the camera light to obtain the pixel color of each image block; based on the pixel color, each frame of the scene RGB image after block processing is rendered and integrated to obtain the 2D image of each frame of the scene RGB image at different viewing angles.

[0049] It can be understood that the Gaussian distribution presented by the Gaussian surface element is elliptical, so the Gaussian distribution of the points in the Gaussian local coordinate system can be expressed as:

[0050] In the formula, express uv Midpoint in space Gaussian distribution, u and v They represent the Gaussian local coordinate system. Axis and Axis coordinates.

[0051] During the rendering process, the shape of Gaussian surfaces may degenerate under certain viewing angles. When Gaussian surfaces degenerate at a certain time, the back-propagated gradient cannot be propagated to the global Gaussian surfaces through the rendering loss (because the Gaussian distribution is 0). Therefore, a variance is set for this situation so that the algorithm can be differentiable even in this situation. Expanded to:

[0052] In the formula, represents a pixel block, Represents an image block The global Gaussian distribution of Represents an image block The coordinates of the corresponding Gaussian surface element in the uv local coordinate system, Indicates based on Gaussian distribution value of c is the projection of the center point of the Gaussian surface element. Intuitively speaking, The lower bound of is determined by a fixed screen-space Gaussian low-pass filter with center c and radius In actual use, setting This ensures that enough pixels are used during rendering.

[0053] In this embodiment, the global Gaussian surface elements in each image block are traversed in the order of depth sorting, and the color of each intersecting global Gaussian surface element is alpha blended along the direction of the camera light to obtain the pixel color of each image block. The rendered pixel color is , then the pixel color is The pixel color rendering process is expressed as:

[0054] In the formula, Indicates the image block along the increasing depth direction The color of the global Gaussian bins, Indicates the image block along the increasing depth direction The opacity of the global Gaussian bins, Indicates the image block along the increasing depth direction The opacity of the global Gaussian bins, Represents an image block In the The global Gaussian distribution in the Gaussian local coordinate system, Represents an image block In the A global Gaussian distribution in a Gaussian local coordinate system.

[0055] In this embodiment, a depth loss function is used in the process of depth sorting of the global Gaussian surface detected in each image block. It is expressed as:

[0056] In the formula, Represents a collection of pixels The camera rays with valid depth information in and Represents image blocks Rendered depth and true depth.

[0057] In this embodiment, the global Gaussian surface elements in each image block are traversed in the order of depth sorting, and the color of each intersecting global Gaussian surface element is alpha blended along the direction of the camera light. In the process of obtaining the pixel color of each image block, a color loss function is used. The color loss function It is expressed as: In the formula, and Represents the rendered pixel color and the real pixel color respectively. A collection of pixels.

[0058] In this embodiment, in order to ensure that all global Gaussian surfaces are locally aligned with the actual surface during the rendering process, a normal consistency loss function is used to align the rendered normal map with the normal prior map. It is expressed as:

[0059] In the formula, represents the index of the facet intersected along the ray, is the mixing weight of the intersection point, is the normal of the strip facing the camera, is the normal estimated by the gradient of the depth map.

[0060] In this embodiment, during the rendering process, it is necessary to input the camera pose of the view, project the camera ray according to the camera pose, calculate the intersection of the camera ray and the global Gaussian surface element, and then calculate the Gaussian distribution at the intersection based on the intersection, thereby solving the shape degradation problem of the elliptical surface element when observed from a specific viewing angle.

[0061] In this embodiment, in step S106, a scene reconstruction surface is generated based on all 2D images using a Poisson reconstruction algorithm, specifically including: processing all 2D images to obtain sparse point clouds and normal vectors thereof corresponding to the 2D images; reconstructing a Poisson surface based on the sparse point clouds and normal vectors thereof corresponding to the 2D images using a Poisson reconstruction algorithm to obtain an isosurface; and smoothing and denoising the isosurface to obtain a scene reconstruction surface.

[0062] In this embodiment, the Poisson reconstruction algorithm generates a continuous three-dimensional surface by solving the Poisson equation, and this process uses normal vector information to maintain surface details and smoothness. The output of the algorithm is an isosurface, which represents the reconstructed three-dimensional surface.

[0063] The present embodiment provides a three-dimensional scene online reconstruction method, which can accurately generate a sparse point cloud of each frame in the camera coordinate system by acquiring multiple frames of ordered scene RGB images and corresponding camera poses, and then convert it to a unified world coordinate system. This process ensures the accuracy and consistency of the data; then, the sparse point cloud is initialized using local Gaussian surfaces, and the global features of the previous frame and the local Gaussian surfaces of the current frame are effectively fused through a fusion mechanism based on a gated recurrent unit, thereby generating a more complete and accurate global Gaussian surface. This fusion strategy not only improves the accuracy of reconstruction, but also enhances the coherence and integrity of the scene; in addition, rendering the global Gaussian surface from multiple perspectives can capture more details and features in the scene, providing rich data support for the subsequent Poisson reconstruction algorithm. Finally, the Poisson reconstruction algorithm can generate a high-quality scene reconstruction surface, which not only retains the high accuracy and detail information in the original scene, but also has good smoothness and continuity, providing a solid foundation for scene analysis, visualization and interaction.

[0064] The above describes the online reconstruction method of a three-dimensional scene in an embodiment of the present invention. The following describes the device in an embodiment of the present invention. Figure 2 , the implementation of the three-dimensional scene online reconstruction device in the embodiment of the present invention includes: The acquisition module 201 is used to acquire multiple frames of ordered scene RGB images, process each frame of the scene RGB image, and obtain a sparse point cloud of each frame of the scene RGB image in the camera coordinate system; A conversion module 202 is used to convert the sparse point cloud in the camera coordinate system to the world coordinate system to obtain the sparse point cloud in the world coordinate system; Initialization module 203, used to initialize the sparse point cloud in the world coordinate system to obtain the local Gaussian surface element of each frame of the scene RGB image; A fusion module 204 is used to read the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system frame by frame starting from the first frame based on the fusion mechanism of the gated recurrent unit, and fuse the global features of the scene RGB image of the previous frame with the local Gaussian surface element of the scene RGB image of the current frame to generate the global Gaussian surface element of the scene RGB image of the current frame; A rendering module 205 is used to render the global Gaussian surface element of each frame of the scene RGB image from multiple viewing angles to obtain a 2D image of each frame of the scene RGB image at different viewing angles; The reconstruction module 206 is used to generate a scene reconstruction surface based on all the 2D images using a Poisson reconstruction algorithm.

[0065] In this embodiment, by acquiring multiple frames of ordered scene RGB images and corresponding camera poses, a sparse point cloud of each frame of image in the camera coordinate system can be accurately generated, and then converted to a unified world coordinate system. This process ensures the accuracy and consistency of the data. Then, the sparse point cloud is initialized using local Gaussian surfaces, and the global features of the previous frame and the local Gaussian surfaces of the current frame are effectively fused through a fusion mechanism based on a gated recurrent unit, thereby generating a more complete and accurate global Gaussian surface. This fusion strategy not only improves the accuracy of reconstruction, but also enhances the coherence and integrity of the scene. In addition, rendering the global Gaussian surface from multiple perspectives can capture more details and features in the scene, providing rich data support for the subsequent Poisson reconstruction algorithm. Finally, the Poisson reconstruction algorithm can generate a high-quality scene reconstruction surface, which not only retains the high precision and detail information in the original scene, but also has good smoothness and continuity, providing a solid foundation for scene analysis, visualization and interaction.

[0066] Figure 2 The structure of the illustrated three-dimensional scene online reconstruction device does not constitute a limitation on the three-dimensional scene online reconstruction device, and can implement the steps of the three-dimensional scene online reconstruction method provided by the above-mentioned method embodiments.

[0067] above Figure 2 The three-dimensional scene online reconstruction device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The three-dimensional scene online reconstruction device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0068] Figure 3 3 is a schematic diagram of the structure of a three-dimensional scene online reconstruction device provided by an embodiment of the present invention. The device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the device 300. Furthermore, the processor 310 may be configured to communicate with the storage medium 330 and execute a series of instruction operations in the storage medium on the device 300.

[0069] The device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc.

[0070] An embodiment of the present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the three-dimensional scene online reconstruction method.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0072] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0073] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional scene online reconstruction method, characterized in that: The three-dimensional scene online reconstruction method comprises: Acquire multiple frames of ordered scene RGB images, process each frame of the scene RGB image, and obtain a sparse point cloud of each frame of the scene RGB image in the camera coordinate system; The sparse point cloud in the camera coordinate system is converted to the world coordinate system to obtain the sparse point cloud in the world coordinate system; Initialize the sparse point cloud in the world coordinate system to obtain the local Gaussian surface element of each frame of the scene RGB image; Based on the fusion mechanism of the gated recurrent unit, the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system are read frame by frame starting from the first frame, and the global features of the scene RGB image of the previous frame are fused with the local Gaussian surface element of the scene RGB image of the current frame to generate the global Gaussian surface element of the scene RGB image of the current frame; Render the global Gaussian surface element of each frame of the scene RGB image from multiple perspectives to obtain a 2D image of each frame of the scene RGB image at different perspectives; Based on all 2D images, a scene reconstruction surface is generated using the Poisson reconstruction algorithm.

2. The three-dimensional scene online reconstruction method according to claim 1, characterized in that: The method of acquiring multiple frames of ordered scene RGB images and processing each frame of the scene RGB images to obtain a sparse point cloud of each frame of the scene RGB images in a camera coordinate system includes: Obtain multiple frames of ordered scene RGB images and the camera pose corresponding to each frame of the scene RGB image; Preprocess each frame of scene RGB image; Based on the camera pose corresponding to each frame of the scene RGB image, a sparse point cloud is constructed from the scene RGB image using a depth estimation method to obtain a sparse point cloud of each frame of the scene RGB image in the camera coordinate system.

3. The three-dimensional scene online reconstruction method according to claim 1, characterized in that: The step of converting the sparse point cloud in the camera coordinate system to the world coordinate system to obtain the sparse point cloud in the world coordinate system includes: Use the camera's position vector and rotation matrix to construct a 4x4 homogeneous transformation matrix; Multiply the sparse point cloud in the camera coordinate system by the transformation matrix to obtain the new coordinates of the sparse point cloud in the world coordinate system; The sparse point cloud is updated with the new coordinates to obtain a sparse point cloud in the world coordinate system.

4. The three-dimensional scene online reconstruction method according to claim 1, characterized in that: The sparse point cloud in the initialized world coordinate system is obtained to obtain the local Gaussian surface element of each frame of the scene RGB image, including: Constructing an initialized Gaussian facet based on a Gaussian function, wherein parameters of the initialized Gaussian facet include a center coordinate, a rotation matrix, a scaling vector, opacity, and a spherical harmonic code; Randomly initializing the parameters of the initialized Gaussian facet to obtain initialization parameters; Use feature extraction algorithm to detect key points in each frame of scene RGB image and calculate the parameters of key points; The initialization parameters are updated using the parameters of the key points, and local Gaussian surfaces of the RGB image of each frame of the scene are generated based on the updated initialization parameters.

5. The three-dimensional scene online reconstruction method according to claim 1, characterized in that: The fusion mechanism based on the gated recurrent unit reads the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system frame by frame starting from the first frame, and fuses the global features of the scene RGB image of the previous frame with the local Gaussian surface element of the scene RGB image of the current frame to generate the global Gaussian surface element of the scene RGB image of the current frame, including: Based on the preset GRU model, the global features of the sparse point cloud of the previous frame scene RGB image in the world coordinate system are read; Reading local features of the current frame scene RGB image based on the local Gaussian surface element of the current frame scene RGB image; The global features of the sparse point cloud of the previous frame scene RGB image in the world coordinate system are fused with the local features of the current frame scene RGB image to obtain fused features; The fused features are input into a preset GRU model to obtain the fused features of the current frame scene RGB image output by the preset GRU model; The global Gaussian surface element of the current frame scene RGB image is generated by combining the fusion features of the current frame scene RGB image and the sparse point cloud of the current frame scene RGB image in the world coordinate system.

6. The three-dimensional scene online reconstruction method according to claim 1, characterized in that: The global Gaussian facets of each frame of the scene RGB image are rendered from multiple perspectives to obtain 2D images of each frame of the scene RGB image at different perspectives, including: Divide each frame of scene RGB image equally into several image blocks; Deep sorting of the global Gaussian surfels detected within each image patch; In the order of depth sorting, traverse the global Gaussian surface elements in each image block, perform alpha blending calculation on the color of each intersecting global Gaussian surface element along the direction of the camera light, and obtain the pixel color of each image block; The scene RGB image of each frame after block processing is rendered and integrated based on pixel color to obtain a 2D image of each frame of the scene RGB image at different viewing angles.

7. The three-dimensional scene online reconstruction method according to claim 1, characterized in that: Based on all the 2D images, a Poisson reconstruction algorithm is used to generate a scene reconstruction surface, including: All 2D images are processed to obtain sparse point clouds and their normal vectors corresponding to the 2D images; Based on the sparse point cloud corresponding to the 2D image and its normal vector, the Poisson reconstruction algorithm is used to reconstruct the Poisson surface and obtain the isosurface; The isosurface is smoothed and denoised to obtain a scene reconstruction surface.

8. A three-dimensional scene online reconstruction device, characterized in that: include: An acquisition module is used to acquire multiple frames of ordered scene RGB images, process each frame of the scene RGB image, and obtain a sparse point cloud of each frame of the scene RGB image in the camera coordinate system; A conversion module is used to convert the sparse point cloud in the camera coordinate system to the world coordinate system to obtain the sparse point cloud in the world coordinate system; Initialization module, used to initialize the sparse point cloud in the world coordinate system and obtain the local Gaussian surface element of each frame of the scene RGB image; A fusion module is used for a fusion mechanism based on a gated recurrent unit, which reads the global features of the sparse point cloud of each frame of the scene RGB image in the world coordinate system frame by frame starting from the first frame, and fuses the global features of the scene RGB image of the previous frame with the local Gaussian surface element of the scene RGB image of the current frame to generate the global Gaussian surface element of the scene RGB image of the current frame; A rendering module is used to render the global Gaussian surface element of each frame of the scene RGB image from multiple perspectives to obtain a 2D image of each frame of the scene RGB image at different perspectives; The reconstruction module is used to generate a scene reconstruction surface based on all 2D images using a Poisson reconstruction algorithm.

9. A three-dimensional scene online reconstruction device, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute the various steps of the three-dimensional scene online reconstruction method as described in any one of claims 1-7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the three-dimensional scene online reconstruction method as described in any one of claims 1 to 7 are implemented.

Images