Streaming three-dimensional scene reconstruction method based on explicit spatial point memory

Abstract

The invention discloses a streaming three-dimensional scene reconstruction method based on explicit spatial point memory, and aims to solve the problems of memory redundancy and information loss existing in an implicit memory method in existing streaming three-dimensional reconstruction. According to the method, firstly, explicit spatial point memory is initialized through feature coding of a first frame of image, and scene features and explicit spatial positions are directly associated through the memory; each time a frame of real-time RGB image is received, image features are firstly extracted and efficiently interacted with stored spatial point memory features, and three-dimensional prior is provided by combining three-dimensional hierarchical position embedding to obtain feature representation under a global coordinate system; decoding point diagrams and camera poses in the global coordinate system, and encoding to generate a new spatial point memory unit; and finally, updating the global explicit spatial point memory through a memory fusion module. According to the method, the non-registration pose streaming visual input is unified to the global coordinate system online, the pixel-by-pixel alignment point diagram is output, and the reconstruction efficiency, uniformity and flexibility are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning technology, and in particular to a streaming 3D scene reconstruction method and apparatus based on explicit spatial point memory. Background Technology

[0002] Dense 3D reconstruction from image sequences has long been a fundamental task in computer vision, widely applied in fields such as autonomous driving, medical modeling, and cultural heritage preservation. Classical 3D reconstruction methods are typically optimization-based and scene-specific. Motion-based structure restoration (SfM) follows a pipeline of feature extraction, image matching, triangulation and 3D reconstruction, and bundle adjustment to obtain sparse geometry and camera pose estimation. Building upon this, subsequent methods such as multi-view stereo vision (MVS), neural radiation fields (NeRF), and 3D Gaussian sputtering (3DGS) use known camera parameters to recover dense geometry or high-fidelity scene representations. These methods rely on the sequential combination of multiple modules, requiring long optimization times and being susceptible to noise. Notably, simultaneous localization and mapping (SLAM) can perform localization and reconstruction online. However, they often rely on specific camera motion assumptions (which can sometimes be misleading or restrictive) or require additional depth / LiDAR sensors to improve performance.

[0003] To improve accuracy and efficiency, 3D reconstruction is gradually shifting towards a learning-based and end-to-end paradigm. Some methods use learnable modules to replace manually designed components in traditional reconstruction workflows, while others attempt to optimize the entire workflow end-to-end. DUSt3R introduces a pointmap representation, which can directly align two images to the same coordinate system, thus unifying all the aforementioned subtasks. MonST3R extends this paradigm to dynamic scenes and fine-tunes it on dynamic datasets. Because it relies on image pairs as input, DUSt3R requires additional global alignment steps when performing dense reconstruction from multiple images, which significantly reduces processing efficiency in multi-view scenarios. To improve efficiency, existing methods can be divided into two paradigms. One type of method processes all input images simultaneously and uses a global attention mechanism to reconstruct them into a representation in a unified coordinate system. This requires significant computational resources and is incompatible with the incremental processing methods in real-world reconstruction tasks. The second type of method adopts a streaming reconstruction paradigm, introducing an external memory mechanism to retain information from past frames, allowing new inputs to be directly integrated into the global coordinate system. The streaming paradigm is closer to real-world applications, improving scalability without incurring excessive computational burden. For example, Spann3R maintains a memory module to cache implicit features of processed frames. However, this implicit memory often contains redundant information, and once the memory capacity is saturated, a discarding strategy must be adopted. CUT3R uses a fixed-length implicit feature memory and updates the memory directly through interaction with image features. However, as the number of processed frames increases, this fixed-capacity memory inevitably leads to the loss of early information.

[0004] Building upon this foundation, inspired by human memory mechanisms, this invention proposes a streaming 3D scene reconstruction device based on explicit spatial point memory. Human memory of previously experienced environments is inherently closely related to spatial location. For example, when discussing a café or a bedroom, the images recalled are entirely different. Similarly, each point memory in the explicit spatial point memory of this invention is assigned a 3D location in a global coordinate system. Each point memory is directly linked to an explored spatial location and points to a dynamically updated spatial feature. Corresponding to the limitations of current work, the proposal and implementation of this device are necessary in two ways: 1) This explicit spatial point memory maps stored memory units to explored spatial locations rather than implicitly mapping features, making the stored memories compact, efficient, and free of redundancy; 2) This explicit spatial point memory directly stores scene features in a global coordinate system, essentially a 3D representation of the current scene, enabling the direct and efficient unification of the latest frame's information into this global coordinate system during streaming reconstruction tasks. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] To address this, this invention proposes a streaming 3D scene reconstruction method based on explicit spatial point memory. The aim of this invention is to solve the problems of memory redundancy and information loss inherent in existing streaming 3D reconstruction methods based on implicit memory. By proposing an explicit spatial point memory, the stored scene feature information is directly associated with explicit spatial locations, resulting in less memory redundancy and a positive correlation between memory capacity and the size of the explored scene. This invention designs an image-spatial point memory interaction mechanism to unify the visual input of the latest frame into the current global coordinate system as a point map output. Then, a designed explicit memory encoding mechanism is used to encode the output of the current frame into new spatial point memories. A memory fusion mechanism is used to supplement and update the overall explicit spatial point memory. Based on this, streaming 3D reconstruction based on explicit spatial point memory can be performed continuously and efficiently.

[0007] This invention introduces explicit spatial point memory into the field of streaming 3D reconstruction research and proposes various processing modules adapted to it. Most existing streaming 3D reconstruction methods rely on implicit scene feature memory, which is redundant and suffers from severe information loss. The explicit spatial point memory introduced in this invention directly maps stored scene features to their positions in 3D space, effectively solving the problems of information redundancy and information loss. An image-spatial point memory interaction mechanism is designed to enable efficient interaction between images and stored spatial point memories, and a corresponding 3D hierarchical position embedding is designed to add direct 3D priors to the interaction. Through this interaction, the visual input of the latest frame is unified into the current global coordinate system in the form of a point map output, enabling streaming 3D reconstruction to continue. An explicit memory encoding mechanism and a memory fusion mechanism are designed to encode and supplement the information of the latest frame into the overall explicit spatial point memory, so that the stored explicit spatial point memory directly corresponds to the currently explored spatial range.

[0008] The implementation process of this invention involves first encoding the image features of the first frame to initialize an explicit spatial point memory. Upon input of each frame, an image-spatial point memory interaction module is used to obtain the feature representation of the current input in the global coordinate system, and the point map representation and corresponding camera pose of the current input are decoded from these global features. Next, an explicit memory encoding module is used to encode the current image features and the output point map into spatial features, assigning each spatial feature an explicit spatial position obtained from the point map. Finally, when these new spatial features are added to the explicit spatial point memory, a memory fusion module is used to make the overall spatial point memory distribution more uniform and unified. By maintaining this explicit spatial point memory that directly corresponds to the explored spatial range, this invention enables the direct online unification of unregistered pose streaming visual input to the global coordinate system and output as a pixel-aligned point map.

[0009] Another objective of this invention is to propose a streaming 3D scene reconstruction device based on explicit spatial point memory.

[0010] To achieve the above objectives, this invention proposes a streaming 3D scene reconstruction method based on explicit spatial point memory, comprising: Initialize explicit spatial point memory by encoding the first frame of input image to obtain initial image features, and generate explicit spatial point memory units with unassigned 3D positions based on the initial image features; The current frame image is encoded using an image encoder to extract the features of the current image; The current image features are interacted with the features in the explicit spatial point memory unit by using an interactive decoder based on an image encoder to obtain a global feature representation of the current image global coordinate system. Based on the global feature representation, the point map representation of the current frame image in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information; Based on the current frame image features, the determined 3D point map representation, and the camera pose information, a new spatial point memory unit is generated; the new spatial point memory unit is then fused with the existing explicit spatial point memory unit, and the feature information of the relevant memory units is updated by comparing their spatial positions.

[0011] In addition, the streaming 3D scene reconstruction method based on explicit spatial point memory according to the above embodiments of the present invention may also have the following additional technical features: Furthermore, in one embodiment of the present invention, the interaction between the current image features and the features in the explicit spatial point memory unit is performed using an interactive decoder based on an image encoder, including: Incorporate three-dimensional spatial positional relationship information during the interaction process; Three-dimensional spatial positional relationship information is introduced through a hierarchical positional embedding method. Corresponding spatial coordinates are assigned to the features participating in the interaction, and the three-dimensional spatial position of the features is encoded based on a rotation matrix. The parameters of the rotation matrix are related to the feature dimension to adapt to different ranges of spatial coordinates.

[0012] Further, in one embodiment of the present invention, the method of representing the point map of the current frame image in its own coordinate system and the global coordinate system, and the corresponding camera pose information based on the global feature representation includes: A camera pose prediction head based on MLP is used to decode the camera pose information in the current coordinate system and the global coordinate system from the global feature representation; Using a DPT-based point plot prediction head, a 3D point plot representation in the current coordinate system and the global coordinate system is decoded from the global feature representation.

[0013] Furthermore, in one embodiment of the present invention, the fusion of the new spatial point memory unit with an existing explicit spatial point memory unit includes: For each new spatial point memory unit, find the existing spatial point memory unit that is the nearest neighbor of its spatial location in the explicit spatial point memory. If the distance between the new spatial point memory unit and the nearest neighbor's spatial location is less than a preset threshold, it is determined to be associated, and the features of the new spatial point memory unit are fused with the features of the existing spatial point memory unit. If the distance is greater than or equal to the preset threshold, the new spatial point memory unit is added as a new spatial point to the explicit spatial point memory.

[0014] To achieve the above objectives, another aspect of the present invention proposes a streaming 3D scene reconstruction device based on explicit spatial point memory, comprising: The initialization module initializes the explicit spatial point memory, encodes the first frame of input image to obtain initial image features, and generates explicit spatial point memory units with unassigned three-dimensional positions based on the initial image features. The feature extraction module encodes the current frame image using an image encoder to extract the features of the current image. The feature interaction module uses an interactive decoder based on an image encoder to interact with the features in the explicit spatial point memory unit to obtain a global feature representation of the current image global coordinate system. The feature decoding module, based on the global feature representation, extracts the point map representation of the current frame image in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information; The memory fusion module generates new spatial point memory units based on the features of the current frame image, the determined 3D point map representation, and the camera pose information; it then fuses the new spatial point memory units with existing explicit spatial point memory units, and updates the feature information of the relevant memory units through spatial position comparison.

[0015] The present invention provides a streaming 3D scene reconstruction method and apparatus based on explicit spatial point memory, which can realize the online unification of registration-free pose streaming visual input to the global coordinate system and output a pixel-by-pixel aligned point map, thereby improving reconstruction efficiency, uniformity and flexibility.

[0016] The beneficial effects of this invention are as follows: 1) Addressing core pain points: Effectively improves the problems of excessive memory redundancy and serious information loss in existing implicit memory methods in streaming 3D reconstruction, and directly associates scene features with spatial location through explicit spatial point memory, making memory storage more accurate.

[0017] 2) More efficient and continuous reconstruction: With the help of the image-spatial point memory interaction mechanism and 3D hierarchical position embedding, the visual input of new frames can be quickly unified into the global coordinate system, ensuring the continuous progress of streaming reconstruction and reducing invalid calculations.

[0018] 3) Better memory management: Through explicit memory encoding and fusion mechanisms, newly generated spatial point memories can update global memory, making the memory capacity match the range of the explored scene, and the memory distribution is even, avoiding local density or sparseness.

[0019] 4) More accurate and practical output: It realizes online global alignment of streaming visual input without registration pose and outputs pixel-aligned point maps, providing high-quality basic data for subsequent scenario applications (such as navigation and modeling).

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a streaming 3D scene reconstruction method based on explicit spatial point memory according to an embodiment of the present invention; Figure 2 This is an architecture diagram of a streaming 3D scene reconstruction method based on explicit spatial point memory according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a streaming 3D scene reconstruction device based on explicit spatial point memory according to an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] The following description, with reference to the accompanying drawings, describes a streaming 3D scene reconstruction method and apparatus based on explicit spatial point memory according to an embodiment of the present invention.

[0025] Figure 1 This is a flowchart of a streaming 3D scene reconstruction method based on explicit spatial point memory, according to an embodiment of the present invention.

[0026] like Figure 1 As shown, the method includes, but is not limited to, the following steps: S1, Initialize explicit spatial point memory, encode the first frame of input image to obtain initial image features, and generate explicit spatial point memory units with unassigned three-dimensional positions based on the initial image features; S2, the current frame image is encoded by an image encoder to extract the features of the current image; S3, using an interactive decoder based on an image encoder to interact with the features in the explicit spatial point memory unit to obtain a global feature representation of the current image global coordinate system; S4, based on the global feature representation, the point map representation of the current frame image in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information; S5. Based on the current frame image features, the determined 3D point map representation, and the camera pose information, generate a new spatial point memory unit; merge the new spatial point memory unit with the existing explicit spatial point memory unit, and update the feature information of the relevant memory unit by comparing spatial positions.

[0027] This invention discloses a streaming 3D scene reconstruction method based on explicit spatial point memory. This method maintains an explicit spatial point memory for each 3D scene, making the representation and reconstruction of the current scene more efficient, uniform, and flexible. Specifically, the explicit spatial point memory is first initialized by encoding the image features of the first frame. Whenever a new real-time RGB image input is received, the image features extracted from the current input and the stored spatial point memory features are efficiently interacted to obtain the feature representation of the current input in the global coordinate system. Different prediction heads are then used to decode the point map representation of the current input in its own coordinate system and the global coordinate system, along with the corresponding camera pose information, from this global feature representation. Finally, the current input and output are encoded into explicit spatial points, and a memory fusion module is used to fuse them into the explicit spatial point memory of the entire scene, thereby achieving continuous online streaming 3D reconstruction.

[0028] The target tracking method for fusing multi-view images and 3D point clouds according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Specifically, as shown... Figure 2 As shown, the steps are as follows: Initialize explicit space point memory: First, the first frame of the input image is encoded, and the output image features are used to further encode and initialize the explicit spatial point memory of the current scene. At this time, the explicit spatial point memory has not yet been assigned individual spatial point positions.

[0029] Efficient interaction between image input and explicit spatial point memory includes: Image Encoder: For each input frame, this invention uses a ViT structure to encode it and outputs the current image features:

[0030] Interactive Decoder: This invention uses a ViT-based interactive decoder to achieve interaction between image features and spatial point features.

[0031] In the interactive structure, this invention designs a three-dimensional hierarchical position embedding encoding for the features participating in the interaction, providing a more direct three-dimensional prior for the interaction. Specifically, each spatial point memory consists of a real spatial location coordinate and an updatable spatial feature vector, while each image feature is assigned local spatial coordinates obtained from the point map of the previous frame (assuming that the change of viewpoint has a certain continuity). Inspired by RoPE encoding, for each spatial location:

[0032] This invention defines a rotation matrix:

[0033] in, , , It is the dimension of the encoded feature. There can be multiple possible values ​​(i.e., hierarchical) to accommodate different ranges of input coordinates. Therefore, this invention can apply this hierarchical position embedding to the query and key involved in the interaction:

[0034] This invention provides a priori spatial location relationships for each interactive feature through this embedding, promoting more efficient interaction and fusion of image features and spatial point features, and finally obtaining a feature representation of image information in the global coordinate system.

[0035] Downstream forecast head: This invention uses an MLP-based camera pose prediction head and a DPT-based point map prediction head to decode the point map representation of the current input in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information, from the global feature representation of the current image.

[0036] Explicit Memory Encoder: This invention designs a lightweight explicit memory encoder based on MLP and ViT to encode new supplementary spatial point memory units from current image features and current output.

[0037] These spatial point memory units will be supplemented into the existing explicit spatial point memory through the memory fusion module.

[0038] Memory Fusion Module: For all newly acquired spatial point memory units, this invention searches for their nearest neighbors in the existing spatial point memory based on their spatial coordinates. If the distance between the spatial point and its nearest neighbor is less than a certain set value, the spatial point is considered to have sufficiently close relevant features in the original memory; otherwise, the spatial point is directly added to the explicit spatial point memory. For each spatial point in the original memory, if it is marked as having relevant features by multiple newly acquired spatial point units, the spatial point memory at that location will be updated.

[0039] in It is the number of times each original spatial point is identified as having relevant features during this update and fusion (i.e., the number of nearest neighbors).

[0040] Through this memory fusion module, the explicit spatial point memory of the present invention can uniformly and effectively store information of the explored scene, enabling continuous and efficient streaming 3D scene reconstruction.

[0041] According to an embodiment of the present invention, a streaming 3D scene reconstruction method based on explicit spatial point memory can realize the online unification of unregistered pose streaming visual input to the global coordinate system and output a pixel-by-pixel aligned point map, thereby improving reconstruction efficiency, uniformity and flexibility.

[0042] To achieve the above embodiments, such as Figure 3 As shown, this embodiment also provides a streaming 3D scene reconstruction device 10 based on explicit spatial point memory. The device 10 includes an initialization module 100, a feature extraction module 200, a feature interaction module 300, a feature decoding module 400, and a memory fusion module 500.

[0043] Initialization module 100 initializes explicit spatial point memory, encodes the first frame input image to obtain initial image features, and generates explicit spatial point memory units with unassigned three-dimensional positions based on the initial image features. The feature extraction module 200 encodes the current frame image using an image encoder to extract the features of the current image; The feature interaction module 300 uses an interactive decoder based on an image encoder to interact with the features in the explicit spatial point memory unit to obtain a global feature representation of the current image global coordinate system. The feature decoding module 400, based on the global feature representation, extracts the point map representation of the current frame image in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information; The memory fusion module 500 generates new spatial point memory units based on the features of the current frame image, the determined three-dimensional point map representation, and the camera pose information; it then fuses the new spatial point memory units with existing explicit spatial point memory units, and updates the feature information of the relevant memory units through spatial position comparison.

[0044] Furthermore, the feature interaction module is also used for: Incorporate three-dimensional spatial positional relationship information during the interaction process; Three-dimensional spatial positional relationship information is introduced through a hierarchical positional embedding method. Corresponding spatial coordinates are assigned to the features participating in the interaction, and the three-dimensional spatial position of the features is encoded based on a rotation matrix. The parameters of the rotation matrix are related to the feature dimension to adapt to different ranges of spatial coordinates.

[0045] Furthermore, the feature decoding module is also used for: Using an MLP-based camera pose prediction head, the camera pose information in the current coordinate system and the global coordinate system is decoded from the global feature representation; Using a DPT-based point plot prediction head, a 3D point plot representation in the current local coordinate system and the global coordinate system is decoded from the global feature representation.

[0046] Furthermore, the memory fusion module is also used for: For each new spatial point memory unit, find the existing spatial point memory unit that is the nearest neighbor of its spatial location in the explicit spatial point memory. If the distance between the new spatial point memory unit and the nearest neighbor's spatial location is less than a preset threshold, it is determined to be associated, and the features of the new spatial point memory unit are fused with the features of the existing spatial point memory unit. If the distance is greater than or equal to the preset threshold, the new spatial point memory unit is added as a new spatial point to the explicit spatial point memory.

[0047] According to an embodiment of the present invention, a streaming 3D scene reconstruction device based on explicit spatial point memory can realize the online unification of unregistered pose streaming visual input to the global coordinate system and output a pixel-by-pixel aligned point map, thereby improving reconstruction efficiency, uniformity and flexibility.

[0048] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0049] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A streaming 3D scene reconstruction method based on explicit spatial point memory, characterized in that, Includes the following steps: Initialize explicit spatial point memory by encoding the first frame of input image to obtain initial image features, and generate explicit spatial point memory units with unassigned 3D positions based on the initial image features; The current frame image is encoded using an image encoder to extract the features of the current image; The current image features are interacted with the features in the explicit spatial point memory unit by using an interactive decoder based on an image encoder to obtain a global feature representation of the current image global coordinate system. Based on the global feature representation, the point map representation of the current frame image in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information; Based on the current frame image features, the determined 3D point map representation, and the camera pose information, a new spatial point memory unit is generated; the new spatial point memory unit is then fused with the existing explicit spatial point memory unit, and the feature information of the relevant memory units is updated by comparing their spatial positions.

2. The method according to claim 1, characterized in that, The interaction between the current image features and the features in the explicit spatial point memory unit is achieved using an interactive decoder based on an image encoder, including: Incorporate three-dimensional spatial positional relationship information during the interaction process; Three-dimensional spatial positional relationship information is introduced through a hierarchical positional embedding method. Corresponding spatial coordinates are assigned to the features participating in the interaction, and the three-dimensional spatial position of the features is encoded based on a rotation matrix. The parameters of the rotation matrix are related to the feature dimension to adapt to different ranges of spatial coordinates.

3. The method according to claim 1, characterized in that, Based on the global feature representation, the point map representation of the current frame image in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information, are obtained, including: A camera pose prediction head based on MLP is used to decode the camera pose information in the current coordinate system and the global coordinate system from the global feature representation; Using a DPT-based point plot prediction head, a 3D point plot representation in the current coordinate system and the global coordinate system is decoded from the global feature representation.

4. The method according to claim 1, characterized in that, The integration of the new spatial point memory unit with the existing explicit spatial point memory unit includes: For each new spatial point memory unit, find the existing spatial point memory unit that is the nearest neighbor of its spatial location in the explicit spatial point memory. If the distance between the new spatial point memory unit and the nearest neighbor's spatial location is less than a preset threshold, it is determined to be associated, and the features of the new spatial point memory unit are fused with the features of the existing spatial point memory unit. If the distance is greater than or equal to the preset threshold, the new spatial point memory unit is added as a new spatial point to the explicit spatial point memory.

5. The method according to claim 1, characterized in that, The current frame image is encoded using an image encoder to extract current image features, including: For each input frame, a ViT structure is used to encode it, and the current image features are output: in, This represents the current image features.

6. The method according to claim 2, characterized in that, The method further includes using an interactive decoder based on an image encoder to interact with the features in the explicit spatial point memory unit to obtain a global feature representation of the current image in the global coordinate system, and also includes: Implement the interaction between image features and spatial point features using a ViT-based interactive decoder: Each spatial point memory consists of a real spatial location coordinate and an updated spatial feature vector, while each image feature is assigned local spatial coordinates obtained from the point map of the previous frame; for each spatial location: Define the rotation matrix: in, , , If the dimension is the encoded feature, then the hierarchical positional embedding is applied to the query and key involved in the interaction: By embedding, each feature involved in the interaction is given a priori spatial location relative to the other features, enabling the fusion of image features and spatial point features, and finally obtaining the feature representation of image information in the global coordinate system.

7. The method according to claim 3, characterized in that, Based on the global feature representation, the point map representation of the current frame image in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information, also includes: The MLP-based camera pose prediction head and the DPT-based point map prediction head decode the point map representation of the current input in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information, from the global feature representation of the current image. 。 8. The method according to claim 1, characterized in that, Based on the current frame image features, the determined 3D point map representation, and camera pose information, new spatial point memory units are generated, including: Design a lightweight explicit memory encoder based on MLP and ViT to encode new supplementary spatial point memory units from current image features and current output: Spatial point memory units are supplemented into existing explicit spatial point memories through memory fusion modules.

9. The method according to claim 4, characterized in that, The process of fusing the new spatial point memory unit with existing explicit spatial point memory units, and updating the feature information of related memory units through spatial location comparison, also includes: For all newly acquired spatial point memory units, their nearest neighbors are searched in the existing spatial point memory based on their spatial coordinates. If the distance between the spatial point and its nearest neighbor is less than a certain set value, the spatial point is considered to have sufficiently close relevance features in the original memory; otherwise, the spatial point is directly added to the explicit spatial point memory. For each spatial point in the original memory, if it is marked as a relevance feature by multiple newly acquired spatial point units, the spatial point memory at that location is updated. in It is the number of times each original spatial point is identified as a relevant feature during this update and fusion.

10. A streaming 3D scene reconstruction device based on explicit spatial point memory, characterized in that, include: The initialization module initializes the explicit spatial point memory, encodes the first frame of input image to obtain initial image features, and generates explicit spatial point memory units with unassigned three-dimensional positions based on the initial image features. The feature extraction module encodes the current frame image using an image encoder to extract the features of the current image. The feature interaction module uses an interactive decoder based on an image encoder to interact with the features in the explicit spatial point memory unit to obtain a global feature representation of the current image global coordinate system. The feature decoding module, based on the global feature representation, extracts the point map representation of the current frame image in its own coordinate system and the global coordinate system, as well as the corresponding camera pose information; The memory fusion module generates new spatial point memory units based on the features of the current frame image, the determined 3D point map representation, and the camera pose information; it then fuses the new spatial point memory units with existing explicit spatial point memory units, and updates the feature information of the relevant memory units through spatial position comparison.

Images