Three-dimensional map sharing system and method based on Gaussian representation
By constructing virtual auxiliary maps and using incremental data transmission technology, the problems of extrapolation view distortion and high data transmission pressure in the Gaussian map sharing system have been solved, achieving efficient view adaptation and data saving.
Patent Information
- Application Number
- CN202511267677.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-05
- Publication Date
- 2025-12-19
AI Technical Summary
Gaussian map sharing systems suffer from problems such as distorted extrapolation views and high data transmission pressure, making it difficult to adapt to user perspective needs and reduce data consumption.
By constructing a virtual auxiliary map, the rendering quality of the global anchor point Gaussian map in the non-training view is improved, and the full map is transmitted only to the first-time access area, while incremental map data is transmitted to the subsequent access areas.
It improves the rendering quality of extrapolated views, reduces data transmission volume, and enhances system transmission efficiency and user experience.
Smart Images
Figure CN121170179A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure belongs to the technical field of map sharing, and particularly relates to a three-dimensional map sharing system and method based on Gaussian representation. BACKGROUND
[0002] Three-dimensional (3D) maps are the core support for mobile navigation, logistics planning, and driverless vehicles. High-fidelity 3D maps require massive data to cover the complete 3D space, so crowdsourcing technology has become the mainstream implementation method: contributors upload regional data, which is aggregated by the server and downloaded by users, who then convert it into an easy-to-read format such as a two-dimensional image, forming a complete collaborative framework of "collection-processing-sharing".
[0003] In the development of three-dimensional maps, the map expression format directly determines the spatial feature restoration accuracy, storage efficiency, and terminal rendering effect. Among them, the Gaussian map uses Gaussian distribution as the basic unit to represent spatial features, which can not only preserve high-fidelity details but also achieve lightweight storage, and has gradually replaced traditional map formats (such as grid models and point clouds) to become the core carrier of high-fidelity maps. Therefore, building a "Gaussian map sharing system" has become an inevitable trend in the industry. Furthermore, the global anchor Gaussian map has become the mainstream application format of Gaussian maps due to its high compression rate and the ability to restore the corresponding Gaussian map without loss.
[0004] However, even based on the global anchor Gaussian map, the landing of the Gaussian map sharing system still faces two core technical bottlenecks:
[0005] 1. Extrapolated view distortion, difficulty in adapting to user perspective requirements
[0006] The user perspective often differs significantly from the original collection perspective (i.e., the "training perspective") of the contributor, and a new view synthesis technique is needed to generate an adapted "extrapolated view". Among them, the perspective that differs greatly from the training perspective is called the "non-training perspective", and the corresponding generated view is the "extrapolated view" - which is the key to meeting the diverse perspective requirements of the map sharing system. The current global Gaussian map has a training perspective overfitting problem in the training stage: the model excessively fits the features of the training perspective (contributor's collection perspective), resulting in insufficient generalization ability in the non-training perspective, making it difficult to accurately restore spatial features, and ultimately generating an extrapolated view that is prone to detail distortion, structural disorder, and other problems, which cannot meet the core requirement of perspective adaptability of the system.
[0007] 2. Large data transmission pressure, difficulty in adapting to user core requirements
[0008] The map sharing system needs to continuously transmit updated map data to users, and the core requirement of users for map services is to balance communication traffic saving and use experience guarantee: both to reduce traffic consumption and to avoid affecting use fluency due to transmission problems. Although the global anchor point Gaussian map has greatly reduced the amount of data through high compression rate, the current mode of "direct transmission of complete map file" still has significant defects: the transmission bandwidth between the server and the user is limited, the transmission of complete map not only takes a long time, but also generates high traffic cost, directly leading to the decline of user experience and the reduction of participation willingness, and finally restricting the practical application of the Gaussian map sharing system. SUMMARY
[0009] Embodiments of the present disclosure propose a three-dimensional map sharing scheme based on Gaussian representation to solve the problems of distortion of extrapolated view and large data transmission pressure faced by the Gaussian map sharing system in landing.
[0010] The first aspect of the embodiments of the present disclosure provides a three-dimensional map sharing system based on Gaussian representation, comprising a contributor and a user respectively connected with a server data.
[0011] The contributor is configured to obtain a color-depth map of a target area and upload it to the server.
[0012] The server is configured to calculate corresponding pose data based on the color-depth map and construct a global anchor point Gaussian map, wherein the global anchor point Gaussian map is composed of a voxel grid, an anchor point is set at the center of each grid, the anchor point is used to store the attribute information of all Gaussian spheres corresponding to the grid, and the anchor point feature of the anchor point can be obtained by decoding the Gaussian map corresponding to the grid through a double-layer fully connected layer.
[0013] The server is further configured to construct a virtual auxiliary map based on the color-depth map to improve the accuracy of rendering the extrapolated view of the global anchor point Gaussian map, wherein the virtual auxiliary map covers the same target area as the global anchor point Gaussian map and retains the color and depth information in the color-depth map.
[0014] The server is further configured to push the quantized global anchor point Gaussian map data and the corresponding first decoder to the user in response to the first access request of the user for the map data of the target area, or generate and push the quantized map incremental data and the corresponding second decoder to the user in response to the non-first access request of the user for the map data of the target area, wherein the map incremental data is used to represent the map update content, and the quantization method of the global anchor point Gaussian map data is consistent with the quantization method of the map incremental data.
[0015] The user receives the map data pushed by the server and the corresponding decoder, restores the complete Gaussian map, and converts it into a view format suitable for terminal display to support the user's interaction with the three-dimensional map and realize the interactive scene, wherein if the quantized global anchor point Gaussian map data is received, the first decoder is used to directly restore the complete Gaussian map, and if the quantized map incremental data is received, the second decoder is used to decode the quantized map incremental data and restore the complete Gaussian map in combination with the local stored expired map.
[0016] The second aspect of the embodiments of the present disclosure provides a three-dimensional map sharing method based on Gaussian representation, which is executed by the server in the system of claim 1, and comprises the following steps:
[0017] The color-depth map of the target area uploaded by the contributor is acquired, the corresponding pose data is calculated, and a global anchor point Gaussian map is constructed, which comprises the following steps: dividing the three-dimensional space of the target area into a voxel grid, setting an anchor point at the center position of each voxel grid, and configuring anchor point features for the anchor points, wherein the anchor point features encode the attribute information of all Gaussian spheres corresponding to the grid;
[0018] A virtual auxiliary map is constructed based on the color-depth map, a pseudo ground truth generated by the virtual auxiliary map is used to optimize the global anchor point Gaussian map, so as to improve the accuracy of the global anchor point Gaussian map in rendering an extrapolated view, wherein the virtual auxiliary map covers the same target area as the global anchor point Gaussian map and retains the color and depth information in the color-depth map;
[0019] In response to a first access request of the user for the map data of the target area, the quantized global anchor point Gaussian map data of the target area and the corresponding first decoder are pushed to the user, or in response to a non-first access request of the user for the map data of the target area, the quantized map incremental data and the corresponding second decoder are generated and pushed to the user, wherein the map incremental data is used to represent the map update content, and the quantization method of the global anchor point Gaussian map data is consistent with the quantization method of the map incremental data.
[0020] In some embodiments of the present disclosure, the anchor point features encode the attribute information of all Gaussian spheres corresponding to the grid, which comprises:
[0021] The color, transparency and rotation of the Gaussian sphere corresponding to the grid are encoded into a low-dimensional vector;
[0022] The anchor point features are composed of the scale of the Gaussian sphere, the offset of the Gaussian sphere relative to the anchor point position, and the low-dimensional vector.
[0023] In some embodiments of the present disclosure, the anchor features can be decoded by a double-layer fully connected layer, and the offset is superimposed to the anchor position to recover the Gaussian sphere position, to obtain the corresponding Gaussian map of the grid.
[0024] In some embodiments of the present disclosure, obtaining the color-depth map of the target region uploaded by the contributor, calculating the corresponding pose data, and constructing the global anchor Gaussian map include:
[0025] Obtaining the color-depth map of the target region uploaded by the contributor, calculating the corresponding pose data, and inputting the color-depth map and the pose data into a Gaussian map generation framework to generate the global anchor Gaussian map.
[0026] The training process of the Gaussian map generation framework includes:
[0027] Quantizing the global anchor Gaussian map, inputting the quantized global anchor Gaussian map into a first decoder, and decoding the Gaussian sphere from the first decoder, wherein the first decoder is a double-layer fully connected layer, and is used to decode the quantized global anchor Gaussian map into a Gaussian sphere.
[0028] Based on the pose data, rendering the Gaussian sphere output by the first decoder into a view of the input view by Gaussian sputtering technology, comparing the rendered view with the input view of the color-depth map, and calculating a first loss function based on the comparison result.
[0029] Adjusting the parameters of the first decoder and the anchor features with the first loss function as the optimization target until the training converges, and the first loss function is:
[0030]
[0031] wherein, and are the L1 loss and the structural similarity loss of the rendered picture and the original picture respectively, is a regularization term, is the L1 loss between the rendered depth map and the original depth map, is the normal vector loss, is the pruning loss.
[0032] In some embodiments of the present disclosure, obtaining the color-depth map of the target region uploaded by the contributor, calculating the corresponding pose data includes:
[0033] Based on the standard method of motion recovery structure, performing feature matching and three-dimensional reconstruction calculation on the color-depth map to obtain the pose data corresponding to the color-depth map.
[0034] In some embodiments of this disclosure, constructing a virtual auxiliary map based on the color-depth map includes:
[0035] The color-depth map is mapped onto three-dimensional space to generate a color point cloud;
[0036] The colored point cloud is then converted into a three-dimensional Gaussian map to generate the virtual auxiliary map.
[0037] In some embodiments of this disclosure, optimizing the global anchor Gaussian map using the pseudo-real values generated by the virtual auxiliary map to improve the accuracy of the extrapolated view rendered by the global anchor Gaussian map includes:
[0038] A virtual view corresponding to the non-training viewpoint pose is rendered on the virtual auxiliary map, and this virtual view is used as the pseudo-true value.
[0039] Obtain the rendered image of the global anchor point Gaussian map under the same non-training view pose;
[0040] The L1 loss of the virtual view and the rendered image is used as a second loss function for training to optimize the global anchor point Gaussian map, thereby improving its accuracy in rendering extrapolated views from the non-training perspective.
[0041] In some embodiments of this disclosure, generating incremental map data representing map update content includes:
[0042] The color-depth map of the target area at the current moment and the historical global anchor Gaussian map at the time of the most recent map update are input into the map incremental training framework to train map incremental data that represents the difference between the current global anchor Gaussian map and the historical global anchor Gaussian map.
[0043] The training method for the map incremental training framework includes:
[0044] Initialize the output parameter F of the map incremental training framework Inc If F is 0, then F Inc As learnable parameters, they are trained synchronously with the second decoder, which is a two-layer fully connected layer structure used to process the quantized F... Inc Decoded as a Gaussian sphere representing the map update content;
[0045] In each round of training, for F Inc Uniform noise is added to each element to simulate the quantization effect, resulting in the intermediate variable F′. Inc ;
[0046] The intermediate variable F′ IncThe Gaussian sphere, decoded to represent the map update content, is combined with the historical global anchor Gaussian map to form the current global anchor Gaussian map.
[0047] The input viewpoint of the color-depth map is determined, and the view of the current global anchor point Gaussian map under the input viewpoint is rendered using Gaussian sputtering technology. This rendered view is compared with the real view of the input viewpoint, and a third loss function is calculated based on the comparison result to optimize the parameters of the second decoder and the learnable parameter F. Inc ;
[0048] The third loss function is:
[0049] L updt =λ q L entropy +(L t +L v ),
[0050] Where: L entropy It is the quantization of information entropy, and the quantization method of the quantized information entropy is consistent with the quantization method used in the global anchor point Gaussian map, λ q It is the loss weight, L t It is the first loss function, L v It is the second loss function.
[0051] A third aspect of this disclosure provides a method for sharing three-dimensional maps based on Gaussian representation, executed by the user in the system of claim 1, comprising:
[0052] Receiving map data and corresponding decoders pushed by the server and restoring them to a complete Gaussian map includes: if quantized global anchor point Gaussian map data is received, then the complete Gaussian map is directly restored through the first decoder; if quantized map incremental data is received, then the quantized map incremental data is decoded through the second decoder and restored to a complete Gaussian map in combination with locally stored expired maps.
[0053] The complete Gaussian map is converted into a view format adapted for terminal display to support user interaction with the 3D map.
[0054] In summary, this disclosure proposes a Gaussian representation-based 3D map sharing system and a Gaussian representation-based 3D map sharing method. By using a non-trained viewpoint rendered by a virtual assisted map as a pseudo-ground value to train a global anchor Gaussian map, the overfitting phenomenon of the training viewpoint during the global anchor Gaussian map training process can be significantly reduced, thereby improving the quality of the extrapolated view of the global anchor Gaussian map rendering. Simultaneously, because only updated incremental map data is transmitted for target areas visited by the user, the amount of data that needs to be transmitted is greatly reduced. The combination of these two aspects makes the implementation of the Gaussian map sharing system possible. Attached Figure Description
[0055] The features and advantages of this disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the scope of this disclosure in any way.
[0056] Figure 1 This is a schematic diagram of the proposed 3D map sharing framework based on Gaussian representation.
[0057] Figure 2 This is a schematic diagram of a Gaussian-based 3D map sharing system according to some embodiments of the present disclosure;
[0058] Figure 3 This is a flowchart illustrating a Gaussian representation-based three-dimensional map sharing method according to some embodiments of this disclosure;
[0059] Figure 4 This is a flowchart illustrating a Gaussian representation-based three-dimensional map sharing method according to some embodiments of this disclosure. Detailed Implementation
[0060] In the following detailed description, numerous specific details of this disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this disclosure are a method of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if they can achieve the same purpose.
[0061] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to, or communicate with, other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used in this disclosure includes any one and all combinations of one or more of the associated listed items.
[0062] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As shown in this specification and claims, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified features, integrals, steps, operations, elements, and / or components, and such expressions do not constitute an exclusive list, in which other features, integrals, steps, operations, elements, and / or components may also be included.
[0063] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this disclosure can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this disclosure. It is understood that the drawings are not drawn to scale.
[0064] Various structural diagrams are used in this disclosure to illustrate various variations of embodiments according to this disclosure. It should be understood that the preceding or following structures are not intended to limit this disclosure. The scope of protection of this disclosure is defined by the claims.
[0065] 3D maps are a core support for mobile navigation, logistics planning, and autonomous driving. High-fidelity 3D maps require massive amounts of data to cover the entire 3D space, so crowdsourcing technology has become the mainstream implementation method: contributors upload regional data, which is then aggregated by the server and made available for users to download. Users then convert the data into easily readable formats such as 2D images, forming a complete collaborative framework of "collection-processing-sharing".
[0066] In the development of 3D maps, the map representation format directly determines the accuracy of spatial feature reconstruction, storage efficiency, and terminal rendering effects. Gaussian maps, using the Gaussian distribution as the basic unit to represent spatial features, can retain high-fidelity details while achieving lightweight storage. They have gradually replaced traditional map formats (such as grid models and point clouds) and become the core carrier of high-fidelity maps, making the construction of a "Gaussian map sharing system" an inevitable industry trend. Furthermore, global anchor point Gaussian maps, with their core advantages of high compression ratio and near-lossless reconstruction of the corresponding Gaussian map, have become the mainstream application format for Gaussian maps.
[0067] However, even with a global anchor point Gaussian map, the implementation of a Gaussian map sharing system still faces two major technical bottlenecks:
[0068] 1. Extrapolated views are prone to distortion and are difficult to adapt to the user's perspective.
[0069] The user's perspective often differs significantly from the contributor's original acquisition perspective (i.e., the "training perspective"), necessitating the generation of adapted "extrapolated views" through new view synthesis techniques. The perspective that differs significantly from the training perspective is called the "non-training perspective," and the corresponding generated view is the "extrapolated view"—this is crucial for map sharing systems to meet the diverse perspective needs of users. Currently, global Gaussian maps suffer from overfitting during the training phase: the model excessively fits the characteristics of the training perspective (contributor's acquisition perspective), resulting in insufficient generalization ability under non-training perspectives, difficulty in accurately reproducing spatial features, and ultimately, the generated extrapolated view is prone to detail distortion and structural inconsistencies, failing to meet the system's core requirement for perspective adaptability.
[0070] 2. High data transmission pressure, making it difficult to adapt to core user needs.
[0071] Map sharing systems need to continuously transmit updated map data to users, and users' core demand for map services lies in balancing communication bandwidth conservation and user experience assurance: they want to reduce bandwidth consumption while avoiding disruptions to smooth operation due to transmission issues. Although global anchor Gaussian maps have significantly reduced data volume through high compression rates, the current model of "directly transmitting complete map files" still has significant drawbacks: the transmission bandwidth between the server and the user is limited, and transmitting the complete map is not only time-consuming but also incurs high bandwidth costs, directly leading to a decline in user experience and reduced willingness to participate, ultimately hindering the practical implementation of Gaussian map sharing systems.
[0072] In view of this, this disclosure proposes a 3D map sharing framework based on Gaussian representation, such as... Figure 1As shown, the framework improves the rendering quality of the Gaussian map in the extrapolation view from the non-training perspective by constructing a virtual auxiliary map, thereby improving the overall performance of the shared map; at the same time, it improves the transmission efficiency of the system by transmitting the full map only for the first-time access area and the incremental map for subsequent access areas.
[0073] Figure 2 This is a schematic diagram of a Gaussian-based 3D map sharing system according to some embodiments of the present disclosure. Figure 2 The illustrated 3D map sharing system includes contributors and users who are connected to the server data, respectively, where:
[0074] The contributor is responsible for acquiring a color-depth map of the target area and uploading it to the server.
[0075] The server is used to calculate the corresponding pose data based on the color-depth map and construct a global anchor point Gaussian map. The global anchor point Gaussian map is composed of a voxel grid, with an anchor point set at the center of each grid. The anchor point is used to store the attribute information of all Gaussian spheres that correspond to the grid. The anchor point features can be decoded by a two-layer fully connected layer to obtain the Gaussian map corresponding to the grid.
[0076] The server is also used to construct a virtual auxiliary map based on the color-depth map to improve the accuracy of the extrapolation view of the global anchor Gaussian map rendering, wherein the virtual auxiliary map covers the same target area as the global anchor Gaussian map and retains the color and depth information in the color-depth map;
[0077] The server is also configured to respond to a user's first access request for map data of a target area by pushing the quantized global anchor Gaussian map data of the target area and the corresponding first decoder to the user, or to respond to a user's non-first access request for map data of the target area by generating and pushing the quantized map incremental data and the corresponding second decoder to the user. The map incremental data is used to represent the map update content, and the quantization method of the global anchor Gaussian map data is consistent with the quantization method of the map incremental data.
[0078] The user is configured to receive map data and corresponding decoders pushed by the server and restore them to a complete Gaussian map, which is then converted into a view format adapted to the terminal display to support the user's interaction with the 3D map. Specifically, if the quantized global anchor point Gaussian map data is received, it is directly restored to a complete Gaussian map through the first decoder. If the quantized map incremental data is received, it is decoded by the second decoder and restored to a complete Gaussian map by combining it with locally stored expired maps.
[0079] The server can be a standalone, clustered, or distributed server; the user is a variety of smart terminal devices, such as personal digital assistants (PDAs), smartphones, and personal computers (PCs); the contributor is a variety of terminal devices capable of acquiring RGB-D images (color-depth maps).
[0080] Figure 3 This is a flowchart illustrating a Gaussian representation-based 3D map sharing method according to some embodiments of the present disclosure. In some embodiments, the Gaussian representation-based 3D map sharing method is... Figure 2 The server shown executes the method, which includes the following steps:
[0081] S310, Obtain the color-depth map of the target area uploaded by the contributor, calculate the corresponding pose data and construct a global anchor point Gaussian map, including: dividing the three-dimensional space of the target area into a voxel grid, setting an anchor point at the center of each voxel grid, configuring anchor point features for the anchor point, and encoding the attribute information of all Gaussian spheres corresponding to the grid.
[0082] The server first calculates the corresponding pose based on the RGB-D image. Some embodiments of this disclosure obtain pose information from the RGB-D image based on the standard SFM (Structure from Motion) method.
[0083] The RGB-D image and its corresponding pose are then input into the Gaussian map generation framework to generate a global anchor point Gaussian map. Global anchor point Gaussian maps are currently the mainstream application format for Gaussian maps, offering advantages such as high compression ratios and the ability to reconstruct the corresponding Gaussian map with virtually no loss.
[0084] Each Gaussian sphere in an existing Gaussian map has five attributes: color, transparency, rotation, position, and scale. This disclosure builds upon this to create a global anchor point Gaussian map. This map is based on a voxel grid, with a central anchor point set within each voxel grid. The anchor point feature F... anc It possesses properties similar to a Gaussian sphere: for color, transparency, and rotation properties, this disclosure encodes them into a low-dimensional vector F. emb In the original Gaussian map, the remaining two geometrically related attributes, location and scale, retain their representation as in the original Gaussian map. Therefore, an anchor point feature can be expressed as:
[0085] F anc ={F S F O ,F emb}
[0086] Among them, F S For the scale, F OThis is the offset of the Gaussian sphere represented by the anchor point relative to the anchor point's position.
[0087] When needed, anchor point feature F anc It can be decoded back into a Gaussian sphere. Specifically, this is achieved using a two-layer fully connected network to convert the offset F... o It is superimposed on the anchor point to restore the position of the Gaussian sphere.
[0088] In some embodiments of this disclosure, the generation of the global anchor point Gaussian map and the training of the global map decoder are completed simultaneously, including:
[0089] Use an RGB-D image as input;
[0090] The global anchor Gaussian map is quantized, and the quantized global anchor Gaussian map is input into a global map decoder, which decodes it to obtain a Gaussian sphere. In some embodiments of this disclosure, the global map decoder is a two-layer fully connected layer used to decode the quantized global anchor Gaussian map into a Gaussian sphere;
[0091] The Gaussian sphere is rendered into a view from the input perspective using Gaussian sputtering technology. This rendered view is compared with the input view of the RGB-D image, and a first loss function is calculated based on the comparison result.
[0092] Using the first loss function as the optimization objective, the parameters of the global map decoder and the anchor features of the global anchor Gaussian map are adjusted until training converges. The first loss function is:
[0093]
[0094] in, and These are the L1 loss and structural similarity loss for the rendered image and the original image, respectively. It is a regularization term. It is the L1 loss between the rendered depth map and the original depth map. It is the normal vector loss. It's the loss from pruning.
[0095] Pruning losses This invention is a key loss term used to encourage the removal of redundant Gaussian spheres during the optimization of global anchor point Gaussian maps. Its core objective is to effectively reduce map storage and computational overhead while ensuring map representation accuracy. The specific design logic and mechanism are as follows:
[0096] The basis for pruning is that during the construction of the global anchor point Gaussian map, each anchor point initially corresponds to a number of Gaussian spheres (original quantity). By setting pruning conditions (such as thresholds for attributes like transparency and scale of Gaussian spheres), redundant Gaussian spheres that contribute little to the scene representation can be marked and deleted. The ratio of the remaining number of Gaussian spheres to the original number is the quantitative basis for pruning loss.
[0097] By introducing pruning loss This disclosure can significantly reduce the total amount of Gaussian spheres in a global anchor Gaussian map while ensuring map representation accuracy, thereby effectively reducing the space overhead required for map storage, while reducing the consumption of computing resources in subsequent rendering and transmission processes, ultimately improving the efficiency and practicality of 3D map sharing.
[0098] Please note that the Gaussian sputtering technique used in this disclosure preferentially employs two-dimensional Gaussian sputtering. Three-dimensional Gaussian sputtering, commonly used in uncompressed Gaussian maps, does not produce ideal results when rendering extrapolated views of globally anchored Gaussian maps with compressed representations. In comparison, two-dimensional Gaussian sputtering demonstrates superior consistency in extrapolated views when compressing data representations.
[0099] This disclosure presents a general quantization method based on Gaussian maps to quantize global anchor point Gaussian maps.
[0100] S320, a virtual auxiliary map is constructed based on the color-depth map. The pseudo-real values generated by the virtual auxiliary map are used to optimize the global anchor Gaussian map to improve the accuracy of the extrapolation view rendered by the global anchor Gaussian map. The virtual auxiliary map covers the same target area as the global anchor Gaussian map and retains the color and depth information in the color-depth map.
[0101] The global anchor point Gaussian map generated in S310 only includes features from the training view (contributor-collected view) during training, and its generalization ability for non-training views is insufficient.
[0102] To this end, this disclosure constructs a virtual auxiliary map based on the RGB-D image, renders a non-training viewpoint using the virtual auxiliary map, and uses this as the ground truth to supervise the training of a global anchor point Gaussian map, thereby eliminating the overfitting problem to the training viewpoint data during the training process. Specifically:
[0103] This disclosure maps an RGB-D image to a 3D space to generate a color point cloud, then converts it into a 3D Gaussian map as a virtual auxiliary map; random sampling is performed throughout the 3D space, discarding sampling poses similar to the input viewpoint, and finally rendering the image on the virtual auxiliary map. This refers to a virtual view. Utilizing a virtual view... The L1 loss between the rendered image and the rendered image from the global anchor Gaussian map at the same pose is used as the loss function L. v Training enhances the overall map quality.
[0104] S330, in response to the user's first access request for the target area map data, push the quantized global anchor point Gaussian map data of the target area and the corresponding first decoder to the user, or in response to the user's non-first access request for the target area map data, generate and push the quantized map incremental data and the corresponding second decoder to the user. The map incremental data is used to represent the map update content, and the quantization method of the global anchor point Gaussian map data is the same as the quantization method of the map incremental data.
[0105] This disclosure divides the global map into an initial access area and a re-access area. For the initial access area, this disclosure transmits the complete map, that is, the Gaussian anchor point map constructed according to the method described in S310-S320 is quantized and directly transmitted to the user. Specifically, the transmitted information is the quantized F... anc And the corresponding global map decoder, and then the F is decoded and quantized on the user device based on the global map decoder. anc To restore Gaussian map G S .
[0106] For revisited areas, map updates will be initiated to improve these areas as the system runs. To facilitate low-bandwidth map updates, this disclosure updates the map incrementally. Specifically, the transmitted information consists of quantized incremental map data and the corresponding incremental data decoder. Then, on the user's device, the quantized incremental map data is decoded based on the incremental data decoder, and combined with locally stored historical maps to reconstruct the complete Gaussian map G. S .
[0107] Map Incremental Data F Inc The representation method is the same as that of anchor point features. However, the process of converting it into a Gaussian sphere is slightly different. First, it is quantized, and then decoded into a Gaussian sphere. The formal description of the steps is as follows:
[0108] F′ Inc =AQM(F Inc ),
[0109] ΔG S =Decode(F′ Inc ),
[0110] Where S represents the number of map updates, starting from 0 and incrementing by 1 for each update, AQM is the standard adaptive quantization module, and F′ IncThis represents the quantized mapping increment. The decoded Gaussian increment ΔG S This will be compared with the expired map G from the previous stage. S-1 By combining these methods, we obtain the updated map G. S .
[0111] The quantization described in this disclosure refers to the process of mapping continuous values of Gaussian sphere attribute information to finite discrete values. Its main function is to effectively reduce storage and computational overhead while maintaining the accuracy of Gaussian map representation as much as possible. The quantization method for incremental map data in this disclosure is the same as the quantization method for global anchor point Gaussian maps, and both can adopt the general quantization method for Gaussian maps.
[0112] This disclosure addresses map increment F. Inc The generation of data and the training of the incremental data decoder are completed simultaneously, specifically including:
[0113] The training inputs are RGB-D images and historical global anchor point Gaussian maps.
[0114] Initialize F Inc If F is 0, then F Inc As learnable parameters, they are trained synchronously with the incremental data decoder, which is a two-layer fully connected layer structure used to process the quantized F... Inc Decoded as a Gaussian sphere representing the map update content;
[0115] In each round of training, for F Inc Uniform noise is added to each element to simulate the quantization effect, resulting in the intermediate variable F′. Inc F′ Inc The Gaussian sphere, decoded to represent the map update content, is combined with the historical global anchor Gaussian map to form the current global anchor Gaussian map.
[0116] The input viewpoint of the RGB-D image is determined, and a view of the current global anchor point Gaussian map under that input viewpoint is rendered using Gaussian sputtering technology. This rendered view is compared with the real view of the input viewpoint, and a rate distortion loss function is calculated based on the comparison result to optimize the parameters of the incremental data decoder and the learnable parameter F. Inc ;
[0117] The rate distortion loss function is:
[0118] L updt =λ q L entropy +(L t +L v ),
[0119] Where: L entropyIt is the quantization of information entropy, and the quantization method of the quantized information entropy is consistent with the quantization method used in the global anchor point Gaussian map, λ q It is the loss weight.
[0120] Figure 4 This is a flowchart illustrating a Gaussian representation-based 3D map sharing method according to some embodiments of the present disclosure. In some embodiments, the Gaussian representation-based 3D map sharing method is... Figure 2 The method, as shown in the user demonstration, includes the following steps:
[0121] S410, receiving map data and corresponding decoder pushed by the server and restoring it to a complete Gaussian map, including: if receiving quantized global anchor point Gaussian map data, directly restoring it to a complete Gaussian map through the first decoder; if receiving the quantized map incremental data, decoding the quantized map incremental data through the second decoder and restoring it to a complete Gaussian map in combination with locally stored expired maps.
[0122] In this disclosure, the user device determines how to recover the complete map based on the received map data.
[0123] Specifically, if the received data consists of quantized global Gaussian anchor point map data and a corresponding global map decoder, then the quantized global Gaussian anchor point map data is decoded based on the global map decoder to recover the complete Gaussian map G. S If the received data is quantized map incremental data and the corresponding incremental data decoder, then the quantized map incremental data is decoded based on the incremental data decoder, and then combined with the locally stored historical map to recover the complete Gaussian map G. S .
[0124] S420, the complete Gaussian map is converted into a view format adapted to the terminal display to support user interaction with the 3D map.
[0125] In summary, this disclosure proposes a Gaussian representation-based 3D map sharing system and a Gaussian representation-based 3D map sharing method. By using a non-trained viewpoint rendered by a virtual assisted map as a pseudo-ground value to train a global anchor Gaussian map, the overfitting phenomenon of the training viewpoint during the global anchor Gaussian map training process can be significantly reduced, thereby improving the quality of the extrapolated view of the global anchor Gaussian map rendering. Simultaneously, because only updated incremental map data is transmitted for target areas visited by the user, the amount of data that needs to be transmitted is greatly reduced. The combination of these two aspects makes the implementation of the Gaussian map sharing system possible.
[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.
[0127] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.
[0128] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.
[0129] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this disclosure and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this disclosure should be included within the protection scope of this disclosure. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.
Claims
1. A 3D map sharing system based on Gaussian representation, comprising contributors and users respectively connected to a server for data sharing, wherein: The contributor is responsible for acquiring a color-depth map of the target area and uploading it to the server. The server is used to calculate the corresponding pose data based on the color-depth map and construct a global anchor point Gaussian map. The global anchor point Gaussian map is composed of a voxel grid, with an anchor point set at the center of each grid. The anchor point is used to store the attribute information of all Gaussian spheres that correspond to the grid. The anchor point features can be decoded by a two-layer fully connected layer to obtain the Gaussian map corresponding to the grid. The server is also used to construct a virtual auxiliary map based on the color-depth map to improve the accuracy of the extrapolation view of the global anchor Gaussian map rendering, wherein the virtual auxiliary map covers the same target area as the global anchor Gaussian map and retains the color and depth information in the color-depth map; The server is also used to respond to a user's first access request for map data of the target area, and push the quantized global anchor Gaussian map data of the target area and the corresponding first decoder to the user, or to respond to a user's non-first access request for map data of the target area, and generate and push the quantized map incremental data and the corresponding second decoder to the user. The map incremental data is used to represent the map update content, and the quantization method of the global anchor Gaussian map data is the same as the quantization method of the map incremental data. The user is configured to receive map data and corresponding decoders pushed by the server and restore them to a complete Gaussian map, which is then converted into a view format adapted to the terminal display to support the user's interaction with the 3D map. Specifically, if the quantized global anchor point Gaussian map data is received, it is directly restored to a complete Gaussian map through the first decoder. If the quantized map incremental data is received, it is decoded by the second decoder and restored to a complete Gaussian map by combining it with locally stored expired maps.
2. A method for sharing three-dimensional maps based on Gaussian representation, executed by the server in the system of claim 1, characterized in that, include: Obtain the color-depth map of the target area uploaded by the contributor, calculate the corresponding pose data and construct a global anchor point Gaussian map, including: dividing the three-dimensional space of the target area into a voxel grid, setting an anchor point at the center of each voxel grid, configuring anchor point features for the anchor point, and encoding the attribute information of all Gaussian spheres corresponding to the grid. A virtual auxiliary map is constructed based on the color-depth map. The pseudo-real values generated by the virtual auxiliary map are used to optimize the global anchor Gaussian map, so as to improve the accuracy of the extrapolation view rendered by the global anchor Gaussian map. The virtual auxiliary map covers the same target area as the global anchor Gaussian map and retains the color and depth information in the color-depth map. In response to a user's first access request to map data of a target area, the system pushes quantized global anchor Gaussian map data of the target area and the corresponding first decoder to the user; or in response to a user's subsequent access request to map data of the target area, the system generates and pushes quantized map incremental data and the corresponding second decoder to the user. The map incremental data is used to represent map update content, and the quantization method of the global anchor Gaussian map data is consistent with the quantization method of the map incremental data.
3. The method according to claim 2, characterized in that, The anchor point feature encoding integrates the attribute information of all Gaussian spheres corresponding to the grid, including: Encode the color, transparency, and rotation of the Gaussian sphere corresponding to the grid into a low-dimensional vector; The anchor point feature is composed of the scale of the Gaussian sphere, the offset of the Gaussian sphere relative to the anchor point position, and the low-dimensional vector.
4. The method according to claim 3, characterized in that: The anchor point features can be decoded through a two-layer fully connected layer, and the offset can be superimposed on the anchor point position to recover the Gaussian sphere position, thereby obtaining the corresponding Gaussian map of the grid.
5. The method according to claim 4, characterized in that, Obtain the color-depth map of the target area uploaded by the contributor, calculate the corresponding pose data, and construct a global anchor point Gaussian map, including: Obtain the color-depth map of the target area uploaded by the contributor, calculate the corresponding pose data, and input the color-depth map and pose data into the Gaussian map generation framework to generate the global anchor point Gaussian map. The training process of the Gaussian map generation framework includes: The global anchor Gaussian map is quantized, and the quantized global anchor Gaussian map is input into the first decoder. The first decoder decodes the Gaussian sphere to obtain the Gaussian sphere. The first decoder is a two-layer fully connected layer used to decode the quantized global anchor Gaussian map into the Gaussian sphere. Based on the pose data, the Gaussian sphere output by the first decoder is rendered as a view from the input perspective using Gaussian sputtering technology. The rendered view is compared with the input view of the color-depth map, and the first loss function is calculated based on the comparison result. Using the first loss function as the optimization objective, adjust the parameters of the first decoder and the anchor features until training converges. The first loss function is: in, and These are the L1 loss and structural similarity loss for the rendered image and the original image, respectively. It is a regularization term. It is the L1 loss between the rendered depth map and the original depth map. It is the normal vector loss. It's the loss from pruning.
6. The method according to claim 5, characterized in that, The step of obtaining the target region color-depth map uploaded by the contributor and calculating the corresponding pose data includes: Based on the standard method of motion recovery structure, feature matching and 3D reconstruction calculations are performed on the color-depth map to obtain the pose data corresponding to the color-depth map.
7. The method according to claim 6, characterized in that, The construction of the virtual auxiliary map based on the color-depth map includes: The color-depth map is mapped onto three-dimensional space to generate a color point cloud; The colored point cloud is then converted into a three-dimensional Gaussian map to generate the virtual auxiliary map.
8. The method according to claim 7, characterized in that, The step of optimizing the global anchor Gaussian map using the pseudo-real values generated by the virtual auxiliary map to improve the accuracy of the extrapolated view rendered by the global anchor Gaussian map includes: A virtual view corresponding to the non-training viewpoint pose is rendered on the virtual auxiliary map, and this virtual view is used as the pseudo-true value. Obtain the rendered image of the global anchor point Gaussian map under the same non-training view pose; The L1 loss of the virtual view and the rendered image is used as a second loss function for training to optimize the global anchor point Gaussian map, thereby improving its accuracy in rendering extrapolated views from the non-training perspective.
9. The method according to claim 8, characterized in that, The incremental map data that generates the map update content includes: The color-depth map of the target area at the current moment and the historical global anchor Gaussian map at the time of the most recent map update are input into the map incremental training framework to train map incremental data that represents the difference between the current global anchor Gaussian map and the historical global anchor Gaussian map. The training method for the map incremental training framework includes: Initialize the output parameter F of the map incremental training framework Inc If F is 0, then F Inc As learnable parameters, they are trained synchronously with the second decoder, which is a two-layer fully connected layer structure used to process the quantized F... Inc Decoded as a Gaussian sphere representing the map update content; In each round of training, for F Inc Uniform noise is added to each element to simulate the quantization effect, resulting in the intermediate variable F′. Inc ; The intermediate variable F′ Inc The Gaussian sphere, decoded to represent the map update content, is combined with the historical global anchor Gaussian map to form the current global anchor Gaussian map. The input viewpoint of the color-depth map is determined, and the view of the current global anchor point Gaussian map under the input viewpoint is rendered using Gaussian sputtering technology. This rendered view is compared with the real view of the input viewpoint, and a third loss function is calculated based on the comparison result to optimize the parameters of the second decoder and the learnable parameter F. Inc ; The third loss function is: L updt =λ q L entropy +(L t +Lv ) , Where: L entropy It is the quantization of information entropy, and the quantization method of the quantized information entropy is consistent with the quantization method used in the global anchor point Gaussian map, λ q It is the loss weight, L t It is the first loss function, L v It is the second loss function.
10. A method for sharing three-dimensional maps based on Gaussian representation, executed by the user in the system of claim 1, characterized in that, include: Receiving map data and corresponding decoders pushed by the server and restoring them to a complete Gaussian map includes: if quantized global anchor point Gaussian map data is received, then the complete Gaussian map is directly restored through the first decoder; if quantized map incremental data is received, then the quantized map incremental data is decoded through the second decoder and restored to a complete Gaussian map in combination with locally stored expired maps. The complete Gaussian map is converted into a view format adapted for terminal display to support user interaction with the 3D map.