A 3DGS semantic field decoupling reconstruction method based on hierarchical compression

Abstract

The application discloses a 3DGS semantic field decoupling reconstruction method based on hierarchical compression. The method comprises the following steps: acquiring a plurality of multi-view images; inputting the plurality of multi-view images into a 2D basic model to obtain a semantic feature map of each multi-view image; obtaining an initial semantic feature codebook based on the semantic feature map of the first multi-view image, wherein the initial semantic feature codebook comprises a plurality of feature vectors, and each feature vector contains an initial index value; determining a target quantization feature map and a target semantic feature codebook of each semantic feature map based on each semantic feature map and the initial semantic feature codebook. The application solves the technical problem that in the prior art, high-dimensional semantic features and a large number of Gaussian balls make the reconstruction process require a large amount of computing power and storage, and weaken the performance of 3DGS.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a 3D GS semantic field decoupling reconstruction method based on hierarchical compression. BACKGROUND

[0002] In the field of computer vision, 3D semantic field reconstruction is a very valuable technology with wide applications. This technology aims to assign semantic information to objects in three-dimensional space, enabling computers to have a deeper understanding of the scene. It promotes the development of intelligent robots, autonomous driving, augmented reality (AR), virtual reality (VR), urban planning and architectural design, and smart homes, among other fields. In robotics, 3D semantic fields enable robots to better understand their environment, enabling efficient navigation and safe operation. In the field of autonomous driving, this technology enhances the vehicle's perception of the surrounding environment, improving driving safety. For AR and VR, 3D semantic fields facilitate the seamless integration of virtual elements into the real world, enhancing user immersion and interaction experience. In urban planning and architectural design, it supports more accurate spatial analysis and scheme presentation, speeding up the design process. In addition, the smart home field also benefits from 3D semantic fields, enabling intelligent arrangement and environmental adaptation of home devices. With the continuous advancement of artificial intelligence and computer vision technology, the application of 3D semantic fields will become more widespread, not only improving the performance of existing systems, but also driving new application scenarios and service models, and promoting the innovative development of related industries.

[0003] 2D Vision Foundation Models (2DVFM) can extract rich semantic features through large-scale image data pre-training, which exhibit significant advantages in multiple aspects: First, 2DVFM has strong generalization ability, which can adapt to multiple visual tasks, and can quickly fine-tune to achieve high performance even when faced with new tasks. Second, these models have high robustness, which can effectively deal with complex scenes such as light changes, perspective transformations, and occlusions, ensuring stability and reliability in practical applications. In addition, 2DVFM supports multi-modal fusion, which can combine information from different modalities such as images and text, enhancing the model's expression ability and application range. Migrating the capabilities of 2DVFM to 3D scenes has become a very meaningful research direction.

[0004] 3D Gaussian Splatting (3DGS) is the latest 3D reconstruction technology after NeRF, which has the advantages of high reconstruction quality, short training time, and fast rendering speed compared to NeRF, and has become a research hotspot in the field of 3D.

[0005] 3DGS was originally designed for color field reconstruction, each 3D Gaussian contains the following attributes: position , scale , rotation quaternion , transparency and color . These parameters represent the spatial and visual characteristics of 3D Gaussian in three-dimensional space, and the rendering of a certain camera pose is realized by using differentiable rasterization R to obtain the image.

[0006] In order to better understand the scene, researchers have tried to use 3DGS and 2DVFM to reconstruct the semantic field, in order to embed semantic information in 3DGS, some methods such as Feature3DGS introduce a semantic feature attribute for each 3D Gaussian based on the original 3DGS, so that it can be optimized and learned with the original and other parameters, and can render image and semantic feature map at the same time; but high-dimensional semantic features and a large number of Gaussian spheres make the reconstruction process require a large amount of computing power and storage, which greatly weakens the performance of 3DGS, therefore, the application proposes a 3DGS semantic field decoupling reconstruction method based on hierarchical compression, which makes the 3D semantic field reconstruction more efficient. SUMMARY

[0007] The embodiment of the application provides a 3DGS semantic field decoupling reconstruction method based on hierarchical compression, which at least solves the technical problem that high-dimensional semantic features and a large number of Gaussian spheres make the reconstruction process require a large amount of computing power and storage, which greatly weakens the performance of 3DGS in the prior art.

[0008] According to an aspect of the embodiment of the application, a 3DGS semantic field decoupling reconstruction method based on hierarchical compression is provided. The method can include: obtaining a plurality of multi-view images, inputting the plurality of multi-view images into a 2D basic model to obtain a semantic feature map of each multi-view image; based on the semantic feature map of the first multi-view image, obtaining an initial semantic feature codebook, wherein the initial semantic feature codebook includes a plurality of feature vectors, and each feature vector contains an initial index value; based on each semantic feature map and the initial semantic feature codebook, determining a target quantized feature map and a target semantic feature codebook of each semantic feature map; training an autoencoder through the target semantic feature codebook to obtain a target encoder, wherein the autoencoder includes an initial encoder and an initial decoder; based on the target encoder and the target quantized feature map of each semantic feature map, obtaining a latent feature map of each semantic feature map; obtaining an initial 3DGS semantic field; based on the latent feature map of each semantic feature map and the initial 3DGS semantic field, obtaining a target 3DGS semantic field.

[0009] Optionally, the initial semantic feature codebook is obtained based on the semantic feature map of the first multi-view image, including: sampling a target number of feature vectors from the semantic feature map of the first multi-view image by farthest point sampling, as the initial semantic feature codebook.

[0010] Optionally, the target quantized feature map and the target semantic feature codebook of each semantic feature map are determined based on each semantic feature map and the initial semantic feature codebook, including: performing feature search on the initial semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map, to obtain a first quantized feature map of the first semantic feature map, wherein the first quantized feature map contains a plurality of first index values; optimizing the initial semantic feature codebook based on the first semantic feature map and the first quantized feature map, to obtain an optimized initial semantic feature codebook; taking the optimized initial semantic feature codebook as the initial semantic feature codebook of the second semantic feature map, and repeating iteration to obtain an optimized initial semantic feature codebook of the last semantic feature map; taking the optimized initial semantic feature codebook of the last semantic feature map as the target semantic feature codebook; and obtaining the target quantized feature map of each semantic feature map based on the target semantic feature codebook and each semantic feature map.

[0011] Optionally, the initial semantic feature codebook is optimized based on the first semantic feature map and the first quantized feature map, to obtain an optimized initial semantic feature codebook, including: obtaining a first semantic feature codebook based on the feature vectors of the first semantic feature map and the feature vectors of the first quantized feature map; obtaining a second quantized feature map of the first semantic feature map based on the first semantic feature codebook and the first semantic feature map, wherein the second quantized feature map includes a plurality of second index values; obtaining a target loss value based on the second quantized feature map, the first semantic feature map, the first index values of the second quantized feature map, and the initial index values of the initial semantic feature codebook; and optimizing the first semantic feature codebook based on the target loss value, to obtain the optimized initial semantic feature codebook.

[0012] Optionally, the first semantic feature codebook is obtained based on the feature vectors of the first semantic feature map and the feature vectors of the first quantized feature map, including: when the cosine similarity between the feature vectors of the first semantic feature map and the feature vectors of the first quantized feature map at a target coordinate is less than a first hyperparameter, adding the feature vector of the first semantic feature map at the target coordinate to the initial semantic feature codebook to obtain a second semantic feature codebook, wherein the second semantic feature codebook includes a plurality of second semantic index values; and when the cosine similarity between any two feature vectors corresponding to different second semantic index values in the second semantic feature codebook is less than a second hyperparameter, deleting the feature vector with the larger second semantic index value to obtain the first semantic feature codebook.

[0013] Optionally, the obtaining the second quantization feature map of the first semantic feature map based on the first semantic feature codebook and the first semantic feature map comprises: performing feature search on the first semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map to obtain the second quantization feature map of the first semantic feature map.

[0014] Optionally, the training the auto-encoder by the target semantic feature codebook to obtain the target encoder comprises: performing dimension compression on the target semantic feature codebook by the initial encoder to obtain compressed features; performing reconstruction on the compressed features by the initial decoder to obtain reconstructed features; and in the process of obtaining the reconstructed features, when a loss function between the reconstructed features and the target semantic feature codebook is the minimum, the training of the auto-encoder is completed, and the target encoder is obtained.

[0015] Optionally, the obtaining the latent feature map of each semantic feature map based on the target encoder and the target quantization feature map of each semantic feature map comprises: performing inference on the target quantization feature map of each semantic feature map by the target encoder to obtain the latent feature map of each semantic feature map.

[0016] Optionally, the obtaining the target 3DGS semantic field based on the latent feature map of each semantic feature map and the initial 3DGS semantic field comprises: training the latent feature map of each semantic feature map by the initial 3DGS semantic field to obtain the target 3DGS semantic field.

[0017] The beneficial effects of the present application are as follows:

[0018] The present application provides a 3DGS semantic field decoupling reconstruction method based on hierarchical compression, which firstly extracts semantic feature maps of multi-view images using 2DVFM, proposes a hierarchical compression technology to construct and quantize the extracted semantic feature maps, then uses an auto-encoder to compress the core semantic features in the codebook, the compressed core semantic features combine with the quantized index map to construct a new multi-view latent feature map, and finally uses the latent feature map to train the 3DGS without color attributes, thereby completing the reconstruction of the semantic field, wherein the semantic field and the color field are decoupled, and the semantic field is independently reconstructed, since the decoupled semantic field is not affected by the high-frequency information in the color field, the number of Gaussian spheres required by the semantic field is greatly reduced, and fewer Gaussian spheres and shorter semantic feature dimensions make the present application require less training time and storage demand to complete the reconstruction of the 3D semantic field, and the rendering speed is faster. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application, the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flow chart of a 3DGS semantic field decoupling reconstruction method based on hierarchical compression according to an embodiment of the present application.

[0021] Figure 2 is a framework diagram of a 3DGS semantic field decoupling reconstruction method based on hierarchical compression according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the personnel in the art better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are used to describe a particular order or sequence. It should be understood that the terms thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0024] Embodiment 1

[0025] According to an embodiment of the present application, a 3DGS semantic field decoupling reconstruction method based on hierarchical compression is provided. It should be noted that the steps shown in the flow chart of the drawings can be executed in a computer system containing at least one set of computer executable instructions, and although a logical order is shown in the flow chart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0026] Figure 1 is a flow chart of a 3DGS semantic field decoupling reconstruction method based on hierarchical compression according to an embodiment of the present application, as Figure 1 shown, the method can include the following steps:

[0027] Step S101, a plurality of multi-view images are obtained, and the plurality of multi-view images are input to a 2D base model to obtain a semantic feature map of each multi-view image.

[0028] In the technical solution provided in the above step S101 of the present application, Figure 2 is a framework diagram of a 3DGS semantic field decoupling reconstruction method based on hierarchical compression according to an embodiment of the present application, as Figure 2 shown, a plurality of multi-view images are obtained, and the plurality of multi-view images are input to a 2D base model to obtain a semantic feature map of each multi-view image, and the semantic feature map of each multi-view image is a feature map in a feature extraction module in Figure 2 .

[0029] Step S102, based on the semantic feature map of the first multi-view image, an initial semantic feature codebook is obtained, wherein the initial semantic feature codebook includes a plurality of feature vectors, and each feature vector contains an initial index value.

[0030] In the technical solution provided in the above step S102 of the present application, as Figure 2 shown, the semantic feature map of the first multi-view image is processed to obtain an initial semantic feature codebook, wherein the initial semantic feature codebook includes a plurality of feature vectors, and the initial semantic feature codebook is also a codebook in Figure 2 , the initial semantic feature codebook includes a plurality of feature vectors, and the plurality of feature vectors are Figure 2 in , and the initial index value is .

[0031] Step S103, based on each semantic feature map and the initial semantic feature codebook, a target quantization feature map and a target semantic feature codebook of each semantic feature map are determined.

[0032] In the technical solution provided in the above step S103 of the present application, each semantic feature map and the initial semantic feature codebook are processed to obtain a target quantization feature map and a target semantic feature codebook of each semantic feature map.

[0033] Step S104, the autoencoder is trained through the target semantic feature codebook to obtain a target encoder, wherein the autoencoder includes an initial encoder and an initial decoder.

[0034] In the technical solution provided in the above step S104 of the present application, as Figure 2 shown, the target semantic codebook is processed through the encoder and the decoder in the autoencoder Figure 2 linked with the codebook to obtain a target encoder, and the target encoder is the encoder with a snowflake mark in Figure 2 , that is, the trained encoder, the initial encoder is the encoder with a fire mark in Figure 2 , and the initial decoder is the decoder with a fire mark.

[0035] Step S105, obtaining an initial 3D GS semantic field.

[0036] In the technical solution provided by the step S105 of the present application, the initial 3D GS semantic field is obtained, that is, the 3D GS semantic field which is not trained by the latent feature map of each semantic feature map.

[0037] Step S106, obtaining a target 3D GS semantic field based on the latent feature map of each semantic feature map and the initial 3D GS semantic field.

[0038] In the technical solution provided by the step S106 of the present application, the initial 3D GS semantic field is trained by the latent feature map of each semantic feature map to obtain the target 3D GS semantic field.

[0039] The above method of the embodiment will be further introduced as follows.

[0040] As an optional embodiment, step S102, the initial semantic feature codebook is obtained based on the semantic feature map of the first multi-view image, including: sampling target feature vectors from the semantic feature map of the first multi-view image by farthest point sampling as the initial semantic feature codebook.

[0041] In this embodiment, the semantic feature map of the first multi-view image is sampled by farthest point sampling target feature vectors as initialization of the initial semantic feature codebook, and the expression of the initial semantic feature codebook is , wherein the farthest point sampling uses cosine similarity as distance measurement, is a real set, and D 1 is the dimension of the feature vector in the initial semantic feature codebook.

[0042] As an optional embodiment, in step S103, the target quantization feature map and the target semantic feature codebook of each semantic feature map are determined based on each semantic feature map and the initial semantic feature codebook, including: performing feature search on the initial semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map to obtain a first quantization feature map of the first semantic feature map, wherein the first quantization feature map contains a plurality of first index values; optimizing the initial semantic feature codebook based on the first semantic feature map and the first quantization feature map to obtain an optimized initial semantic feature codebook; taking the optimized initial semantic feature codebook as the initial semantic feature codebook of the second semantic feature map, repeating iteration to obtain an optimized initial semantic feature codebook of the last semantic feature map; taking the optimized initial semantic feature codebook of the last semantic feature map as the target semantic feature codebook; and obtaining the target quantization feature map of each semantic feature map based on the target semantic feature codebook and each semantic feature map.

[0043] In this embodiment, the first semantic feature map is divided into a plurality of feature vectors, and each feature vector is used to search the initial semantic feature codebook using cosine similarity to obtain a first index value. Each feature vector In the most similar feature search using cosine similarity on the initial semantic feature codebook , an index map is obtained based on the index map, and the first quantization feature map of the first semantic feature map is . is the height of the first semantic feature map, is the width of the first semantic feature map, is the dimension of the feature vector of the first semantic feature map, is the height of the index map, is the width of the index map.

[0044] The first semantic feature map is divided into a plurality of feature vectors, and each feature vector is used to search the initial semantic feature codebook using cosine similarity to obtain a first index value. Any one feature vector In the most similar feature search using cosine similarity on the initial semantic feature codebook , an index value expression is obtained as follows:

[0045]

[0046] An index map is obtained based on a plurality of index values, is an index value, is the horizontal coordinate and the vertical coordinate, respectively.

[0047] An index map is obtained based on the index map.

[0048] Based on the first semantic feature map and the first quantized feature map, the initial semantic feature codebook is optimized to obtain an optimized initial semantic feature codebook. The optimized initial semantic feature codebook is used as the initial semantic feature codebook for the second semantic feature map. The semantic feature map of the second multi-view image and the optimized initial semantic feature codebook are processed repeatedly and iteratively using the same process as the semantic feature map and the initial semantic feature codebook of the first multi-view image to obtain the final optimized initial semantic feature codebook. The final optimized initial semantic feature codebook is used as the target semantic feature codebook. Based on the target semantic feature codebook and each semantic feature map, the target quantized feature map of each semantic feature map is obtained.

[0049] As an optional embodiment, the optimization of the initial semantic feature codebook based on the first semantic feature map and the first quantized feature map to obtain an optimized initial semantic feature codebook includes: obtaining a first semantic feature codebook based on the feature vectors of the first semantic feature map and the feature vectors of the first quantized feature map; obtaining a second quantized feature map of the first semantic feature map based on the first semantic feature codebook and the first semantic feature map, wherein the second quantized feature map includes multiple second index values; obtaining a target loss value based on the second quantized feature map, the first semantic feature map, the first index value of the second quantized feature map, and the initial index value of the initial semantic feature codebook; and optimizing the first semantic feature codebook based on the target loss value to obtain the optimized initial semantic feature codebook.

[0050] In this embodiment, a first semantic feature codebook is obtained based on the feature vectors of the first semantic feature map and the first quantized feature map. A second quantized feature map of the first semantic feature map is obtained based on the first semantic feature codebook and the first semantic feature map, wherein the second quantized feature map includes multiple second index values. Based on the second quantized feature map, the first semantic feature map, the first index values ​​of the second quantized feature map, and the initial index values ​​of the initial semantic feature codebook, the expression for the target loss value is obtained as follows:

[0051]

[0052] in, yes The number of categories, yes Medium category The number of pixels, It is the Kronecker function. For the target loss value, For the first semantic feature map in The feature vector at that location, For the second quantization feature map in The feature vector at that location, for the second quantized feature map in the target coordinate, the first index value corresponding to the feature vector at the target coordinate, all initial index values of the initial semantic feature codebook;

[0053] The definition of the Kronecker function is as follows:

[0054]

[0055] The initial semantic feature codebook is optimized by the target loss value to obtain an optimized initial semantic feature codebook.

[0056] As an optional embodiment, the first semantic feature codebook is obtained based on the feature vector of the first semantic feature map and the feature vector of the first quantized feature map, including: when the cosine similarity between the feature vector of the first semantic feature map in the target coordinate and the feature vector of the first quantized feature map is less than a first hyperparameter, the feature vector of the first semantic feature map in the target coordinate is added to the initial semantic feature codebook to obtain a second semantic feature codebook, wherein the second semantic feature codebook includes a plurality of second semantic index values; when the cosine similarity between any two feature vectors corresponding to different second semantic index values in the second semantic feature codebook is less than a second hyperparameter, the feature vector with a larger second semantic index value is deleted to obtain the first semantic feature codebook.

[0057] In this embodiment, when the cosine similarity between the feature vector of the first semantic feature map in the target coordinate and the feature vector of the first quantized feature map is less than the first hyperparameter, the feature vector of the first semantic feature map in the target coordinate is added to the initial semantic feature codebook to obtain the expression of the second semantic feature codebook as follows:

[0058]

[0059] wherein, the first hyperparameter, is set to 0.9, and the initial semantic feature codebook , is the second semantic feature codebook.

[0060] When the cosine similarity between any two feature vectors corresponding to different second semantic index values in the second semantic feature codebook is less than the second hyperparameter, the feature vector with a larger second semantic index value is deleted to obtain the expression of the first semantic feature codebook as follows:

[0061]

[0062] wherein, the first semantic feature codebook, For any two feature vectors corresponding to different second semantic index values in the second semantic feature codebook, as a second super parameter.

[0063] As an optional embodiment, the second quantization feature map of the first semantic feature map is obtained based on the first semantic feature codebook and the first semantic feature map, comprising: performing feature search on the first semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map to obtain the second quantization feature map of the first semantic feature map.

[0064] In this embodiment, each feature vector of the first semantic feature map is searched for features on the first semantic feature codebook using cosine similarity to obtain the second quantization feature map of the first semantic feature map.

[0065] As an optional embodiment, step S104, the target encoder is obtained by training the autoencoder based on the target semantic feature codebook, comprising: dimensionally compressing the target semantic feature codebook by the initial encoder to obtain compressed features; reconstructing the compressed features by the initial decoder to obtain reconstructed features; in the process of obtaining the reconstructed features, when the loss function between the reconstructed features and the target semantic feature codebook is minimized, the autoencoder training is completed, and the target encoder is obtained.

[0066] In this embodiment, the compressed features are obtained by dimensionally compressing the target semantic feature codebook ( ) by the initial encoder ( ). The reconstructed features are obtained by reconstructing the compressed features ( ) by the initial decoder ( ).

[0067] The expression of the loss function between the reconstructed features and the target semantic feature codebook is:

[0068]

[0069] wherein, is the loss function between the reconstructed features and the target semantic feature codebook.

[0070] As an optional embodiment, the latent feature map of each semantic feature map is obtained based on the target encoder and the target quantization feature map of each semantic feature map, comprising: inferring the target quantization feature map of each semantic feature map by the target encoder to obtain the latent feature map of each semantic feature map. ​

[0071] In this embodiment, as shown in Figure 2 , the target quantized feature map is inferred by the target encoder to obtain the latent feature map of each semantic feature map.

[0072] As an optional embodiment, obtaining the target 3DGS semantic field based on the latent feature map of each semantic feature map and the initial 3DGS semantic field comprises: training the latent feature map of each semantic feature map by the initial 3DGS semantic field to obtain the target 3DGS semantic field.

[0073] In this embodiment, the latent feature map of each semantic feature map is trained by the initial 3DGS semantic field to obtain the target 3DGS semantic field, as shown in Figure 2 , the 3DGS semantic field decoupled from the color field is trained: each 3D Gaussian sphere contains attributes of: position , scale , rotation quaternion , transparency and semantic feature , wherein is the dimension of the compressed feature.

[0074] First, the semantic feature map under the perspective is obtained through differentiable rendering, and the target quantized feature map of each semantic feature map is modified to the size of the feature map using bilinear interpolation , that is, then the target quantized feature map of each semantic feature map is used for supervised training, wherein the training process is iterated for 30K times.

[0075] New perspective semantic segmentation inference: for a given set of semantic categories , the feature vector is classified in the target feature codebook, and then the semantic feature map under the new perspective is rendered from the 3DGS semantic field, at which time the bilinear interpolation is selectively used, the initial decoder is used to map the semantic feature map under the new perspective rendered in the 3DGS semantic field to the original feature dimension, and then the target feature codebook is used to quantize the semantic feature map under the new perspective rendered in the 3DGS semantic field after the dimension is restored to obtain the feature index map, and finally the classification label of each feature vector in the target feature codebook is indexed using the feature index map to complete the new perspective semantic segmentation inference.

[0076] Each 3D Gaussian in the color field is associated with a 3D Gaussian in the semantic field through spatial position information. For a 3D Gaussian in the color field, the 3D Gaussian closest to it in the semantic field or less than a threshold is obtained, and the semantic feature of the 3D Gaussian Sharing. When classifying each 3D Gaussian ball in the color field, the classification process is similar to the new view semantic segmentation inference, and the semantic features shared from the semantic field are used for calculation. The target semantic codebook is used to quantize the semantic features of the 3D Gaussian ball, and the classification result is calculated on the quantized version of the features. The semantic classification result of the 3D Gaussian ball can be used for editing of the 3D scene.

[0077] In the embodiment of the application, by acquiring a plurality of multi-view images, the plurality of multi-view images are input to a 2D base model to obtain a semantic feature map of each multi-view image; based on the semantic feature map of the first multi-view image, an initial semantic feature codebook is obtained, wherein the initial semantic feature codebook includes a plurality of feature vectors, and each feature vector contains an initial index value; based on each semantic feature map and the initial semantic feature codebook, a target quantized feature map and a target semantic feature codebook of each semantic feature map are determined; a target encoder is obtained by training the autoencoder through the target semantic feature codebook, wherein the autoencoder includes an initial encoder and an initial decoder; based on the target encoder and the target quantized feature map of each semantic feature map, a latent feature map of each semantic feature map is obtained; an initial 3DGS semantic field is acquired; based on the latent feature map of each semantic feature map and the initial 3DGS semantic field, a target 3DGS semantic field is obtained, which solves the technical problem that the high-dimensional semantic features and a large number of Gaussian balls in the prior art require a large amount of computing power and storage in the reconstruction process, and weaken the performance of 3DGS. The technical effect of decoupling the semantic field reconstruction process from the color field is achieved, which greatly reduces the number of 3D Gaussians required for semantic field reconstruction, and the effective compression of the semantic features by the compression level improves the reconstruction speed of the semantic field and the feature map rendering speed, reduces the storage required by the semantic field, and improves the performance of the new view semantic segmentation.

[0078] The above-mentioned embodiment numbers of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0079] In the above-mentioned embodiments of the application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0080] In the several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit described as the division is only a logical function division, and there can be another division manner during actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0081] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0082] In addition, each functional unit in each embodiment of the present application can be integrated in a first processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0083] The above is only the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A 3DGS semantic field decoupling and reconstruction method based on hierarchical compression, characterized in that, include: Acquire several multi-view images, input them into a 2D base model, and obtain the semantic feature map of each multi-view image; Based on the semantic feature map of the first multi-view image, an initial semantic feature codebook is obtained, which includes several feature vectors, each of which contains an initial index value. Based on each semantic feature map and the initial semantic feature codebook, determine the target quantized feature map and the target semantic feature codebook for each semantic feature map; The autoencoder is trained by the target semantic feature codebook to obtain the target encoder, which includes an initial encoder and an initial decoder. Based on the target encoder and the target quantized feature map of each semantic feature map, the latent feature map of each semantic feature map is obtained; Obtain the initial 3DGS semantic field; The target 3DGS semantic field is obtained based on the latent feature map of each semantic feature map and the initial 3DGS semantic field.

2. The method according to claim 1, characterized in that, The semantic feature map based on the first multi-view image yields an initial semantic feature codebook, including: The semantic feature map of the first multi-view image is sampled by sampling the farthest point to obtain the target feature vectors, which are used as the initial semantic feature codebook.

3. The method according to claim 2, characterized in that, The step of determining the target quantized feature map and target semantic feature codebook for each semantic feature map based on each semantic feature map and the initial semantic feature codebook includes: Each feature vector of the first semantic feature map is used to perform feature search on the initial semantic feature codebook using cosine similarity to obtain the first quantized feature map of the first semantic feature map, wherein the first quantized feature map contains multiple first index values; Based on the first semantic feature map and the first quantized feature map, the initial semantic feature codebook is optimized to obtain the optimized initial semantic feature codebook. The optimized initial semantic feature codebook is used as the initial semantic feature codebook for the second semantic feature map. This process is repeated until the optimized initial semantic feature codebook for the last semantic feature map is obtained. The initial semantic feature codebook optimized from the last semantic feature map is used as the target semantic feature codebook; Based on the target semantic feature codebook and each semantic feature map, the target quantized feature map of each semantic feature map is obtained.

4. The method according to claim 3, characterized in that, The optimization of the initial semantic feature codebook based on the first semantic feature map and the first quantized feature map to obtain the optimized initial semantic feature codebook includes: Based on the feature vectors of the first semantic feature map and the feature vectors of the first quantized feature map, the first semantic feature codebook is obtained; Based on the first semantic feature codebook and the first semantic feature map, a second quantized feature map of the first semantic feature map is obtained, wherein the second quantized feature map includes multiple second index values; Based on the second quantized feature map, the first semantic feature map, the first index value of the second quantized feature map, and the initial index value of the initial semantic feature codebook, the target loss value is obtained. The first semantic feature codebook is optimized based on the target loss value to obtain the optimized initial semantic feature codebook.

5. The method according to claim 4, characterized in that, The first semantic feature codebook is obtained based on the feature vectors of the first semantic feature map and the feature vectors of the first quantized feature map, including: When the cosine similarity between the feature vector of the first semantic feature map under the target coordinates and the feature vector of the first quantized feature map is less than the first hyperparameter, the feature vector of the first semantic feature map under the target coordinates is added to the initial semantic feature codebook to obtain the second semantic feature codebook, wherein the second semantic feature codebook includes multiple second semantic index values; When the cosine similarity between any two feature vectors corresponding to different second semantic index values ​​in the second semantic feature codebook is less than the second hyperparameter, the feature vector with the larger second semantic index value is deleted to obtain the first semantic feature codebook.

6. The method according to claim 5, characterized in that, The process of obtaining a second quantized feature map based on the first semantic feature codebook and the first semantic feature map includes: Each feature vector of the first semantic feature map is used to perform feature search on the first semantic feature codebook using cosine similarity, resulting in the second quantized feature map of the first semantic feature map.

7. The method according to claim 3, characterized in that, The step of training the autoencoder using the target semantic feature codebook to obtain the target encoder includes: The target semantic feature codebook is dimensionally compressed using an initial encoder to obtain compressed features; The compressed features are reconstructed using the initial decoder to obtain the reconstructed features; During the process of obtaining reconstructed features, the autoencoder training ends when the loss function between the reconstructed features and the target semantic feature codebook is minimized, and the target encoder is obtained.

8. The method according to claim 7, characterized in that, The latent feature map of each semantic feature map is obtained based on the target encoder and the target quantized feature map of each semantic feature map, including: The target encoder infers the target quantized feature map of each semantic feature map to obtain the latent feature map of each semantic feature map.

9. The method according to claim 7, characterized in that, The process of obtaining the target 3DGS semantic field based on the latent feature map of each semantic feature map and the initial 3DGS semantic field includes: The target 3DGS semantic field is obtained by training the latent feature map of each semantic feature map using the initial 3DGS semantic field.

Images