3D GS semantic field decoupling reconstruction method based on hierarchical compression

Abstract

The invention discloses a 3D GS semantic field decoupling reconstruction method based on hierarchical compression. The method comprises the following steps: acquiring a plurality of multi-view images, and inputting the plurality of multi-view images into a 2D basic model to obtain a semantic feature map of each multi-view image; based on the semantic feature map of the first multi-view image, an initial semantic feature codebook is obtained, the initial semantic feature codebook comprises a plurality of feature vectors, and each feature vector comprises an initial index value; and based on each semantic feature map and the initial semantic feature codebook, determining a target quantitative feature map and a target semantic feature codebook of each semantic feature map. The technical problem that in the prior art, due to high-dimensional semantic features and a large number of Gaussian balls, the reconstruction process needs a large amount of computing power and storage, and the performance of the 3DGS is weakened is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a 3DGS semantic field decoupling and reconstruction method based on hierarchical compression. Background Art

[0002] In the field of computer vision, 3D semantic field reconstruction is a technology with great application value. This technology aims to assign semantic information to objects in three-dimensional space, so that computers can have a deeper understanding of scenes. It has promoted the development of many fields such as intelligent robots, autonomous driving, augmented reality (AR), virtual reality (VR), urban planning and architectural design, and smart homes. In robotics, 3D semantic fields enable robots to understand the environment more accurately, achieve efficient navigation and safe operation; in the field of autonomous driving, this technology enhances the vehicle's perception of the surrounding environment and improves driving safety. For AR and VR, 3D semantic fields promote the seamless integration of virtual elements and the real world, enhancing the user's immersion and interactive experience. In urban planning and architectural design, it supports more accurate spatial analysis and solution presentation, and accelerates the design process. In addition, the smart home field also benefits from 3D semantic fields, realizing the intelligent layout of home appliances and environmental adaptability adjustment. With the continuous advancement of artificial intelligence and computer vision technology, the application of 3D semantic fields will be more extensive, which will not only improve the performance of existing systems, but also give rise to new application scenarios and service models, and promote the innovative development of related industries.

[0003] 2D Vision Foundation Models (2DVFM) can extract rich semantic features through pre-training on large-scale image data. These features show significant advantages in many aspects: First, 2DVFM has strong generalization capabilities and can adapt to a variety of visual tasks. Even when faced with new tasks, it can be quickly fine-tuned to achieve high performance. Secondly, these models are highly robust and can effectively cope with complex scenes such as lighting changes, perspective changes, and occlusions, ensuring stability and reliability in practical applications. In addition, 2DVFM supports multimodal fusion and can combine information from different modalities such as images and text to enhance the model's expressiveness and application scope. Migrating the capabilities of 2DVFM to 3D scenes has become a very meaningful research direction.

[0004] 3DGaussianSplatting (3DGS) is the latest 3D reconstruction technology after NeRF. Compared with NeRF, it has the advantages of high reconstruction quality, short training time, and fast rendering speed. It has become a research hotspot in the current 3D field.

[0005] 3DGS was originally designed for color field reconstruction. Each 3D Gaussian contains the following attributes: position ,scale , rotation quaternion ,transparency and color These parameters represent the spatial and visual properties of the 3D Gaussian in three dimensions, and are used to perform differentiable rasterization R on a particular camera pose. Render the image below.

[0006] In order to better understand the scene, researchers have tried to use 3DGS and 2DVFM to reconstruct the semantic field. In order to embed semantic information in 3DGS, some methods such as Feature3DGS introduce a semantic feature attribute for each 3D Gaussian based on the original 3DGS to make it consistent with the original The same parameters are optimized and learned simultaneously, and the image and semantic feature map can be rendered simultaneously; however, the high-dimensional semantic features and a large number of Gaussian spheres require a lot of computing power and storage in the reconstruction process, which greatly weakens the performance of 3DGS. Therefore, the present invention proposes a 3DGS semantic field decoupling reconstruction method based on hierarchical compression, which makes the 3D semantic field reconstruction more efficient. Summary of the invention

[0007] The embodiment of the present invention provides a 3DGS semantic field decoupling reconstruction method based on hierarchical compression, so as to at least solve the technical problem in the prior art that high-dimensional semantic features and a large number of Gaussian spheres require a large amount of computing power and storage in the reconstruction process, thereby weakening the performance of 3DGS.

[0008] According to one aspect of an embodiment of the present invention, a 3DGS semantic field decoupling reconstruction method based on hierarchical compression is provided. The method may include: obtaining a plurality of multi-view images, inputting the plurality of multi-view images into a 2D basic model, and obtaining a semantic feature map of each multi-view image; obtaining an initial semantic feature codebook based on the semantic feature map of the first multi-view image, wherein the initial semantic feature codebook includes a plurality of feature vectors, each feature vector including an initial index value; determining a target quantization feature map and a target semantic feature codebook of each semantic feature map based on each semantic feature map and the initial semantic feature codebook; training an autoencoder through the target semantic feature codebook to obtain a target encoder, wherein the autoencoder includes an initial encoder and an initial decoder; obtaining a potential feature map of each semantic feature map based on the target encoder and the target quantization feature map of each semantic feature map; obtaining an initial 3DGS semantic field; obtaining a target 3DGS semantic field based on the potential feature map of each semantic feature map and the initial 3DGS semantic field.

[0009] Optionally, obtaining an initial semantic feature codebook based on the semantic feature map of the first multi-view image includes: sampling a target number of feature vectors of the semantic feature map of the first multi-view image by farthest point sampling as the initial semantic feature codebook.

[0010] Optionally, the method of determining a target quantitative feature map and a target semantic feature codebook for each semantic feature map based on each semantic feature map and the initial semantic feature codebook includes: performing a feature search on the initial semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map to obtain a first quantitative feature map of the first semantic feature map, wherein the first quantitative feature map includes multiple first index values; optimizing the initial semantic feature codebook based on the first semantic feature map and the first quantitative feature map to obtain an optimized initial semantic feature codebook; using the optimized initial semantic feature codebook as the initial semantic feature codebook of the second semantic feature map, repeating the iteration to obtain the optimized initial semantic feature codebook of the last semantic feature map; using the optimized initial semantic feature codebook of the last semantic feature map as the target semantic feature codebook; and obtaining a target quantitative feature map of each semantic feature map based on the target semantic feature codebook and each semantic feature map.

[0011] Optionally, the optimizing the initial semantic feature codebook based on the first semantic feature map and the first quantitative feature map to obtain an optimized initial semantic feature codebook includes: obtaining the first semantic feature codebook based on the feature vector of the first semantic feature map and the feature vector of the first quantitative feature map; obtaining a second quantitative feature map of the first semantic feature map based on the first semantic feature codebook and the first semantic feature map, wherein the second quantitative feature map includes multiple second index values; obtaining a target loss value based on the second quantitative feature map, the first semantic feature map, the first index value of the second quantitative feature map and the initial index value of the initial semantic feature codebook; optimizing the first semantic feature codebook based on the target loss value to obtain an optimized initial semantic feature codebook.

[0012] Optionally, the obtaining of the first semantic feature codebook based on the feature vector of the first semantic feature map and the feature vector of the first quantized feature map includes: when the cosine similarity between the feature vector of the first semantic feature map at the target coordinates and the feature vector of the first quantized feature map is less than a first hyperparameter, adding the feature vector of the first semantic feature map at the target coordinates to the initial semantic feature codebook to obtain a second semantic feature codebook, wherein the second semantic feature codebook includes multiple second semantic index values; when the cosine similarity between any two feature vectors corresponding to different second semantic index values ​​in the second semantic feature codebook is less than a second hyperparameter, deleting the feature vector with a larger second semantic index value to obtain the first semantic feature codebook.

[0013] Optionally, obtaining the second quantized feature map of the first semantic feature map based on the first semantic feature codebook and the first semantic feature map includes: performing a feature search on the first semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map to obtain the second quantized feature map of the first semantic feature map.

[0014] Optionally, the method of training the autoencoder through the target semantic feature codebook to obtain the target encoder includes: dimensionally compressing the target semantic feature codebook through the initial encoder to obtain compressed features; reconstructing the compressed features through the initial decoder to obtain reconstructed features; in the process of obtaining the reconstructed features, when the loss function between the reconstructed features and the target semantic feature codebook is minimized, the autoencoder training is completed to obtain the target encoder.

[0015] Optionally, obtaining a latent feature map of each semantic feature map based on the target encoder and the target quantized feature map of each semantic feature map includes: inferring the target quantized feature map of each semantic feature map through the target encoder to obtain a latent feature map of each semantic feature map.

[0016] Optionally, obtaining a target 3DGS semantic field based on the latent feature map of each semantic feature map and the initial 3DGS semantic field includes: training the latent feature map of each semantic feature map through the initial 3DGS semantic field to obtain the target 3DGS semantic field.

[0017] Beneficial effects of the present invention: The present invention proposes a 3DGS semantic field decoupling reconstruction method based on hierarchical compression. The method first uses 2DVFM to extract semantic feature maps of multi-view images, proposes a hierarchical compression technology to adaptively construct and quantize the extracted semantic feature maps, and then uses an autoencoder to compress the dimensions of the core semantic features in the codebook. The compressed core semantic features are combined with the index map of the quantized output to construct a new multi-view potential feature map. Finally, the potential feature map is used to train the 3DGS with color attributes removed, thereby completing the reconstruction of the semantic field. The semantic field and the color field are decoupled, and the semantic field is reconstructed independently. Since the decoupled semantic field is not affected by the high-frequency information in the color field, the number of Gaussian spheres required for the semantic field is greatly reduced. Fewer Gaussian spheres and shorter semantic feature dimensions enable the present invention to complete the reconstruction of the 3D semantic field with less training time and storage requirements, and faster rendering speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1is a flow chart of a 3DGS semantic field decoupling and reconstruction method based on hierarchical compression according to an embodiment of the present invention; Figure 2 It is a framework diagram of a 3DGS semantic field decoupling and reconstruction method based on hierarchical compression according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] Example 1 According to an embodiment of the present invention, a 3DGS semantic field decoupling and reconstruction method based on hierarchical compression is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0022] Figure 1 is a flowchart of a 3DGS semantic field decoupling and reconstruction method based on hierarchical compression according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps: Step S101, obtaining a plurality of multi-view images, inputting the plurality of multi-view images into a 2D basic model, and obtaining a semantic feature map of each multi-view image.

[0023] In the technical solution provided in the above step S101 of the present invention, Figure 2is a framework diagram of a 3DGS semantic field decoupling and reconstruction method based on hierarchical compression according to an embodiment of the present invention, such as Figure 2 As shown, several multi-view images are obtained, and several multi-view images are input into the 2D basic model to obtain the semantic feature map of each multi-view image. The semantic feature map of each multi-view image is as follows Figure 2 The purple feature map in the feature extraction module.

[0024] Step S102: obtaining an initial semantic feature codebook based on the semantic feature map of the first multi-view image, wherein the initial semantic feature codebook includes a plurality of feature vectors, and each feature vector includes an initial index value.

[0025] In the technical solution provided in step S102 of the present invention, if Figure 2 As shown, the semantic feature map of the first multi-view image is processed to obtain an initial semantic feature codebook, wherein the initial semantic feature codebook includes several feature vectors. The initial semantic feature codebook is Figure 2 The codebook in the initial semantic feature codebook includes several feature vectors, which are Figure 2 In The initial index value is .

[0026] Step S103: determining a target quantized feature map and a target semantic feature codebook for each semantic feature map based on each semantic feature map and the initial semantic feature codebook.

[0027] In the technical solution provided in the above step S103 of the present invention, each semantic feature map and the initial semantic feature codebook are processed to obtain a target quantized feature map and a target semantic feature codebook for each semantic feature map.

[0028] Step S104, training the autoencoder using the target semantic feature codebook to obtain a target encoder, wherein the autoencoder includes an initial encoder and an initial decoder.

[0029] In the technical solution provided in step S104 of the present invention, if Figure 2 As shown, through the autoencoder ( Figure 2 The target semantic codebook is processed by the encoder and decoder linked to the codebook in the target semantic codebook to obtain the target encoder, which is Figure 2 The encoder with snowflake marks is the trained encoder. The initial encoder is Figure 2 The encoder with fire mark in the middle, and the initial decoder is the decoder with fire mark.

[0030] Step S105, obtaining an initial 3DGS semantic field.

[0031] In the technical solution provided in the above step S105 of the present invention, an initial 3DGS semantic field is obtained, that is, a 3DGS semantic field that has not been trained with the latent feature map of each semantic feature map.

[0032] Step S106, obtaining a target 3DGS semantic field based on the potential feature map of each semantic feature map and the initial 3DGS semantic field.

[0033] In the technical solution provided in the above step S106 of the present invention, the initial 3DGS semantic field is trained by the potential feature map of each semantic feature map to obtain the target 3DGS semantic field.

[0034] The above method of this embodiment is further introduced below.

[0035] As an optional implementation method, step S102, the initial semantic feature codebook is obtained based on the semantic feature map of the first multi-view image, including: sampling a target number of feature vectors of the semantic feature map of the first multi-view image by farthest point sampling as the initial semantic feature codebook.

[0036] In this embodiment, the semantic feature map of the first multi-view image is sampled using the farthest point sampling. The target feature vector is The initialization of the semantic feature codebook is expressed as , , where the farthest point sampling uses cosine similarity as the distance metric, is a set of real numbers, D 1 is the dimension of the feature vector in the initial semantic feature codebook.

[0037] As an optional implementation method, step S103, based on each semantic feature map and the initial semantic feature codebook, determining the target quantitative feature map and the target semantic feature codebook of each semantic feature map, includes: performing feature search on the initial semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map to obtain a first quantitative feature map of the first semantic feature map, wherein the first quantitative feature map includes multiple first index values; based on the first semantic feature map and the first quantitative feature map, optimizing the initial semantic feature codebook to obtain an optimized initial semantic feature codebook; using the optimized initial semantic feature codebook as the initial semantic feature codebook of the second semantic feature map, repeating the iteration to obtain the optimized initial semantic feature codebook of the last semantic feature map; using the optimized initial semantic feature codebook of the last semantic feature map as the target semantic feature codebook; based on the target semantic feature codebook and each semantic feature map, obtaining the target quantitative feature map of each semantic feature map.

[0038] In this embodiment, the first semantic feature map Each feature vector Using cosine similarity in the initial semantic feature codebook By searching for the most similar features on , the first quantized feature map of the first semantic feature map obtained based on the index map is , is the height of the first semantic feature map, is the width of the first semantic feature map, is the dimension of the feature vector of the first semantic feature map, is the height of the index graph, The width of the index image.

[0039] The first semantic feature map Any feature vector Using cosine similarity in the initial semantic feature codebook By searching for the most similar feature, we can get an index value expression as follows:

[0040] Get an index map based on multiple index values , is an index value, are the horizontal and vertical axes respectively.

[0041] The first quantized feature map of the first semantic feature map obtained based on the index map is: .

[0042] According to the first semantic feature map and the first quantitative feature map, the initial semantic feature codebook is optimized to obtain an optimized initial semantic feature codebook; the optimized initial semantic feature codebook is used as the initial semantic feature codebook of the second semantic feature map, the semantic feature map of the second multi-view image and the optimized initial semantic feature codebook are subjected to the same processing as the semantic feature map and the initial semantic feature codebook of the first multi-view image, and the optimized initial semantic feature codebook of the last semantic feature map is obtained by repeated iteration; the initial semantic feature codebook after the optimization of the last semantic feature map is used as the target semantic feature codebook; according to the target semantic feature codebook and each semantic feature map, a target quantitative feature map of each semantic feature map is obtained.

[0043] As an optional implementation method, the initial semantic feature codebook is optimized based on the first semantic feature map and the first quantitative feature map to obtain an optimized initial semantic feature codebook, including: obtaining the first semantic feature codebook based on the feature vector of the first semantic feature map and the feature vector of the first quantitative feature map; obtaining a second quantitative feature map of the first semantic feature map based on the first semantic feature codebook and the first semantic feature map, wherein the second quantitative feature map includes multiple second index values; obtaining a target loss value based on the second quantitative feature map, the first semantic feature map, the first index value of the second quantitative feature map, and the initial index value of the initial semantic feature codebook; optimizing the first semantic feature codebook based on the target loss value to obtain an optimized initial semantic feature codebook.

[0044] In this embodiment, a first semantic feature codebook is obtained according to the feature vector of the first semantic feature map and the feature vector of the first quantized feature map, and a second quantized feature map of the first semantic feature map is obtained according to the first semantic feature codebook and the first semantic feature map, wherein the second quantized feature map includes a plurality of second index values; according to the second quantized feature map, the first semantic feature map, the first index value of the second quantized feature map and the initial index value of the initial semantic feature codebook, an expression for obtaining a target loss value is obtained as follows:

[0045] in, yes The number of categories in yes Medium Category The number of pixels, is the Kronecker function, is the target loss value, The first semantic feature map is The eigenvector at The second quantized feature map is The eigenvector at The second quantized feature map is The first index value corresponding to the eigenvector at , are all the initial index values ​​of the initial semantic feature codebook; The Kronecker function is defined as:

[0046] The first semantic feature codebook is optimized by the target loss value to obtain an optimized initial semantic feature codebook.

[0047] As an optional implementation method, the first semantic feature codebook is obtained based on the feature vector of the first semantic feature map and the feature vector of the first quantized feature map, including: when the cosine similarity between the feature vector of the first semantic feature map at the target coordinate and the feature vector of the first quantized feature map is less than a first hyperparameter, the feature vector of the first semantic feature map at the target coordinate is added to the initial semantic feature codebook to obtain a second semantic feature codebook, wherein the second semantic feature codebook includes multiple second semantic index values; when the cosine similarity between any two feature vectors corresponding to different second semantic index values ​​in the second semantic feature codebook is less than the second hyperparameter, the feature vector with a larger second semantic index value is deleted to obtain the first semantic feature codebook.

[0048] In this embodiment, when the cosine similarity between the feature vector of the first semantic feature map at the target coordinate and the feature vector of the first quantized feature map is less than the first hyperparameter, the feature vector of the first semantic feature map at the target coordinate is added to the initial semantic feature codebook, and the expression of the second semantic feature codebook is obtained as follows:

[0049] in, is the first hyperparameter, Set to 0.9, the initial semantic feature codebook , is the second semantic feature codebook.

[0050] When the cosine similarity between any two feature vectors corresponding to different second semantic index values ​​in the second semantic feature codebook is less than the second hyperparameter, the feature vector with a larger second semantic index value is deleted, and the expression of the first semantic feature codebook is obtained as follows:

[0051] in, is the first semantic feature codebook, are any two feature vectors corresponding to different second semantic index values ​​in the second semantic feature codebook, is the second hyperparameter.

[0052] As an optional implementation method, the method of obtaining a second quantized feature map of the first semantic feature map based on the first semantic feature codebook and the first semantic feature map includes: performing a feature search on the first semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map to obtain a second quantized feature map of the first semantic feature map.

[0053] In this embodiment, each feature vector of the first semantic feature map is subjected to feature search on the first semantic feature codebook using cosine similarity to obtain a second quantized feature map of the first semantic feature map.

[0054] As an optional implementation method, step S104, the training of the autoencoder through the target semantic feature codebook to obtain the target encoder includes: dimensionally compressing the target semantic feature codebook through the initial encoder to obtain compressed features; reconstructing the compressed features through the initial decoder to obtain reconstructed features; in the process of obtaining the reconstructed features, when the loss function between the reconstructed features and the target semantic feature codebook is minimized, the autoencoder training is completed to obtain the target encoder.

[0055] In this embodiment, the initial encoder ( ) for the target semantic feature codebook ( ) to perform dimension compression and obtain the compressed features ( ); through the initial decoder ( ) for the compression feature ( ) to reconstruct and obtain the reconstructed features ( ); In the process of obtaining the reconstructed features, when the loss function between the reconstructed features and the target semantic feature codebook is minimized, the autoencoder training is completed and the target encoder is obtained.

[0056] The expression of the loss function between the reconstructed features and the target semantic feature codebook is:

[0057] in, is the loss function between the reconstructed features and the target semantic feature codebook.

[0058] As an optional implementation method, the target encoder and the target quantized feature map of each semantic feature map are used to obtain a potential feature map of each semantic feature map, including: inferring the target quantized feature map of each semantic feature map through the target encoder to obtain a potential feature map of each semantic feature map.

[0059] In this embodiment, if Figure 2 As shown in Figure 1, the target quantized feature map is inferred through the target encoder to obtain the potential feature map of each semantic feature map.

[0060] As an optional implementation method, the target 3DGS semantic field is obtained based on the latent feature map of each semantic feature map and the initial 3DGS semantic field, including: training the latent feature map of each semantic feature map through the initial 3DGS semantic field to obtain the target 3DGS semantic field.

[0061] In this embodiment, the potential feature map of each semantic feature map is trained by the initial 3DGS semantic field to obtain the target 3DGS semantic field, such as Figure 2 As shown, the 3DGS semantic field decoupled from the color field is trained: each 3D Gaussian ball contains the following attributes: position ,scale , rotation quaternion ,transparency and semantic features ,in, is the dimension of compressed features.

[0062] First, the viewing angle is obtained through differentiable rendering The semantic feature map below uses bilinear interpolation The target quantized feature map of each semantic feature map is modified to the feature map size, that is, the target quantized feature map of each semantic feature map is then used for supervised training, wherein the training process is iterated 30K times.

[0063] New perspective on semantic segmentation reasoning: for a given set of semantic categories , classify the feature vectors of the target feature codebook, and then render the semantic feature map from the new perspective from the 3DGS semantic field. At this time, the bilinear interpolation It is used selectively. The initial decoder is used to map the semantic feature map of the new perspective rendered in the 3DGS semantic field to the original feature dimension. Then the target feature codebook is used to quantize the semantic feature map of the new perspective rendered in the 3DGS semantic field after dimension restoration to obtain its feature index map. Finally, the feature index map is used to index the classification label of each feature vector in the target feature codebook to complete the new perspective semantic segmentation reasoning.

[0064] Each 3D Gaussian in the color field is connected to the 3D Gaussian in the semantic field through spatial position information. , get its closest Euclidean distance to it in the semantic field or less than the threshold 3D Gaussian , and its semantic features are combined with Sharing. When classifying each 3D Gaussian ball in the color field, the semantic features shared from the semantic field are used for calculation. The classification process is similar to the new perspective semantic segmentation reasoning. Both require the use of the target semantic codebook to first quantize the semantic features of the 3D Gaussian ball, and then calculate the classification results on the quantized version of the features. The semantic classification results of the 3D Gaussian ball can be used for editing 3D scenes.

[0065] In an embodiment of the present invention, a plurality of multi-view images are obtained, and the plurality of multi-view images are input into a 2D basic model to obtain a semantic feature map of each multi-view image; an initial semantic feature codebook is obtained based on the semantic feature map of the first multi-view image, wherein the initial semantic feature codebook includes a plurality of feature vectors, and each feature vector includes an initial index value; a target quantization feature map and a target semantic feature codebook of each semantic feature map are determined based on each semantic feature map and the initial semantic feature codebook; an autoencoder is trained through the target semantic feature codebook to obtain a target encoder, wherein the autoencoder includes an initial encoder and an initial decoder; a target quantization feature map and a target semantic feature codebook of each semantic feature map are determined based on the target encoder and the target semantic feature codebook of each semantic feature map; a target encoder is obtained by training the autoencoder through the target semantic feature codebook ... Quantize the feature map to obtain the latent feature map of each semantic feature map; obtain the initial 3DGS semantic field; based on the latent feature map of each semantic feature map and the initial 3DGS semantic field, obtain the target 3DGS semantic field, which solves the technical problem in the prior art that the high-dimensional semantic features and a large number of Gaussian spheres require a lot of computing power and storage in the reconstruction process, weakening the performance of 3DGS, and achieves the decoupling of the semantic field reconstruction process from the color field, which greatly reduces the number of 3D Gaussians required for semantic field reconstruction. In addition, the compression level effectively compresses the semantic features, improves the reconstruction speed of the semantic field and the rendering speed of the feature map, reduces the storage required for the semantic field, and at the same time improves the technical effect of the performance of new perspective semantic segmentation.

[0066] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0067] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0068] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0069] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0070] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0071] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A 3DGS semantic field decoupling reconstruction method based on hierarchical compression, characterized in that: include: Obtain a number of multi-view images, input the multi-view images into the 2D basic model, and obtain a semantic feature map of each multi-view image; Based on the semantic feature map of the first multi-view image, an initial semantic feature codebook is obtained, wherein the initial semantic feature codebook includes a plurality of feature vectors, each of which includes an initial index value; Based on each semantic feature map and the initial semantic feature codebook, a target quantization feature map and a target semantic feature codebook are determined for each semantic feature map; The autoencoder is trained by using a target semantic feature codebook to obtain a target encoder, wherein the autoencoder includes an initial encoder and an initial decoder; Based on the target encoder and the target quantized feature map of each semantic feature map, a latent feature map of each semantic feature map is obtained; Get the initial 3DGS semantic field; Based on the latent feature map of each semantic feature map and the initial 3DGS semantic field, the target 3DGS semantic field is obtained.

2. The method according to claim 1, characterized in that The initial semantic feature codebook is obtained based on the semantic feature map of the first multi-view image, including: The semantic feature map of the first multi-view image is sampled by farthest point sampling to sample target feature vectors as the initial semantic feature codebook.

3. The method according to claim 2, characterized in that The step of determining a target quantized feature map and a target semantic feature codebook for each semantic feature map based on each semantic feature map and the initial semantic feature codebook includes: Performing feature search on the initial semantic feature codebook using cosine similarity for each feature vector of the first semantic feature map to obtain a first quantized feature map of the first semantic feature map, wherein the first quantized feature map includes a plurality of first index values; Based on the first semantic feature map and the first quantized feature map, the initial semantic feature codebook is optimized to obtain an optimized initial semantic feature codebook; The optimized initial semantic feature codebook is used as the initial semantic feature codebook of the second semantic feature map, and the iteration is repeated to obtain the optimized initial semantic feature codebook of the last semantic feature map; The initial semantic feature codebook after optimizing the last semantic feature map is used as the target semantic feature codebook; Based on the target semantic feature codebook and each semantic feature map, a target quantized feature map of each semantic feature map is obtained.

4. The method according to claim 3, characterized in that: The step of optimizing the initial semantic feature codebook based on the first semantic feature map and the first quantized feature map to obtain an optimized initial semantic feature codebook includes: Obtaining a first semantic feature codebook based on the feature vector of the first semantic feature map and the feature vector of the first quantized feature map; Based on the first semantic feature codebook and the first semantic feature map, a second quantized feature map of the first semantic feature map is obtained, wherein the second quantized feature map includes a plurality of second index values; Obtaining a target loss value based on the second quantized feature map, the first semantic feature map, the first index value of the second quantized feature map, and the initial index value of the initial semantic feature codebook; The first semantic feature codebook is optimized based on the target loss value to obtain an optimized initial semantic feature codebook.

5. The method according to claim 4, characterized in that The step of obtaining a first semantic feature codebook based on the feature vector of the first semantic feature map and the feature vector of the first quantized feature map includes: When the cosine similarity between the feature vector of the first semantic feature map at the target coordinate and the feature vector of the first quantized feature map is less than the first hyperparameter, adding the feature vector of the first semantic feature map at the target coordinate to the initial semantic feature codebook to obtain a second semantic feature codebook, wherein the second semantic feature codebook includes multiple second semantic index values; When the cosine similarity between any two feature vectors corresponding to different second semantic index values ​​in the second semantic feature codebook is less than the second hyperparameter, the feature vector with a larger second semantic index value is deleted to obtain the first semantic feature codebook.

6. The method according to claim 5, characterized in that The method of obtaining a second quantized feature map of the first semantic feature map based on the first semantic feature codebook and the first semantic feature map includes: Each feature vector of the first semantic feature map is subjected to feature search on the first semantic feature codebook using cosine similarity to obtain a second quantized feature map of the first semantic feature map.

7. The method according to claim 3, characterized in that The method of training the autoencoder by using the target semantic feature codebook to obtain the target encoder includes: The target semantic feature codebook is dimensionally compressed through the initial encoder to obtain compressed features; Reconstruct the compressed features through the initial decoder to obtain the reconstructed features; In the process of obtaining the reconstructed features, when the loss function between the reconstructed features and the target semantic feature codebook is minimized, the autoencoder training is completed and the target encoder is obtained.

8. The method according to claim 7, characterized in that The method of obtaining a potential feature map of each semantic feature map based on the target encoder and the target quantized feature map of each semantic feature map includes: The target quantized feature map of each semantic feature map is inferred through the target encoder to obtain the latent feature map of each semantic feature map.

9. The method according to claim 7, characterized in that: The target 3DGS semantic field is obtained based on the potential feature map of each semantic feature map and the initial 3DGS semantic field, including: The latent feature map of each semantic feature map is trained through the initial 3DGS semantic field to obtain the target 3DGS semantic field.

Images