Three-dimensional semantic scene completion method and system based on semantic-physical joint representation

Abstract

The invention provides a three-dimensional semantic scene completion method and system based on semantic-physical joint representation. The method comprises the following steps: extracting image features according to multi-view image input, and mapping the image features to a three-dimensional voxel space; voxel features containing local semantic information and three-view features containing global semantic information are extracted through a double-branch encoder, key anchor points with identifiable semantic features are screened out according to semantic consistency of the voxel features and the three-view features, and efficient initialization of Gaussian representation in a key area is guided; and then introducing a semantic spherical harmonic function, projecting Gaussian semantic representation to an orthogonal spherical harmonic space, enhancing the modeling capability of local context geometric structure information, and promoting the semantic-geometric representation consistency of the context by using key voxel distribution alignment, thereby realizing three-dimensional semantic scene completion with both precision and efficiency.

Description

Technical Field

[0001] This invention relates to the field of scene perception technology, specifically to a three-dimensional semantic scene completion method and system based on semantic-physical joint representation. Background Technology

[0002] In recent years, with the development of autonomous driving and intelligent space perception technologies, 3D semantic scene completion has become a key task for understanding and perceiving complex environments. Its goal is to reconstruct a complete 3D semantic scene from multi-view image inputs and predict the geometric occupancy state and semantic category of each voxel in space. It has important research and application value in scenarios such as autonomous driving and robot navigation.

[0003] Existing 3D semantic scene completion methods can be broadly categorized into two types based on their 3D representation methods: voxel-based methods and Gaussian-based methods. Voxel-based methods divide the 3D space into a regular voxel grid and perform semantic classification and occupancy prediction for each voxel using a 3D convolutional network and attention mechanisms. Their advantages lie in their clear spatial structure, efficient convolutional operations, and suitability for dense spatial modeling in large-scale scenes. However, voxel-based methods are limited by the discrete representation units of voxels, making it difficult to model complex spatial contexts and underlying physical laws. Consequently, the final semantic scene completion results often suffer from missing geometric details and a lack of realism.

[0004] On the other hand, Gaussian-based representation methods continuously model the geometric and semantic information of a scene by arranging a series of Gaussian representations in three-dimensional space, such as Neural Radiation Field (NeRF) and 3D Gaussian Splatting. The advantage of these methods lies in their good geometric continuity, enabling them to model realistic geometric details well. However, when dealing with large-scale, complex scenes such as autonomous driving, the large number of Gaussian representations leads to enormous computational overhead, and many of these representations reside in semantically sparse regions, severely impacting the convergence speed during training. This makes it difficult to accurately learn the contextual semantic information of each Gaussian representation, resulting in semantic confusion in the final semantic scene completion result.

[0005] To address the aforementioned issues, balancing model accuracy and efficiency to achieve efficient and accurate 3D semantic scene completion has become a significant challenge. Summary of the Invention

[0006] To address the aforementioned challenges, this invention proposes a three-dimensional semantic scene completion method and system based on semantic-physical joint representation. Through semantic-guided Gaussian initialization, Gaussian representations are efficiently initialized in key regions. Furthermore, through physically-aware spherical harmonic enhancement, the modeling capability for local contextual geometric structure information is improved. Finally, through key voxel distribution alignment, the consistency of semantic-geometric representations is promoted, achieving three-dimensional semantic scene completion that balances accuracy and efficiency.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A 3D semantic scene completion method based on semantic-physical joint representation is used to extract the semantic-physical joint representation of the input image and achieve 3D semantic scene completion, including the following steps: (1) Data preparation and feature extraction: Extract image features based on multi-view image input and map them to three-dimensional voxel space; (2) Semantic-guided Gaussian initialization: Based on image features mapped to three-dimensional voxel space, key anchor points are selected through the consistency of semantic representation, guiding the efficient initialization of Gaussian representation in key regions; (3) Physical perception enhancement of spherical harmonics: The initialized Gaussian representation obtained in step (2) is projected onto the orthogonal spherical harmonic space to enhance the modeling ability of context geometric structure information, and the semantic-geometric representation consistency of the context is promoted by aligning the distribution of key voxels, thereby achieving three-dimensional semantic scene completion that balances accuracy and efficiency.

[0008] Furthermore, in the above-mentioned three-dimensional semantic scene completion method based on semantic-physical joint representation, in step (1), the features of multi-view image input are extracted using an image encoder, and the image features are mapped to three-dimensional voxel space through a view converter.

[0009] Furthermore, in the above-mentioned three-dimensional semantic scene completion method based on semantic-physical joint representation, in step (2), a dual-branch encoder is used to extract voxel features containing local semantic information and three-view features containing global semantic information, and key anchor points with discernible semantic features are selected based on the semantic consistency of the two, so as to guide the efficient initialization of Gaussian representation in key regions.

[0010] Furthermore, in the above-mentioned three-dimensional semantic scene completion method based on semantic-physical joint representation, step (3) introduces a semantic spherical harmonic function to project the Gaussian representation onto the orthogonal spherical harmonic space, thereby enhancing the ability to model local contextual geometric structure information.

[0011] A 3D semantic scene completion system based on semantic-physical joint representation, comprising: The data preparation and feature extraction module is used to extract image features from multi-view image input and map them to a three-dimensional voxel space. The semantically guided Gaussian initialization module is used to select key anchor points based on the consistency of semantic representations of image features mapped to three-dimensional voxel space, and guide the efficient initialization of Gaussian representations in key regions. The physics-aware spherical harmonic enhancement module projects the initialized Gaussian representation onto the orthogonal spherical harmonic space, enhancing the modeling capability of contextual geometric structure information. It also utilizes key voxel distribution alignment to promote the semantic-geometric representation consistency of the context, thereby achieving 3D semantic scene completion that balances accuracy and efficiency.

[0012] The advantages of this invention are as follows: Compared with existing 3D semantic scene completion methods, the semantic-physical joint representation method of this invention integrates the efficient computing power of voxel representation and the geometric modeling power of Gaussian representation, which can balance accuracy and efficiency. In large-scale complex scenarios such as autonomous driving, it can achieve accurate and realistic 3D semantic scene completion with less computing resources.

[0013] The reason why this invention has the above-mentioned effects is that: by filtering the semantic consistency between voxels and the projection features of three views, spatial anchor points with strong discriminative power are accurately selected, achieving sparse and efficient Gaussian initialization and avoiding the waste of resources in the traditional Gaussian method due to a large amount of redundant modeling; at the same time, the ability to model local context geometric structure information is enhanced by semantic spherical harmonic functions, and the consistency between semantic and geometric features is further enhanced by key voxel distribution alignment, thereby achieving semantic scene completion that balances accuracy and efficiency. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a three-dimensional semantic scene completion method based on semantic-physical joint representation according to the present invention.

[0015] Figure 2 This is a schematic diagram of semantically guided Gaussian initialization in the embodiment.

[0016] Figure 3 This is a schematic diagram of the physical perception spherical harmonic enhancement in the embodiment. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0018] This embodiment presents a 3D semantic scene completion method based on semantic-physical joint representation, the process of which is as follows: Figure 1 As shown, the specific steps include: (1) Data preparation and feature extraction First, for multi-view image input, ResNet50 is used to extract multi-scale representations, and then the multi-scale representations are aggregated through an FPN network to obtain two-dimensional image features. Then, using a view converter based on the Flosp method, the center of the 3D voxel is projected onto the 2D image feature map, and the corresponding features are sampled to obtain the 3D features. .

[0019] (2) Semantic-guided Gaussian initialization First, in order to better capture local and global semantic features, based on the three-dimensional features obtained in step (1), voxel features containing local semantic information and three-view features containing global semantic information are extracted by a dual-branch encoder. This process can be expressed as the following formula: (1) (2) in, Indicates voxel characteristics, Representing the features of the three views, Represents a 3D convolutional network layer. This represents a three-dimensional feature pyramid network layer. This indicates a pooling network layer. This represents a two-dimensional convolutional network layer.

[0020] Next, key anchor point screening is performed. First, the semantic consistency score of voxel features and three-view features is calculated. The calculation process can be expressed as the following formula: (3) in, The fusion weights represent the features of the three views. Calculate the cosine similarity in the feature space. Further select the K voxels with the highest semantic consistency scores as Gaussian feature initialization anchors with discriminative semantic information, i.e., key anchors. The specific process can be expressed as the following formula: (4) (5) in, This represents the coordinates of the anchor points obtained after filtering. This represents the Gaussian representation corresponding to the anchor point.

[0021] Finally, through semantic Gaussian initialization, the semantic features of the aforementioned key anchor points are mapped and activated to obtain efficient initialization attributes of Gaussian representations in the key regions. ,in, These represent the center coordinates, scaling matrix, rotation matrix, and opacity of the Gaussian representation, respectively. Figure 2 This is a schematic diagram of semantically guided Gaussian initialization.

[0022] (3) Enhanced spherical harmonicity of physical perception Based on the initialized Gaussian representation obtained in step (2), a semantic spherical harmonic function is introduced to project the Gaussian representation onto the orthogonal spherical harmonic space, thereby enhancing the ability to model local contextual geometric structure information. The specific process can be expressed as the following formula: in, Let these represent the degree and order of the spherical harmonic function, respectively. Represents spherical harmonic coordinates, Represents semantic spherical harmonic functions. Denotes the order of the spherical harmonic function. Represents the spherical harmonic coefficients. Denotes basis functions. This represents the extended Gaussian representation. This represents a one-dimensional convolutional network layer. During the projection process, orthogonal loss is further utilized to promote the orthogonality of different representations, thereby enhancing semantic expressiveness. in, This represents the weight matrix in the projection layer. Represents the weight parameters. Represents the identity matrix.

[0023] Then, consistency between contextual semantic and geometric features is promoted through key voxel distribution alignment. Prediction results based on voxel representation and Gaussian representation are obtained using the semantic scene completion task head and Gaussian superposition method, respectively. and Furthermore, an alignment loss is introduced to promote semantic and geometric distribution consistency at key anchor points: in, This represents the KL divergence, used to measure the consistency of a distribution; Indicates key Gaussian prediction; This represents the key voxel prediction. Finally, the aligned voxels and Gaussian prediction results are integrated to obtain the final 3D semantic scene completion result that balances efficiency and accuracy. Figure 3 This is a schematic diagram of the enhancement of spherical harmonics in physical perception.

[0024] The experimental results below show that, compared with existing semantic scene completion methods, the three-dimensional semantic scene completion method based on semantic-physical joint representation of the present invention can generate more accurate and realistic semantic scene completion results, achieving the best performance to date.

[0025] This embodiment is based on experiments using the SSCBench-KITTI-360 autonomous driving dataset, proposed in the paper "SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Driving" (authors Yiming Li, Sihang Li, Xinhao Liu, Moonjun Gong, Kenan Li, Nuo Chen, Zijun Wang, Zhiheng Li, Tao Jiang, Fisher Yu, Yue Wang, HangZhao, Zhiding Yu, and Chen Feng, published in 2023). This dataset includes target objects of 19 different semantic categories. We compare this invention with the following three existing 3D semantic scene completion methods in our experiments: Existing Method 1: The method in the paper "GaussianFormer-2: Probabilistic GaussianSuperposition for Efficient 3D Occupancy Prediction" (authors Yuanhui Huang, Amonnut Thammatadatrakoon, Wenzhao Zheng, Yunpeng Zhang, Dalong Du and Jiwen Lu, published at the 2025 IEEE Conference on Computer Vision and Pattern Recognition) is used to complete 3D semantic scenes through a distribution-initialized image-to-Gaussian mapping and a probability-superimposed Gaussian mapping.

[0026] The second existing method is the one in the paper "Camera-based 3d Semantic Scene Completion with SparseGuidance Network" (authors Jianbiao Mei, Yu Yang, Mengmeng Wang, Junyu Zhu, Jongwon Ra, Yukai Ma, Laijian Li and Yong Liu, published in IEEE Transactions on Image Processing in 2024). This method designs a dense-sparse-dense architecture to enhance the clarity of object boundaries in the semantic scene completion results.

[0027] The third existing method is the one in the paper "Context and Geometry Aware Voxel Transformer for Semantic Scene Completion" (authors Zhu Yu, Runmin Zhang, Jiacheng Ying, Junchen Yu, Xiaohai Hu, Lun Luo, Si-Yuan Cao and Hui-Liang Shen, published at the 2024 Conference on Neural Information Processing Systems). This method introduces context-dependent semantic queries and multiple 3D scene representations to enhance the performance of semantic scene completion.

[0028] This embodiment uses the Intersection over Union (IoU) and Mean Intersection over Union (mIoU) metrics to measure the predictive performance of different semantic scene completion methods. Specifically, the IoU metric is used to evaluate the accuracy of the model in semantic scene completion in 3D space. It is calculated as the ratio of the intersection to the union of the voxels predicted by the model as occupied and the actual occupied voxels. The mIoU metric measures the classification prediction performance of the model across all semantic categories. The IoU is calculated for each category and then averaged. The larger the values ​​of IoU and mIoU, the better the performance of the semantic scene completion method.

[0029] Table 1. Experimental results of IoU and mIoU metrics compared with existing semantic scene completion methods. As can be seen from Table 1, the present invention achieves better 3D semantic scene completion results. Existing semantic scene completion methods only use a single representation method, either voxel representation or Gaussian representation, which makes it difficult to take into account both geometric details and semantic information. In contrast, the present invention adopts a joint semantic-physical representation, which enhances the model's ability to model the context's geometric structure and aligns the semantic-geometric feature distribution, thereby generating more accurate and realistic semantic scene completion results.

[0030] Another embodiment of the present invention provides a three-dimensional semantic scene completion system based on semantic-physical joint representation, which includes: The data preparation and feature extraction module is used to extract image features from multi-view image input and map them to a three-dimensional voxel space. The semantically guided Gaussian initialization module is used to select key anchor points based on the consistency of semantic representations of image features mapped to three-dimensional voxel space, and guide the efficient initialization of Gaussian representations in key regions. The physics-aware spherical harmonic enhancement module projects the initialized Gaussian representation onto the orthogonal spherical harmonic space, enhancing the modeling capability of contextual geometric structure information. It also utilizes key voxel distribution alignment to promote the semantic-geometric representation consistency of the context, thereby achieving 3D semantic scene completion that balances accuracy and efficiency.

[0031] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be found in the corresponding processes in the aforementioned method embodiments.

[0032] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing steps of the method of the present invention.

[0033] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0034] Another embodiment of the present invention provides a computer program product, the computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0035] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for three-dimensional semantic scene completion based on semantic-physical joint representation, characterized in that, The method comprises the following steps: According to the multi-view image input, image features are extracted and mapped to a three-dimensional voxel space; Based on the image features mapped to the three-dimensional voxel space, key anchor points are screened through the consistency of semantic representation, and the efficient initialization of Gaussian representation in the key area is guided; The initialized Gaussian representation is projected to an orthogonal spherical harmonic space to enhance the modeling ability of the context geometric structure information, and the semantic-geometric representation consistency of the context is promoted by using key voxel distribution alignment, so that the three-dimensional semantic scene completion with consideration of both precision and efficiency is realized.

2. The method of claim 1, wherein, The method comprises the following steps:

3. The method of claim 1, wherein, The method comprises the following steps:

4. The method of claim 3, wherein, The method comprises the following steps: The method comprises the following steps: wherein, represents a voxel feature, represents a three-view feature, represents a fusion weight of the three-view feature, computes a cosine similarity of the feature space; The method comprises the following steps: wherein, represents the anchor point coordinates screened out, represents the Gaussian representation corresponding to the anchor point.

5. The method of claim 4, wherein, The guided Gaussian representation is efficient initialization in the key region, including: through semantic Gaussian initialization, the semantic features of the key anchor points are mapped and activated to obtain the efficient initialization attribute of the Gaussian representation in the key region wherein, respectively represent the center coordinates, the scaling matrix, the rotation matrix and the opacity of the Gaussian representation.

6. The method of claim 5, wherein, The method comprises the following steps: wherein, denote degree and order of spherical harmonics, respectively, denote spherical harmonics coordinates, denote semantic spherical harmonics, denote order of spherical harmonics, denote spherical harmonic coefficients, denote basis functions, denote expanded Gaussian representation, denote one-dimensional convolution network layer; in the projection process, orthogonal loss is used to promote the orthogonality of different representations and enhance the semantic expression ability.

7. The method of claim 6, wherein, The method comprises the following steps: The prediction results based on voxel representation and Gaussian representation are obtained by using a semantic scene completion task head and a Gaussian superposition method, respectively and introducing an alignment loss promote semantic and geometric distribution consistency located at key anchor points: wherein, represents the KL divergence, which is used to measure the consistency of the distribution; and the aligned voxels and the Gaussian prediction results are integrated to obtain a three-dimensional semantic scene completion result with both efficiency and accuracy. 8.A three-dimensional semantic scene completion system based on semantic-physical joint representation, characterized in that, The method comprises the following steps: The method comprises the following steps: The computer readable storage medium stores a computer program, and the computer program is executed by a computer to realize the method in any one of claims 1-7. The computer readable storage medium stores a computer program, and the computer program is executed by a computer to realize the method in any one of claims 1-7.

9. A computer device, comprising: ​ 10. A computer-readable storage medium, characterized in that, ​

Images