Scene reconstruction and semantic understanding method and system based on visual large model

Abstract

This invention provides a method and system for scene reconstruction and semantic understanding based on a large visual model, relating to the fields of computer vision and 3D reconstruction. The method includes acquiring multi-view image data of a target scene, inputting it into a pre-trained large visual model, and outputting a joint feature representation and an attention weight matrix. Based on the joint feature representation, it generates 3D coordinates and semantic probability distributions of spatial sampling points in 3D space, constructing a spatial semantic field. The attention weight matrix is ​​used to cluster the spatial sampling points and enhance semantic consistency, converting them into deterministic semantic labels. A geometric optimization objective function is constructed to adjust the 3D coordinates. Continuous spatial sampling points with the same semantic labels are extracted to form object boundaries, a scene topology map is constructed, and the scene's functional structure is inferred, generating a navigation path. This invention achieves the unification of accurate geometric reconstruction and deep semantic understanding of a scene, improving the accuracy of 3D reconstruction and semantic analysis, and providing reliable support for intelligent navigation.

Description

Technical Field

[0001] This invention relates to computer vision and 3D reconstruction technology, and more particularly to a method and system for scene reconstruction and semantic understanding based on a large visual model. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, scene reconstruction and semantic understanding have become core technologies in fields such as robot navigation, augmented reality, and intelligent monitoring. Traditional scene reconstruction methods mainly rely on multi-view geometry principles, reconstructing the 3D scene structure through feature point matching, triangulation, and other methods. Existing scene reconstruction technologies typically use methods such as structured light scanning, LiDAR, or multi-view stereo vision to acquire 3D point cloud data, and then generate 3D models through mesh reconstruction algorithms. In terms of semantic understanding, researchers project the semantic segmentation results of 2D images into 3D space, or directly perform semantic annotation on 3D point clouds. Some methods attempt to combine neural radiation fields with semantic features to achieve implicit scene representation. At the same time, scene graph generation technology is also used to describe the relationships between objects in a scene, providing support for higher-level scene understanding.

[0003] Traditional scene understanding methods mainly focus on object recognition and segmentation, lacking the ability to reason deeply about the functional structure of the scene. They cannot effectively utilize the spatial topological relationships and functional correlations between objects in the scene, making it difficult to generate high-quality navigation paths that conform to the semantic constraints of the scene, thus limiting their performance in practical applications such as robot autonomous navigation. Summary of the Invention

[0004] This invention provides a method and system for scene reconstruction and semantic understanding based on a large visual model, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a method for scene reconstruction and semantic understanding based on a large visual model, comprising:

[0006] Acquire multi-view image data of the target scene, input the multi-view image data into a pre-trained large visual model, and output joint feature representation and attention weight matrix;

[0007] Based on the joint feature representation, the three-dimensional coordinates and semantic probability distribution of spatial sampling points are generated in three-dimensional space to construct a spatial semantic field.

[0008] The spatial sampling points in the spatial semantic field are clustered into a set of spatial sampling points using an attention weight matrix. The semantic probability distribution within the set of spatial sampling points is then enhanced to obtain an enhanced semantic probability distribution. The enhanced semantic probability distribution is then converted into a deterministic semantic label.

[0009] A semantically guided geometric optimization objective function is constructed based on deterministic semantic labels and three-dimensional coordinate values. The three-dimensional coordinate values ​​of spatial sampling points are adjusted by minimizing the geometric optimization objective function to obtain the optimized three-dimensional coordinate values.

[0010] The optimized 3D coordinates of continuous spatial sampling points with the same deterministic semantic labels are extracted to form object boundaries. A scene topology graph is constructed based on the inclusion and adjacency relationships between object boundaries. The scene functional structure is inferred based on the scene topology graph. A navigation path is generated based on the object boundaries and the scene functional structure.

[0011] Acquire multi-view image data of the target scene, input the multi-view image data into a pre-trained large-scale visual model, and output a joint feature representation and attention weight matrix, including:

[0012] The original multi-view image data of the target scene is acquired, the spatial pose calibration of the original multi-view image data is performed, the spatial transformation matrix between each view is calculated, and geometric correction is performed based on the spatial transformation matrix to generate multi-view image data with spatial pose information.

[0013] Multi-view image data with spatial pose information is divided into multiple view groups. Image preprocessing is performed on the images within the multiple view groups. Spatial consistency correction is then performed on the preprocessed view groups based on the spatial pose information to generate standardized multi-view image data.

[0014] Standardized multi-view image data is input into a pre-trained large visual model according to the viewpoint order determined by spatial pose information. Multi-level visual features are extracted, and the feature similarity between viewpoints is calculated based on the multi-level visual features and the spatial transformation matrix to generate an attention weight matrix.

[0015] Multi-level visual features and attention weight matrices are aggregated to generate a joint feature representation.

[0016] The training process of the large visual model includes:

[0017] A feature extraction network is constructed, which includes a multi-scale feature decomposition unit and a spatial structure coding unit;

[0018] The input image is input into the multi-scale feature decomposition unit to perform feature decomposition and obtain multi-level feature maps; the multi-level feature maps are input into the spatial structure encoding unit to extract scene geometric information and generate scene spatial feature representation and scene structural features;

[0019] The scene spatial feature representation is input into the feature matching module to calculate the spatial transformation matrix between viewpoints. Based on the spatial transformation matrix, a geometric consistency loss is constructed. The geometric consistency loss and the multi-level feature mapping are input into the feature alignment module to generate feature alignment results and feature matching constraints.

[0020] The scene structure features and feature matching constraints are input into the attention calculation module to generate feature attention weights. The feature attention weights and feature alignment results are input into the feature fusion module to perform feature aggregation operations and output joint feature representations.

[0021] The joint feature representation and geometric consistency loss are input into the network optimization module to calculate the parameter gradient, update the parameters in the feature extraction network, and repeat the iteration until the geometric consistency loss value converges to obtain the trained large visual model.

[0022] Based on the joint feature representation, the three-dimensional coordinates and semantic probability distribution of spatial sampling points are generated in three-dimensional space, and the spatial semantic field is constructed, including:

[0023] The joint feature representation is input into the multi-scale feature decomposition unit and the spatial structure coding unit to generate multi-level feature maps and scene spatial feature representations.

[0024] A three-dimensional spatial sampling grid is constructed based on the scene spatial feature representation, and initial spatial sampling points are generated within the three-dimensional spatial sampling grid.

[0025] Calculate the local density distribution of the initial spatial sampling points, and adjust the position of the spatial sampling points according to the local density distribution to generate uniformly distributed spatial sampling points.

[0026] The uniformly distributed spatial sampling points are projected onto a multi-level feature mapping space, and the feature vectors at the projection positions are extracted. The three-dimensional coordinate offset of the spatial sampling points is calculated based on the feature vectors.

[0027] The three-dimensional coordinate offset is superimposed onto the coordinates of the uniformly distributed spatial sampling points to generate the three-dimensional coordinate values ​​of the spatial sampling points.

[0028] The three-dimensional coordinates of the spatial sampling points and the joint feature representation are used to generate semantic features of the spatial sampling points through feature aggregation operations. Based on the semantic features and the scene space feature representation, a probability distribution function is constructed to calculate the semantic category probability distribution at the location of the spatial sampling point and generate a semantic probability distribution. The three-dimensional coordinates of the spatial sampling points and the semantic probability distribution are combined to construct a spatial semantic field.

[0029] The spatial sampling points in the spatial semantic field are clustered into a spatial sampling point set using an attention weight matrix. Consistency enhancement is applied to the semantic probability distribution within the spatial sampling point set to obtain an enhanced semantic probability distribution. Converting the enhanced semantic probability distribution into deterministic semantic labels includes:

[0030] Project the spatial sampling points in the spatial semantic field onto the feature space corresponding to the attention weight matrix, and calculate the structural representation vector of the spatial sampling points.

[0031] The structural representation matrix of the spatial sampling points is calculated based on the structural representation vector. Spatial continuity constraints are introduced into the structural representation matrix, and matrix decomposition is performed to divide the spatial sampling points into multiple sets of spatial sampling points.

[0032] Calculate the spatial distance distribution and semantic probability distribution within the set of spatial sampling points, construct probability propagation weights based on the spatial distance distribution, and combine the probability propagation weights with the semantic probability distribution to generate a probability update matrix;

[0033] The semantic probability distribution is updated in multiple rounds using a probability update matrix. Spatial distance constraints are applied in each round of iteration, and the enhanced semantic probability distribution is output.

[0034] Spatial structural features are extracted from the structural representation matrix, and semantic distribution features are extracted from the enhanced semantic probability distribution. The spatial structural features and semantic distribution features are combined to construct a feature transformation matrix. The enhanced semantic probability distribution is mapped to deterministic semantic labels using the feature transformation matrix.

[0035] A semantically guided geometric optimization objective function is constructed based on deterministic semantic labels and 3D coordinate values. The 3D coordinate values ​​of spatial sampling points are adjusted by minimizing the geometric optimization objective function, resulting in optimized 3D coordinate values ​​including:

[0036] Semantic structural components and spatial structural components are decomposed from deterministic semantic labels. A semantic topology matrix is ​​calculated based on the semantic structural components, and a spatial geometry matrix is ​​calculated based on the spatial structural components. The semantic topology matrix and the spatial geometry matrix are combined to generate a structural constraint matrix.

[0037] Calculate the distance matrix between three-dimensional coordinate values, and obtain the neighborhood coordinate difference based on the distance matrix; combine the neighborhood coordinate difference with the structural constraint matrix to obtain weight coefficients, use the weight coefficients to construct spatial constraint terms, and combine the spatial constraint terms with semantic structural components to generate a semantically guided geometric optimization objective function;

[0038] Calculate the gradient vector of the semantically guided geometric optimization objective function with respect to the 3D coordinate values, calculate the coordinate update step size based on the distance matrix, and multiply the coordinate update step size by the gradient vector to obtain the coordinate offset.

[0039] The coordinate offset is superimposed on the three-dimensional coordinate value, and the iteration is repeated until the semantically guided geometric optimization objective function value is less than the preset convergence threshold, thus obtaining the optimized three-dimensional coordinate value.

[0040] Optimized 3D coordinates of continuous spatial sampling points with the same deterministic semantic labels are extracted to form object boundaries. A scene topology graph is constructed based on the inclusion and adjacency relationships between object boundaries. The scene functional structure is inferred from the scene topology graph. Navigation paths are generated based on the object boundaries and the scene functional structure, including:

[0041] Calculate the density distribution vector of continuous spatial sampling points with the same deterministic semantic labels, construct a multi-scale density field, extract the density gradient direction in the multi-scale density field and convert it into the boundary growth direction;

[0042] Adaptive region expansion is performed on the optimized 3D coordinate values ​​along the boundary growth direction to form the object boundary and extract the spatial contact points and spatial nesting points between the object boundaries.

[0043] Map spatial contact points and spatial nested points to topological nodes, and construct containment relationship connecting edges and adjacency relationship connecting edges; construct a scene topology graph based on the topological nodes, containment relationship connecting edges and adjacency relationship connecting edges, extract the topological node connection patterns from the scene topology graph, and infer the scene functional structure;

[0044] The object boundary is converted into a spatial cost matrix, the scene functional structure is converted into a motion constraint matrix, the motion constraint matrix is ​​superimposed on the spatial cost matrix, and path search is performed on the superimposed spatial cost matrix to generate a navigation path.

[0045] A second aspect of this invention provides a scene reconstruction and semantic understanding system based on a large visual model, comprising:

[0046] The first unit is used to acquire multi-view image data of the target scene, input the multi-view image data into a pre-trained large visual model, and output joint feature representation and attention weight matrix.

[0047] The second unit is used to generate the three-dimensional coordinates and semantic probability distribution of spatial sampling points in three-dimensional space based on the joint feature representation, and to construct a spatial semantic field.

[0048] The third unit is used to cluster spatial sampling points in the spatial semantic field into a set of spatial sampling points using an attention weight matrix, perform consistency enhancement on the semantic probability distribution within the set of spatial sampling points to obtain an enhanced semantic probability distribution, and convert the enhanced semantic probability distribution into a deterministic semantic label.

[0049] The fourth unit is used to construct a semantically guided geometric optimization objective function based on deterministic semantic labels and three-dimensional coordinate values. The three-dimensional coordinate values ​​of the spatial sampling points are adjusted by minimizing the geometric optimization objective function to obtain the optimized three-dimensional coordinate values.

[0050] The fifth unit is used to extract the optimized 3D coordinate values ​​of continuous spatial sampling points with the same deterministic semantic labels to form object boundaries, construct scene topology graphs based on the inclusion and adjacency relationships between object boundaries, infer scene functional structure based on scene topology graphs, and generate navigation paths based on object boundaries and scene functional structure.

[0051] A third aspect of the present invention provides an electronic device, comprising:

[0052] processor;

[0053] Memory used to store processor-executable instructions;

[0054] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0055] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0056] In this embodiment, by combining a large visual model with 3D scene reconstruction, end-to-end processing from multi-view images to semantic 3D scenes is achieved. The large visual model can extract rich image features and attention information, providing high-quality feature representations for subsequent 3D reconstruction, avoiding the problem of insufficient feature extraction in traditional methods, and improving the accuracy and robustness of scene reconstruction. A semantic consistency enhancement mechanism guided by the attention weight matrix effectively solves the problem of inconsistent semantic annotations in 3D space. This achieves mutual promotion between semantic understanding and geometric reconstruction, ensuring that the reconstructed 3D scene has both accurate geometric structure and accurate semantic information, improving the completeness of scene representation. By constructing a scene topology graph and inferring the scene's functional structure, a leap from low-level geometric reconstruction to high-level scene understanding is achieved. The topology structure is automatically constructed based on the spatial relationships between object boundaries, thereby inferring the scene's functional layout and ultimately generating navigation paths usable in practical applications. This allows the reconstruction results to go beyond mere visualization, directly supporting downstream tasks such as robot navigation and augmented reality, significantly enhancing the practical value of the technology. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the scene reconstruction and semantic understanding method based on a large visual model according to an embodiment of the present invention.

[0058] Figure 2 This is a flowchart of the three-dimensional scene navigation path planning process according to an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0061] Figure 1 This is a flowchart illustrating the scene reconstruction and semantic understanding method based on a large visual model according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0062] Acquire multi-view image data of the target scene, input the multi-view image data into a pre-trained large visual model, and output joint feature representation and attention weight matrix;

[0063] Based on the joint feature representation, the three-dimensional coordinates and semantic probability distribution of spatial sampling points are generated in three-dimensional space to construct a spatial semantic field.

[0064] The spatial sampling points in the spatial semantic field are clustered into a set of spatial sampling points using an attention weight matrix. The semantic probability distribution within the set of spatial sampling points is then enhanced to obtain an enhanced semantic probability distribution. The enhanced semantic probability distribution is then converted into a deterministic semantic label.

[0065] A semantically guided geometric optimization objective function is constructed based on deterministic semantic labels and three-dimensional coordinate values. The three-dimensional coordinate values ​​of spatial sampling points are adjusted by minimizing the geometric optimization objective function to obtain the optimized three-dimensional coordinate values.

[0066] The optimized 3D coordinates of continuous spatial sampling points with the same deterministic semantic labels are extracted to form object boundaries. A scene topology graph is constructed based on the inclusion and adjacency relationships between object boundaries. The scene functional structure is inferred based on the scene topology graph. A navigation path is generated based on the object boundaries and the scene functional structure.

[0067] In one optional implementation, multi-view image data of the target scene is acquired, and the multi-view image data is input into a pre-trained large visual model to output a joint feature representation and attention weight matrix, including:

[0068] The original multi-view image data of the target scene is acquired, the spatial pose calibration of the original multi-view image data is performed, the spatial transformation matrix between each view is calculated, and geometric correction is performed based on the spatial transformation matrix to generate multi-view image data with spatial pose information.

[0069] Multi-view image data with spatial pose information is divided into multiple view groups. Image preprocessing is performed on the images within the multiple view groups. Spatial consistency correction is then performed on the preprocessed view groups based on the spatial pose information to generate standardized multi-view image data.

[0070] Standardized multi-view image data is input into a pre-trained large visual model according to the viewpoint order determined by spatial pose information. Multi-level visual features are extracted, and the feature similarity between viewpoints is calculated based on the multi-level visual features and the spatial transformation matrix to generate an attention weight matrix.

[0071] Multi-level visual features and attention weight matrices are aggregated to generate a joint feature representation.

[0072] First, raw multi-view image data of the target scene needs to be acquired. This acquisition can be achieved by simultaneously capturing images from multiple cameras positioned around the target scene, or by sequentially capturing images from different angles using a single camera. The acquired raw image data typically contains scene information from different perspectives, but lacks accurate spatial pose information. Spatial pose calibration of the raw multi-view image data can be performed using feature point matching. This involves extracting salient feature points from each image, such as corner points or SIFT feature points, and calculating the spatial transformation matrix between each perspective based on the matching relationships between corresponding feature points. Taking two adjacent perspective images as an example, after extracting feature points, the RANSAC algorithm is used to filter feature point matching pairs. Based on the filtered matching pairs, the fundamental or essential matrix between perspectives is calculated, which is then decomposed into rotation matrices and translation vectors to form the spatial transformation matrix. In practical applications, if 512 feature points are extracted from each of the two perspective images, initial matching may yield 200 matching pairs. After RANSAC filtering, 120 high-quality matching pairs are retained for calculating the spatial transformation matrix.

[0073] Geometric correction is performed based on the calculated spatial transformation matrix to eliminate perspective distortion and aberration caused by differences in viewpoints between images. Specifically, one viewpoint is selected as a reference viewpoint, and images from other viewpoints are transformed to the coordinate system of the reference viewpoint using the spatial transformation matrix. For each pixel, its corresponding position in the original image is determined using inverse mapping, and the transformed pixel value is calculated using bilinear interpolation. The geometrically corrected image data, together with the corresponding spatial pose information, constitutes multi-view image data with spatial pose information.

[0074] Multi-view image data with spatial pose information is divided into multiple view groups based on their spatial proximity. This grouping can be achieved using spatial clustering methods, calculating the Euclidean distance or cosine similarity between viewpoints and grouping spatially close viewpoints together. For large scenes, this may result in 3-5 view groups, each containing 5-10 images from adjacent viewpoints. Image preprocessing is performed on the images within each view group, including brightness adjustment, color balancing, and contrast enhancement. Brightness adjustment is achieved through histogram equalization, stretching the image's grayscale distribution to the entire grayscale range; color balancing adjusts the image's color temperature using a white balance algorithm; and contrast enhancement is achieved by stretching the image's pixel value distribution range. After image preprocessing, spatial consistency correction is performed on the preprocessed view groups based on spatial pose information to ensure accurate spatial correspondence between images within the same view group. Spatial consistency correction is achieved by refining the spatial transformation matrix. For images within a view group, existing spatial pose information is used, combined with local feature matching, to optimize the spatial transformation matrix and reduce accumulated errors. The multi-view image data after spatial consistency correction is called standardized multi-view image data.

[0075] Standardized multi-view image data is input into a pre-trained large-scale visual model according to the viewpoint order determined by spatial pose information. The viewpoint order can be determined based on the spatial continuity between viewpoints, ensuring high content coherence between adjacent input viewpoint images. The large-scale visual model employs a multi-layer convolutional neural network structure, including a feature extraction module and a feature fusion module. The feature extraction module consists of multiple convolutional layers, pooling layers, and normalization layers, capable of extracting visual features at different scales and levels of abstraction. For each viewpoint image, the feature extraction module extracts multi-level visual features, including low-level texture features, mid-level structural features, and high-level semantic features. Taking an input image from a certain viewpoint as an example, after processing by the large-scale visual model, feature maps corresponding to shallow, mid, and deep layers can be obtained, with feature dimensions potentially being 64×256×256, 128×128×128, and 256×64×64.

[0076] Feature similarity between viewpoints is calculated based on extracted multi-level visual features and a spatial transformation matrix. For any two viewpoint feature maps, the feature map of one viewpoint is first transformed into the coordinate system of the other viewpoint using the spatial transformation matrix. Then, the cosine similarity or dot product of the feature vectors at corresponding positions is calculated to obtain the feature similarity score between the two viewpoints. The feature similarity between all viewpoint pairs is organized into a matrix form, namely the attention weight matrix. The attention weight matrix reflects the degree of information association between different viewpoints; the higher the weight value, the stronger the content similarity between the corresponding viewpoints.

[0077] Multi-level visual features and attention weight matrices are aggregated to generate a joint feature representation. The feature aggregation process employs a weighted fusion method, calculating a weighted sum for each viewpoint's features based on its attention weights with other views. Specifically, for the target viewpoint's features, the features from all views are weighted and summed according to the corresponding row in the attention weight matrix to obtain the enhanced features for that viewpoint. The enhanced features from all views are then fused again, which can be achieved through average pooling, max pooling, or an attention mechanism, ultimately generating a joint feature representation containing multi-view information. The joint feature representation typically has a dimension of B×C×H×W, where B is the batch size, C is the number of feature channels, and H and W are the feature space dimensions.

[0078] The aforementioned scene reconstruction and semantic understanding methods based on large visual models fully utilize complementary information from multi-view image data, improving the accuracy and completeness of scene reconstruction. They effectively address the occlusion and detail loss problems caused by insufficient information from a single viewpoint, significantly enhancing the 3D reconstruction quality and semantic understanding accuracy of complex scenes. The multi-view image processing strategy based on spatial pose information ensures spatial consistency across different viewpoints, reducing feature mismatch issues caused by viewpoint changes. The multi-level visual feature extraction and fusion mechanism considers both the geometric structure and semantic content of the scene, ensuring that the reconstruction results conform to physical spatial constraints while possessing rich semantic information.

[0079] In one optional implementation, the training process of the large visual model includes:

[0080] A feature extraction network is constructed, which includes a multi-scale feature decomposition unit and a spatial structure coding unit;

[0081] The input image is input into the multi-scale feature decomposition unit to perform feature decomposition and obtain multi-level feature maps; the multi-level feature maps are input into the spatial structure encoding unit to extract scene geometric information and generate scene spatial feature representation and scene structural features;

[0082] The scene spatial feature representation is input into the feature matching module to calculate the spatial transformation matrix between viewpoints. Based on the spatial transformation matrix, a geometric consistency loss is constructed. The geometric consistency loss and the multi-level feature mapping are input into the feature alignment module to generate feature alignment results and feature matching constraints.

[0083] The scene structure features and feature matching constraints are input into the attention calculation module to generate feature attention weights. The feature attention weights and feature alignment results are input into the feature fusion module to perform feature aggregation operations and output joint feature representations.

[0084] The joint feature representation and geometric consistency loss are input into the network optimization module to calculate the parameter gradient, update the parameters in the feature extraction network, and repeat the iteration until the geometric consistency loss value converges to obtain the trained large visual model.

[0085] The training process for the large-scale visual model begins with the construction of a feature extraction network. This network employs a deep convolutional neural network architecture with 50 layers. The input layer receives RGB image data with a resolution of 512×512 pixels. The multi-scale feature decomposition unit (MLU) contains five convolutional branches with different receptive fields. The kernel sizes for each branch are 3×3, 5×5, 7×7, 9×9, and 11×11, with a uniform stride of 1 and symmetrical edge padding. Each convolutional branch is followed by a batch normalization layer and an activation function layer, using the ReLU function. The number of output channels for the MLU increases from shallow to deep layers, specifically set to 64, 128, 256, 512, and 1024 channels.

[0086] The input image is processed by a multi-scale feature decomposition unit to generate a five-layer feature map. The first layer, with a size of 128×128×64, captures image edge and texture information. The second layer, with a size of 64×64×128, extracts local pattern features. The third layer, with a size of 32×32×256, encodes mid-level semantic information. The fourth layer, with a size of 16×16×512, represents high-level abstract features. The fifth layer, with a size of 8×8×1024, contains global semantic representation. In a real-world data example, given a street view image containing buildings, the first layer extracts details such as window frames and wall textures, the third layer identifies the complete structure of windows and doors, and the fifth layer represents the type and spatial layout of the entire building.

[0087] The spatial structure encoding unit receives five layers of feature maps as input and extracts spatial relationships through three-dimensional convolution operations. The unit includes a spatial pyramid pooling module, which divides the input features into 1×1, 2×2, 4×4, and 8×8 grid regions, performing max pooling and average pooling operations on each region to extract spatial statistical information at different granularities. The pooled feature vectors are concatenated into an 85-channel feature descriptor. The unit also includes a position encoder, which generates a two-dimensional coordinate code for each position in the feature map. The coordinate encoding uses a sine-cosine position encoding method, mapping the horizontal and vertical coordinate values ​​to a 128-dimensional vector space. The position encoded vector is element-wise added to the original feature vector to generate an enhanced feature representation containing spatial position information.

[0088] The scene spatial feature representation has a dimension of 8×8×1152, where the 1152 dimensions consist of 1024 original features and 128 positional encodings. Scene structural features are obtained by performing global average pooling on the spatial feature representation, forming a 1152-dimensional global description vector. In the case of building street view images, the first 256 dimensions of the scene structural features primarily encode the vertical line information of the buildings, the middle 384 dimensions represent horizontal structural elements, and the last 512 dimensions capture the overall geometric shape.

[0089] The feature matching module receives scene spatial feature representations from two images with different viewpoints and establishes pixel-level correspondences through cross-correlation calculations. For a feature vector with 4×4 position coordinates in the first viewpoint feature representation, the most similar feature vector is searched within the 8×8 spatial range of the second viewpoint feature representation, with cosine similarity used for similarity measurement. After finding the corresponding position, the coordinate offset is recorded, and 64 sets of coordinate correspondences are obtained after traversing all positions. Based on these correspondences, the spatial transformation matrix between the two viewpoints is estimated using a least-squares fitting method. This matrix is ​​a 3×3 homography matrix containing rotation, translation, and scaling parameters.

[0090] The geometric consistency loss is calculated by projecting the coordinates of feature points from the first viewpoint using a spatial transformation matrix to obtain the predicted coordinates from the second viewpoint. The predicted coordinates are then compared with the actual matched coordinates to calculate the Euclidean distance error. The sum of the distance errors for all feature points is divided by the total number of feature points to obtain the average geometric consistency loss value. In a real-world example of building images, for two images with a 15-degree difference in viewpoint, the initial geometric consistency loss value is 23.6 pixels, indicating an average positioning error of 23.6 pixels.

[0091] The feature alignment module employs a deformable convolutional network to achieve spatial alignment of features. The deformable convolution adds a spatial offset parameter to each sampling point on top of the standard convolution; this offset parameter is predicted and generated through independent convolutional layers. The feature alignment module contains three deformable convolutional layers, each with a 3×3 kernel size and maintaining 1024 output channels. The first deformable convolutional layer predicts a coarse spatial offset based on initial transformation information provided by the geometric consistency loss, with the offset range limited to between -5 and +5 pixels. The second and third layers progressively refine the offset prediction, ultimately achieving sub-pixel accuracy. The feature alignment result maintains an 8×8×1024 spatial dimension, but the feature vectors are spatially aligned with the reference viewpoint.

[0092] Feature matching constraints consist of a matching confidence map and a matching mask. The matching confidence map records the reliability of each feature point's match, with values ​​ranging from 0 to 1, where 1 represents a high-confidence match. The matching mask is a binary map that marks which locations have valid matches and which locations cannot be matched due to occlusion or being outside the field of view. In the building example, the matching confidence of the window areas on the building's facade is generally above 0.85, while the sky areas at the building's edge have a matching confidence of only around 0.3 due to a lack of texture features.

[0093] The attention calculation module uses scene structural features as the query vector and feature matching constraints as key-value pairs, calculating feature importance weights through a dot product attention mechanism. The 1152-dimensional vector of scene structural features is mapped to a 512-dimensional query vector through a linear transformation layer. The matching confidence map in the feature matching constraints is flattened into a 64-dimensional vector, with each element corresponding to a spatial location, and mapped to a 512-dimensional key vector and value vector through a linear transformation. The query vector is dot-productted with each of the 64 key vectors to obtain 64 attention scores. A softmax normalization function is applied to the attention scores to generate 64 weight coefficients, the sum of which is 1. These 64 weight coefficients constitute the feature attention weights, reflecting the contribution of different spatial locations to the final feature representation. In the building example, the attention weight for clearly defined areas such as windows reaches 0.025, while the weight for blurred areas drops to 0.008.

[0094] The feature fusion module performs a weighted summation operation, expanding the 8×8×1024 feature map of the feature alignment result into 64 1024-dimensional feature vectors according to their spatial location. Each feature vector is multiplied by its corresponding attention weight and then summed to obtain a single 1024-dimensional joint feature representation vector. This vector integrates complementary information from multiple perspectives, highlighting the features of reliable matching regions and suppressing interference from uncertain regions.

[0095] The network optimization module receives the joint feature representation and geometric consistency loss, and calculates the gradient of the network parameters using the backpropagation algorithm. The optimizer employs the adaptive moment estimation algorithm, with an initial learning rate of 0.0001, which decays to 0.9 times its original value every 10 training epochs. The batch size is set to 16, meaning 16 pairs of images are processed simultaneously each time. After gradient calculation, the weight and bias parameters of all convolutional, normalization, and fully connected layers in the feature extraction network are updated. The training process lasts for 100 epochs, traversing all training samples in each epoch. During training on the building dataset, the geometric consistency loss gradually decreases from an initial 23.6 pixels, down to 12.4 pixels in epoch 20, 5.8 pixels in epoch 50, 2.3 pixels in epoch 80, and converges to 1.9 pixels in epoch 100. Convergence is defined as a loss change of less than 0.1 pixels within 5 consecutive epochs, resulting in a trained large visual model.

[0096] In this embodiment, multi-scale feature decomposition and spatial structure encoding effectively capture the details and overall structural information of the scene. The feature matching module and geometric consistency constraints ensure spatial consistency of features from different viewpoints, resolving the feature mismatch problem caused by viewpoint changes. Attention mechanisms and feature fusion strategies enhance the system's ability to perceive key areas of the scene, improving its adaptability to complex scenes. The overall training process promotes a deep understanding of the geometric relationships in multi-view scenes, laying the foundation for high-precision 3D reconstruction and semantic analysis.

[0097] In one optional implementation, the spatial semantic field is constructed by generating the three-dimensional coordinates and semantic probability distribution of spatial sampling points in three-dimensional space based on the joint feature representation, including:

[0098] The joint feature representation is input into the multi-scale feature decomposition unit and the spatial structure coding unit to generate multi-level feature maps and scene spatial feature representations.

[0099] A three-dimensional spatial sampling grid is constructed based on the scene spatial feature representation, and initial spatial sampling points are generated within the three-dimensional spatial sampling grid.

[0100] Calculate the local density distribution of the initial spatial sampling points, and adjust the position of the spatial sampling points according to the local density distribution to generate uniformly distributed spatial sampling points.

[0101] The uniformly distributed spatial sampling points are projected onto a multi-level feature mapping space, and the feature vectors at the projection positions are extracted. The three-dimensional coordinate offset of the spatial sampling points is calculated based on the feature vectors.

[0102] The three-dimensional coordinate offset is superimposed onto the coordinates of the uniformly distributed spatial sampling points to generate the three-dimensional coordinate values ​​of the spatial sampling points.

[0103] The three-dimensional coordinates of the spatial sampling points and the joint feature representation are used to generate semantic features of the spatial sampling points through feature aggregation operations. Based on the semantic features and the scene space feature representation, a probability distribution function is constructed to calculate the semantic category probability distribution at the location of the spatial sampling point and generate a semantic probability distribution. The three-dimensional coordinates of the spatial sampling points and the semantic probability distribution are combined to construct a spatial semantic field.

[0104] The joint feature representation is input into the multi-scale feature decomposition unit and the spatial structure encoding unit to generate multi-level feature maps and scene spatial feature representations. The multi-scale feature decomposition unit adopts a residual network structure, containing multiple convolutional blocks. Each convolutional block consists of multiple convolutional layers, batch normalization layers, and activation function layers. Downsampling operations are used between convolutional blocks to progressively reduce the spatial size of the feature maps while increasing the number of feature channels, forming a feature hierarchy from shallow to deep layers. In practical applications, the joint feature representation size is 256×64×64. After processing by the multi-scale feature decomposition unit, three-layer feature maps with sizes of 128×32×32, 64×16×16, and 32×8×8 are obtained, forming a multi-level feature map. The spatial structure encoding unit adopts an encoder-decoder architecture. The encoder part uses a self-attention mechanism to extract spatial correlation information within the feature map, and the decoder part restores the spatial resolution of the features through transposed convolutions and skip connections. After processing by the spatial structure coding unit, the joint feature representation is converted into a scene spatial feature representation that includes geometric attributes such as scene depth information and surface normal vectors, and its size is the same as the original joint feature representation.

[0105] A 3D spatial sampling mesh is constructed based on scene spatial feature representation, and initial spatial sampling points are generated within the 3D spatial sampling mesh. When constructing the 3D spatial sampling mesh, the spatial boundaries of the scene must first be determined. By analyzing the depth distribution in the scene spatial feature representation, the minimum and maximum values ​​of the scene in the x, y, and z dimensions are determined, forming a cubic space enclosing the scene. This cubic space is uniformly divided into a mesh structure, and the mesh resolution can be dynamically adjusted according to scene complexity and computational resources. In practical applications, the space can be divided into a 64×64×64 mesh. Initial spatial sampling points are generated within the constructed 3D spatial sampling mesh; the sampling point location can be selected from the vertices or center points of the mesh. For a 64×64×64 mesh, a total of 262,144 initial spatial sampling points are generated. The density of the initial spatial sampling points is uniformly distributed in space, but this uniform distribution may not accurately represent the geometric details of the scene.

[0106] The local density distribution of initial spatial sampling points is calculated, and the positions of these points are adjusted based on this distribution to generate a uniformly distributed spatial sampling point network. The calculation of the local density distribution is based on geometric information in the scene's spatial feature representation, particularly the depth gradient and surface normal vectors. For each initial spatial sampling point, the magnitude of the depth gradient within its surrounding region is calculated. Regions with large gradients indicate significant geometric changes in the scene, requiring higher sampling density. Simultaneously, the variation in surface normal vectors is considered; regions with drastic normal vector changes typically correspond to edges or corners in the scene, also requiring denser sampling. Based on the calculated local density distribution, the positions of the initial spatial sampling points are adjusted. Sampling point density is increased in geometrically complex regions and appropriately decreased in flat regions. This adjustment process is achieved through iterative optimization. In each iteration, a displacement vector is calculated based on the difference between the current sampling point's local density and the target density, and the sampling point is moved along the direction of this displacement vector. After multiple iterations, the initial spatial sampling points gradually form a distribution adapted to the scene's geometric complexity, generating a uniformly distributed spatial sampling point network.

[0107] Uniformly distributed spatial sampling points are projected onto a multi-level feature map space. Feature vectors at the projected locations are extracted, and the 3D coordinate offsets of the spatial sampling points are calculated based on these feature vectors. The projection process transforms the sampling points in 3D space onto a 2D feature map plane. For each spatial sampling point, its projected coordinates under different viewpoints are calculated, and feature vectors at the projected locations are extracted from the corresponding multi-level feature map. If a sampling point is invisible from a specific viewpoint, the feature value for that viewpoint is set to zero. Feature vectors from different viewpoints and levels are concatenated to form a comprehensive feature vector. This comprehensive feature vector is processed by a fully connected layer, outputting the 3D coordinate offsets of the spatial sampling points. The offset represents the adjustment of the sampling point's position relative to its current position, used to further refine the spatial location of the sampling points. The calculation of the 3D coordinate offsets considers the consistency of features across multiple viewpoints, effectively reducing geometric inconsistencies between viewpoints.

[0108] The 3D coordinate offsets are superimposed onto the coordinates of uniformly distributed spatial sampling points to generate the 3D coordinate values ​​of these points. The superposition process is a simple coordinate addition operation, adding the calculated offsets to the x, y, and z coordinate components of each sampling point to obtain the adjusted spatial sampling point coordinates. To prevent the superimposed coordinates from exceeding the scene boundaries, a cropping operation can be performed on the superimposed results to ensure that all sampling points remain within the predefined scene space. By superimposing the offsets, the positions of the sampling points are further optimized, more accurately reflecting the scene's geometry.

[0109] The semantic features of spatial sampling points are generated by combining their 3D coordinates with their joint feature representation through feature aggregation. A probability distribution function is then constructed based on these semantic features and the scene spatial feature representation. The semantic category probability distribution at the sampling point's location is calculated, generating the semantic probability distribution. The feature aggregation operation determines the projection position of the sampling point in different viewpoint feature spaces using its 3D coordinates and extracts the corresponding feature values ​​from the joint feature representation. For each sampling point, its feature values ​​are collected from all visible viewpoints, and these features are fused using a weighted average method. The weights are determined based on viewpoint visibility and projection distance. The fused features constitute the semantic features of that sampling point. A multilayer perceptron structure, containing multiple fully connected layers and activation function layers, is used to construct the probability distribution function based on the semantic features. The multilayer perceptron maps semantic features to probability vectors of semantic categories, where each element corresponds to a probability value for a semantic category. In practical applications, if the scene contains 20 semantic categories, the probability vector has a dimension of 20, and each element represents the probability that the sampling point belongs to that category. Normalization of the probability vector ensures that the sum of all category probabilities is 1, resulting in the semantic category probability distribution at the sampling point's location.

[0110] A spatial semantic field is constructed by combining the 3D coordinates of spatial sampling points with semantic probability distributions. The combination process associates the 3D coordinates of each sampling point with its corresponding semantic probability distribution, forming a data structure containing both location and semantic information. This data structure can be represented as a high-dimensional tensor, where the first three dimensions correspond to the x, y, and z coordinates of space, and subsequent dimensions store the probability values ​​of different semantic categories. Through interpolation algorithms, the semantic probability distribution of any point in space can be queried, achieving a continuous semantic representation of the scene space. The constructed spatial semantic field contains both the geometric structure information of the scene and the semantic classification information of objects within the scene, providing fundamental support for subsequent scene understanding and interactive applications.

[0111] In this embodiment, multi-scale feature decomposition and spatial structure encoding effectively capture multi-level features of the scene, preserving rich visual details. An adaptive sampling strategy based on local density distribution optimizes the spatial distribution of sampling points, providing finer sampling in complex regions and improving the accuracy of geometric reconstruction. A multi-view feature fusion mechanism solves the occlusion problem caused by insufficient information from a single viewpoint, enhancing the system's robustness to complex scenes. The introduction of semantic probability distribution enables a leap from pixel-level to scene-level semantic understanding, providing a solid foundation for object recognition and relational reasoning within the scene.

[0112] In one optional implementation, spatial sampling points in the spatial semantic field are clustered into a set of spatial sampling points using an attention weight matrix. Consistency enhancement is applied to the semantic probability distribution within the set of spatial sampling points to obtain an enhanced semantic probability distribution. Converting the enhanced semantic probability distribution into deterministic semantic labels includes:

[0113] Project the spatial sampling points in the spatial semantic field onto the feature space corresponding to the attention weight matrix, and calculate the structural representation vector of the spatial sampling points.

[0114] The structural representation matrix of the spatial sampling points is calculated based on the structural representation vector. Spatial continuity constraints are introduced into the structural representation matrix, and matrix decomposition is performed to divide the spatial sampling points into multiple sets of spatial sampling points.

[0115] Calculate the spatial distance distribution and semantic probability distribution within the set of spatial sampling points, construct probability propagation weights based on the spatial distance distribution, and combine the probability propagation weights with the semantic probability distribution to generate a probability update matrix;

[0116] The semantic probability distribution is updated in multiple rounds using a probability update matrix. Spatial distance constraints are applied in each round of iteration, and the enhanced semantic probability distribution is output.

[0117] Spatial structural features are extracted from the structural representation matrix, and semantic distribution features are extracted from the enhanced semantic probability distribution. The spatial structural features and semantic distribution features are combined to construct a feature transformation matrix. The enhanced semantic probability distribution is mapped to deterministic semantic labels using the feature transformation matrix.

[0118] The spatial sampling points in the spatial semantic field are projected onto the feature space corresponding to the attention weight matrix, and the structural representation vector of the spatial sampling points is calculated. The projection process requires determining the corresponding position of the spatial sampling points in the feature space of the attention weight matrix. The attention weight matrix is ​​usually represented as a high-dimensional tensor, whose dimension is related to the spatial size of the feature map in the large visual model. The spatial sampling points are mapped to the two-dimensional feature space through their three-dimensional coordinates, which can be done using perspective projection or orthogonal projection methods. For perspective projection, the three-dimensional points are transformed to the camera coordinate system according to the camera's intrinsic and extrinsic parameters, and then projected onto the image plane. The projected two-dimensional coordinates are used to index eigenvalues ​​from the attention weight matrix. If the projected coordinates are not integers, the eigenvalues ​​are calculated using bilinear interpolation. For each sampling point in the spatial semantic field, eigenvalues ​​are extracted from the attention weight matrices of all visible viewpoints to form the structural representation vector of that sampling point. The structural representation vector reflects the correlation strength between the sampling point and other regions in the scene and contains rich structural information. In practical applications, if there are 8 viewpoints and the attention weight matrix size is 64×64, the dimension of the structural representation vector for each sampling point can reach 32768.

[0119] The structural representation matrix of spatial sampling points is calculated based on structural representation vectors. Spatial continuity constraints are introduced into the structural representation matrix, and matrix factorization is performed to divide the spatial sampling points into multiple sets of spatial sampling points. The structural representation matrix consists of the structural representation vectors of all sampling points, with the number of rows equal to the number of sampling points and the number of columns equal to the dimension of the structural representation vectors. The spatial continuity constraint can be introduced into the structural representation matrix by adding spatial distance weights. The spatial distance weights are calculated based on the Euclidean distance between sampling points, with closer points having larger weights. The modified structural representation matrix considers both feature similarity and spatial proximity, making it more suitable for spatial clustering. Matrix factorization can be performed on the structural representation matrix using singular value decomposition or nonnegative matrix factorization methods. Matrix factorization yields a low-dimensional representation and a component matrix. The low-dimensional representation reflects the main structural features of the sampling points, and the component matrix indicates the degree of association between the sampling points and potential categories. Based on the values ​​of the component matrix, each sampling point can be assigned to the most relevant category, forming a set of spatial sampling points. In a scenario containing 10,000 sampling points, it is possible to divide the sampling point into 50-200 sets, each set containing sampling points with similar structural features.

[0120] The process calculates the spatial distance distribution and semantic probability distribution within a set of spatial sampling points. Based on the spatial distance distribution, probability propagation weights are constructed. These weights are then combined with the semantic probability distribution to generate a probability update matrix. For each set of spatial sampling points, the spatial distance matrix between all sampling points within the set is calculated, recording the 3D Euclidean distance between point pairs. Simultaneously, the semantic probability distribution of each sampling point is extracted from the spatial semantic field, forming the semantic probability distribution matrix of the set. When constructing the probability propagation weights based on the spatial distance distribution, a Gaussian kernel function can be used; closer point pairs have higher weights, while point pairs with distances exceeding a threshold have weights close to zero. Each element of the probability propagation weight matrix represents the probability influence strength from one sampling point to another. Combining the probability propagation weights with the semantic probability distribution can be achieved through matrix multiplication to obtain the probability update matrix. The probability update matrix guides how each sampling point updates its own semantic probability distribution during subsequent iterations.

[0121] The semantic probability distribution is iteratively updated multiple times using a probability update matrix. Spatial distance constraints are applied in each iteration, resulting in an enhanced semantic probability distribution. The iterative update process employs an information propagation mechanism, where each sampling point updates its semantic probability based on its relationship with its neighbors. In the first iteration, the initial semantic probabilities of the sampling points come from the spatial semantic field. Each iteration includes two steps: probability propagation and normalization. In the probability propagation phase, each sampling point receives weighted probability values ​​from other sampling points within the set, with the weights determined by the probability update matrix. In the normalization phase, the propagated probability values ​​are normalized to ensure the sum of probabilities equals 1. To maintain spatial continuity, spatial distance constraints are applied in each iteration, suppressing probability propagation between distant sampling points. The iterative process continues until the change in the semantic probability distribution is less than a preset threshold or the maximum number of iterations is reached. In practice, the number of iterations is typically set to 5-10 rounds, sufficient to make the semantic probabilities of sampling points within the set tend to be consistent. After iteration, the enhanced semantic probability distribution is output, characterized by more consistent semantic labels and clearer boundaries in similar regions within the set.

[0122] Spatial structural features are extracted from the structural representation matrix, and semantic distribution features are extracted from the enhanced semantic probability distribution. These spatial structural features and semantic distribution features are combined to construct a feature transformation matrix. The feature transformation matrix is ​​then used to map the enhanced semantic probability distribution to deterministic semantic labels. Spatial structural features can be extracted by performing dimensionality reduction operations on the structural representation matrix, such as principal component analysis or an autoencoder. The extracted spatial structural features reflect the main geometric characteristics and spatial relationships of the sampling points. Semantic distribution features are extracted from the enhanced semantic probability distribution and can be obtained by calculating the statistical properties of the probability distribution, such as entropy, kurtosis, or maximum probability value. Semantic distribution features reflect the degree of determinism and class bias of the probability distribution. The spatial structural features and semantic distribution features are combined to construct the feature transformation matrix. The feature transformation matrix models the association between spatial structure and semantic categories and can be trained using machine learning methods, such as logistic regression or decision trees. When mapping the enhanced semantic probability distribution to deterministic semantic labels using the feature transformation matrix, for each sampling point, its feature vector is multiplied by the transformation matrix to obtain the score for each semantic category. The category with the highest score is selected as the deterministic semantic label for that sampling point. If the semantic category contains 20 options, each sample point will be assigned an integer label from 0 to 19 after transformation.

[0123] In this embodiment, the structural features of spatial sampling points are effectively captured through the calculation of structural representation vectors and matrix factorization, laying the foundation for subsequent clustering. The dual constraints of spatial distance and feature similarity ensure that the clustering results respect both geometric continuity and semantic consistency. The probability propagation mechanism enables effective sharing of semantic information within the set, reducing misclassification of isolated points. The introduction of spatial distance constraints prevents excessive smoothing of semantic labels at object boundaries, preserving detailed information about the scene structure. The combined use of structural and semantic features enhances the robustness of label assignment and reduces label uncertainty caused by insufficient local observations.

[0124] In one optional implementation, a semantically guided geometric optimization objective function is constructed based on deterministic semantic labels and three-dimensional coordinate values. The optimized three-dimensional coordinate values ​​are obtained by adjusting the three-dimensional coordinate values ​​of the spatial sampling points by minimizing the geometric optimization objective function.

[0125] Semantic structural components and spatial structural components are decomposed from deterministic semantic labels. A semantic topology matrix is ​​calculated based on the semantic structural components, and a spatial geometry matrix is ​​calculated based on the spatial structural components. The semantic topology matrix and the spatial geometry matrix are combined to generate a structural constraint matrix.

[0126] Calculate the distance matrix between three-dimensional coordinate values, and obtain the neighborhood coordinate difference based on the distance matrix; combine the neighborhood coordinate difference with the structural constraint matrix to obtain weight coefficients, use the weight coefficients to construct spatial constraint terms, and combine the spatial constraint terms with semantic structural components to generate a semantically guided geometric optimization objective function;

[0127] Calculate the gradient vector of the semantically guided geometric optimization objective function with respect to the 3D coordinate values, calculate the coordinate update step size based on the distance matrix, and multiply the coordinate update step size by the gradient vector to obtain the coordinate offset.

[0128] The coordinate offset is superimposed on the three-dimensional coordinate value, and the iteration is repeated until the semantically guided geometric optimization objective function value is less than the preset convergence threshold, thus obtaining the optimized three-dimensional coordinate value.

[0129] This process decomposes deterministic semantic labels into semantic structural components and spatial structural components. A semantic topology matrix is ​​calculated based on the semantic structural components, and a spatial geometry matrix is ​​calculated based on the spatial structural components. These matrices are then combined to generate a structural constraint matrix. The decomposition process leverages the multi-attribute characteristics of semantic information. Deterministic semantic labels include category identifiers, boundary information, and spatial relationships. Semantic structural components primarily reflect the topological relationships between different semantic categories, extracted through similarity analysis of semantic labels. For each pair of spatial sampling points, high semantic similarity is assigned if their semantic labels are the same; medium similarity if they belong to adjacent categories; and low similarity if the semantic categories differ significantly. In a scenario with 20 semantic categories, a 20×20 category similarity lookup table can be constructed, and the semantic similarity between any two sampling points is obtained by looking up this table. Spatial structural components primarily reflect semantic boundaries and gradient information, obtained by calculating the rate of change of semantic labels in space. For each sampling point, the changes in semantic labels within its neighborhood are analyzed. Regions with drastic label changes correspond to object boundaries and are assigned high structural importance; regions with consistent labels correspond to the interior of objects and are assigned low structural importance. The semantic topology matrix is ​​constructed based on semantic structural components, with the same size as the number of sampling points. Matrix element values ​​represent the semantic association strength between corresponding sampling point pairs. The spatial geometry matrix is ​​constructed based on spatial structural components, reflecting the geometric characteristics of spatial sampling points, such as curvature and normal vector consistency. The semantic topology matrix and spatial geometry matrix are combined using a weighted approach to generate a structural constraint matrix, with the weights dynamically adjusted according to scene characteristics.

[0130] The distance matrix between 3D coordinate values ​​is calculated. Based on this distance matrix, neighborhood coordinate differences are obtained. These neighborhood coordinate differences are combined with the structural constraint matrix to obtain weighting coefficients. Spatial constraint terms are constructed using these weighting coefficients. These spatial constraint terms are then combined with semantic structural components to generate a semantically guided geometric optimization objective function. The distance matrix is ​​calculated using Euclidean distance. For any two spatial sampling points, the square root of the sum of the squares of their 3D coordinates in the x, y, and z directions is calculated. The size of the distance matrix is ​​the same as the number of sampling points, and the diagonal elements are zero. When obtaining neighborhood coordinate differences based on the distance matrix, for each sampling point, points with a distance less than a preset threshold are selected as its neighbors, and the coordinate difference between this sampling point and its neighbors is calculated. The neighborhood radius can be set to 2-3 times the average spacing between sampling points to ensure that each point has a sufficient number of neighbors. The neighborhood coordinate differences are combined with the structural constraint matrix to obtain weighting coefficients. The combination method is element-wise multiplication, i.e., the neighborhood coordinate difference at the corresponding position is multiplied by the value in the structural constraint matrix. The weighting coefficients reflect the mutual influence strength between different pairs of sampling points, while considering both spatial distance and structural similarity. Spatial constraints are constructed using weighted coefficients. These constraints are in the form of a weighted sum of squared coordinate differences; a larger weight indicates that the coordinates of corresponding point pairs should be closer. The spatial constraints are combined with semantic structure components to generate a semantically guided geometric optimization objective function. The combination method is a weighted sum, and the semantic structure components serve as regularization terms to prevent the loss of semantic information during the optimization process.

[0131] The gradient vector of the semantically guided geometric optimization objective function with respect to 3D coordinates is calculated. The coordinate update step size is calculated based on the distance matrix, and the coordinate offset is obtained by multiplying the coordinate update step size by the gradient vector. The gradient calculation process must consider the influence of spatial constraints and semantic structure components on the 3D coordinates. For each sampling point, its gradient consists of two parts: the gradient from spatial constraints and the gradient from semantic structure. The spatial constraint gradient is mainly determined by the weight coefficients and the difference in neighborhood coordinates; during calculation, the derivative of all constraint terms related to that point must be calculated and accumulated. The semantic structure gradient reflects the constraint effect of semantic labels on the coordinates, ensuring semantic consistency during the optimization process. The dimension of the gradient vector is related to the number of sampling points and the coordinate dimension. For a scenario with 10,000 sampling points, the gradient vector dimension is 30,000, corresponding to the x, y, and z directions of each point. The coordinate update step size is calculated based on the distance matrix. An adaptive strategy can be adopted, with smaller step sizes for farther points and larger step sizes for closer points to prevent excessive point movement during optimization. The update step size can be set as an inverse proportional function of the distance; the larger the distance, the smaller the step size. The coordinate offset is obtained by multiplying the coordinate update step size by the gradient vector. The multiplication operation is performed separately for each coordinate component.

[0132] The coordinate offsets are superimposed onto the 3D coordinate values, and this process is repeated iteratively until the semantically guided geometric optimization objective function value is less than a preset convergence threshold, yielding the optimized 3D coordinate values. The superposition operation is a simple vector addition, adding the calculated offsets to the x, y, and z coordinate components of the sampling point. To prevent instability in the optimization process, an upper limit can be set on the offsets to ensure that the movement distance of the points in each iteration does not exceed the preset maximum value. After each iteration, the geometric optimization objective function value is recalculated and compared with the previous iteration result to calculate the relative change in the function value. The iteration process stops when the relative change is less than the preset convergence threshold or when the maximum number of iterations is reached. The convergence threshold can be set to 0.001, and the maximum number of iterations can be set to 100. After iteration, the obtained 3D coordinate values ​​are the optimized 3D coordinate values, which satisfy both geometric continuity requirements and maintain the integrity of the semantic structure.

[0133] In this embodiment, by decomposing semantic and spatial structural components from deterministic semantic labels, the topological relationships and boundary features of objects in the scene are effectively captured. The combination of the semantic topological matrix and the spatial geometric matrix forms a structural constraint system that takes into account both semantic association and geometric constraints. Spatial constraint terms based on the distance matrix and neighborhood coordinate differences ensure the smoothness and continuity of the reconstructed surface. The semantically guided geometric optimization objective function incorporates semantic knowledge into the coordinate optimization process, enabling the reconstruction results to better preserve the semantic boundaries between objects while maintaining geometric accuracy. The adaptive coordinate update strategy effectively prevents excessive smoothing during the optimization process, preserving the detailed features of the scene.

[0134] like Figure 2 The diagram shown illustrates the flowchart for the 3D scene navigation path planning in this embodiment.

[0135] In one optional implementation, the optimized 3D coordinates of continuous spatial sampling points with the same deterministic semantic labels are extracted to form object boundaries. A scene topology graph is constructed based on the inclusion and adjacency relationships between object boundaries. The scene functional structure is inferred based on the scene topology graph. A navigation path is generated based on the object boundaries and the scene functional structure, including:

[0136] Calculate the density distribution vector of continuous spatial sampling points with the same deterministic semantic labels, construct a multi-scale density field, extract the density gradient direction in the multi-scale density field and convert it into the boundary growth direction;

[0137] Adaptive region expansion is performed on the optimized 3D coordinate values ​​along the boundary growth direction to form the object boundary and extract the spatial contact points and spatial nesting points between the object boundaries.

[0138] Map spatial contact points and spatial nested points to topological nodes, and construct containment relationship connecting edges and adjacency relationship connecting edges; construct a scene topology graph based on the topological nodes, containment relationship connecting edges and adjacency relationship connecting edges, extract the topological node connection patterns from the scene topology graph, and infer the scene functional structure;

[0139] The object boundary is converted into a spatial cost matrix, the scene functional structure is converted into a motion constraint matrix, the motion constraint matrix is ​​superimposed on the spatial cost matrix, and path search is performed on the superimposed spatial cost matrix to generate a navigation path.

[0140] The density distribution vector of continuous spatial sampling points with the same deterministic semantic labels is calculated to construct a multi-scale density field. The density gradient direction is extracted from this field and converted into a boundary growth direction. The density distribution vector calculation is based on the distribution characteristics of spatial sampling points. For a given semantic label category, the number of sampling points with that label in each region of the space is counted to form a spatial density distribution. Density statistics employ the kernel density estimation method. A spherical region of a certain radius is set around the sampling point, and the ratio of the number of sampling points within the sphere to the volume of the sphere is calculated as the density value at the center point. The multi-scale density field is achieved by changing the radius of the statistical sphere; a smaller radius results in a richer density field detail, while a larger radius results in a smoother density field. In practical applications, three statistical radii can be set, which are 1, 2, and 4 times the average spacing between sampling points, respectively, to construct density fields at three scales: fine-grained, medium-grained, and coarse-grained. The density gradient direction is extracted from the multi-scale density field. The gradient calculation is based on the density difference between adjacent spatial locations, pointing in the direction of the fastest density increase. The density gradient direction is consistent with the normal vector direction of the object boundary. By rotating it by 90 degrees, the gradient direction is transformed into the boundary growth direction, that is, it extends along the tangent direction of the object surface. For the semantic category of a table in an indoor scene, when calculating the density distribution, it may be found that the sampling point density in the tabletop area is 0.05 points / cubic centimeter, while the density gradient in the tabletop edge area is larger and points inward. After rotating it by 90 degrees, the boundary growth direction extending along the tabletop edge is obtained.

[0141] Adaptive region expansion is performed on the optimized 3D coordinate values ​​along the boundary growth direction to form object boundaries and extract spatial contact points and spatial nesting points between object boundaries. Adaptive region expansion starts from a boundary seed point and gradually expands along the boundary growth direction. Seed points are selected from sampling points with large density gradients, which are usually located at object boundaries. During expansion, the next boundary point is searched along the boundary growth direction from the current boundary point. The search radius is adaptively adjusted according to local density; a smaller search radius is used for high-density regions, and a larger search radius is used for low-density regions. For each new boundary point found, it is determined whether it meets the boundary point conditions: consistent semantic label, density gradient greater than a threshold, and forming a continuous surface with an existing boundary. If the conditions are met, it is added to the boundary point set, and the boundary growth direction is updated. The expansion process continues until no new boundary point that meets the conditions can be found, or a closed boundary is formed. After forming the object boundaries, spatial contact points and spatial nesting points between object boundaries are extracted. Spatial contact points are pairs of points where the distance between different object boundaries in space is less than a preset threshold, representing the adjacency relationship between objects. Spatial nesting points are boundary points where one object boundary is completely contained within another object boundary, representing the containment relationship between objects. For chairs and floors in indoor scenes, when the distance between the chair legs and the floor is less than 5 centimeters, they are identified as spatial contact points; for bookshelves and books, when all the boundary points of the books are located inside the bookshelf, they are identified as spatial nesting points.

[0142] Spatial contact points and spatial nesting points are mapped to topological nodes, and containment and adjacency relationship edges are constructed. A scene topology graph is built based on these topological nodes, containment relationship edges, and adjacency relationship edges. Topological node connection patterns are extracted from the scene topology graph to infer the scene's functional structure. The topological node mapping process treats each object boundary as a node, with node attributes including the object's semantic label, spatial location, and size. For detected spatial contact points, adjacency relationship edges are established between the corresponding two object nodes, with the edge weight determined based on the contact area or the number of contact points. For detected spatial nesting points, containment relationship edges are established between the containing object node and the contained object node, with the edge direction pointing from the containing object to the contained object. The constructed topological nodes and connection edges form a scene topology graph, whose structure reflects the spatial relationships between objects in the scene. Extracting topological node connection patterns from the scene topology graph can be achieved using graph pattern matching algorithms or subgraph isomorphism detection methods. Typical connection patterns include support relationships, containment relationships, and functional combinations. Based on the identified connection patterns, the functional structure of the scene can be inferred. For example, a dining table and chair combination is identified as a dining area, a sofa and coffee table combination as a rest area, and a desk, chair, and bookshelf combination as a study area. For the identified "table-chair-floor" connection pattern, where the chair is in contact with the floor and the chair is adjacent to the table, it can be inferred that it is a work area.

[0143] The process involves converting object boundaries into a spatial cost matrix, converting scene functional structures into motion constraint matrices, and then superimposing the motion constraint matrices onto the spatial cost matrix. Path searching is then performed on the superimposed spatial cost matrix to generate a navigation path. During the conversion of object boundaries into the spatial cost matrix, the 3D space is discretized into a mesh structure, with the mesh resolution determined based on scene size and accuracy requirements. For each mesh cell, its distance to the object boundary is calculated; closer distances result in higher costs, forming a distance-based cost distribution.

[0144] The interior regions of objects are set as high-cost, impassable areas, while safety buffer zones are set around object boundaries, with costs decreasing with distance. When converting the scene's functional structure into a motion constraint matrix, the characteristics and usage requirements of different functional areas are considered. For functional areas such as work areas and rest areas, appropriate passage priorities and obstacle avoidance rules are set. Transition zones are set at the boundaries of functional areas to ensure smooth path transitions. The motion constraint matrix is ​​superimposed on the spatial cost matrix, using a weighted sum method where weights are dynamically adjusted based on the navigation task type. Path search is performed on the superimposed spatial cost matrix, using a fast expanding random tree algorithm. The path search process considers the principle of cost minimization while satisfying constraints such as path smoothness and safety. The searched path is smoothed to eliminate jagged inflection points, generating the final navigation path. For a navigation task from the bedroom to the kitchen, the generated path avoids furniture obstacles, maintains an appropriate distance from walls, and considers the accessibility characteristics of different functional areas.

[0145] In this embodiment, a systematic fusion of visual reconstruction and functional understanding enables deep semantic parsing and intelligent navigation planning of a scene. A boundary extraction method based on density distribution effectively captures the geometric contours and structural features of objects, improving the accuracy and robustness of boundary recognition. An adaptive region expansion strategy adapts to the characteristics of scenes with different density distributions, overcoming the boundary breakage problem caused by uneven sampling. Spatial relationships between objects are mapped to topological structures, constructing a semantic association network for the scene, transcending the scope of traditional geometric reconstruction. Functional structure inference based on the topological graph achieves a cognitive leap from object recognition to functional understanding, endowing the scene with deeper semantic connotations. The unified representation of object boundaries and functional structures provides a comprehensive decision-making basis for navigation path planning that considers both spatial constraints and functional semantics.

[0146] A second aspect of this invention provides a scene reconstruction and semantic understanding system based on a large visual model, the system comprising:

[0147] The first unit is used to acquire multi-view image data of the target scene, input the multi-view image data into a pre-trained large visual model, and output joint feature representation and attention weight matrix.

[0148] The second unit is used to generate the three-dimensional coordinates and semantic probability distribution of spatial sampling points in three-dimensional space based on the joint feature representation, and to construct a spatial semantic field.

[0149] The third unit is used to cluster spatial sampling points in the spatial semantic field into a set of spatial sampling points using an attention weight matrix, perform consistency enhancement on the semantic probability distribution within the set of spatial sampling points to obtain an enhanced semantic probability distribution, and convert the enhanced semantic probability distribution into a deterministic semantic label.

[0150] The fourth unit is used to construct a semantically guided geometric optimization objective function based on deterministic semantic labels and three-dimensional coordinate values. The three-dimensional coordinate values ​​of the spatial sampling points are adjusted by minimizing the geometric optimization objective function to obtain the optimized three-dimensional coordinate values.

[0151] The fifth unit is used to extract the optimized 3D coordinate values ​​of continuous spatial sampling points with the same deterministic semantic labels to form object boundaries, construct scene topology graphs based on the inclusion and adjacency relationships between object boundaries, infer scene functional structure based on scene topology graphs, and generate navigation paths based on object boundaries and scene functional structure.

[0152] A third aspect of the present invention provides an electronic device, comprising:

[0153] processor;

[0154] Memory used to store processor-executable instructions;

[0155] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0156] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0157] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for scene reconstruction and semantic understanding based on a large visual model, characterized in that, include: Acquire multi-view image data of the target scene, input the multi-view image data into a pre-trained large visual model, and output joint feature representation and attention weight matrix; Based on the joint feature representation, the three-dimensional coordinates and semantic probability distribution of spatial sampling points are generated in three-dimensional space to construct a spatial semantic field. The spatial sampling points in the spatial semantic field are clustered into a set of spatial sampling points using an attention weight matrix. The semantic probability distribution within the set of spatial sampling points is then enhanced to obtain an enhanced semantic probability distribution. The enhanced semantic probability distribution is then converted into a deterministic semantic label. A semantically guided geometric optimization objective function is constructed based on deterministic semantic labels and three-dimensional coordinate values. The three-dimensional coordinate values ​​of spatial sampling points are adjusted by minimizing the geometric optimization objective function to obtain the optimized three-dimensional coordinate values. The optimized 3D coordinates of continuous spatial sampling points with the same deterministic semantic labels are extracted to form object boundaries. A scene topology graph is constructed based on the inclusion and adjacency relationships between object boundaries. The scene functional structure is inferred based on the scene topology graph. A navigation path is generated based on the object boundaries and the scene functional structure.

2. The method according to claim 1, characterized in that, Acquire multi-view image data of the target scene, input the multi-view image data into a pre-trained large-scale visual model, and output a joint feature representation and attention weight matrix, including: The original multi-view image data of the target scene is acquired, the spatial pose calibration of the original multi-view image data is performed, the spatial transformation matrix between each view is calculated, and geometric correction is performed based on the spatial transformation matrix to generate multi-view image data with spatial pose information. Multi-view image data with spatial pose information is divided into multiple view groups. Image preprocessing is performed on the images within the multiple view groups. Spatial consistency correction is then performed on the preprocessed view groups based on the spatial pose information to generate standardized multi-view image data. Standardized multi-view image data is input into a pre-trained large visual model according to the viewpoint order determined by spatial pose information. Multi-level visual features are extracted, and the feature similarity between viewpoints is calculated based on the multi-level visual features and the spatial transformation matrix to generate an attention weight matrix. Multi-level visual features and attention weight matrices are aggregated to generate a joint feature representation.

3. The method according to claim 1, characterized in that, The training process of the large visual model includes: A feature extraction network is constructed, which includes a multi-scale feature decomposition unit and a spatial structure coding unit; The input image is input into the multi-scale feature decomposition unit to perform feature decomposition and obtain multi-level feature maps; the multi-level feature maps are input into the spatial structure encoding unit to extract scene geometric information and generate scene spatial feature representation and scene structural features; The scene spatial feature representation is input into the feature matching module to calculate the spatial transformation matrix between viewpoints. Based on the spatial transformation matrix, a geometric consistency loss is constructed. The geometric consistency loss and the multi-level feature mapping are input into the feature alignment module to generate feature alignment results and feature matching constraints. The scene structure features and feature matching constraints are input into the attention calculation module to generate feature attention weights. The feature attention weights and feature alignment results are input into the feature fusion module to perform feature aggregation operations and output joint feature representations. The joint feature representation and geometric consistency loss are input into the network optimization module to calculate the parameter gradient, update the parameters in the feature extraction network, and repeat the iteration until the geometric consistency loss value converges to obtain the trained large visual model.

4. The method according to claim 1, characterized in that, Based on the joint feature representation, the three-dimensional coordinates and semantic probability distribution of spatial sampling points are generated in three-dimensional space, and the spatial semantic field is constructed, including: The joint feature representation is input into the multi-scale feature decomposition unit and the spatial structure coding unit to generate multi-level feature maps and scene spatial feature representations. A three-dimensional spatial sampling grid is constructed based on the scene spatial feature representation, and initial spatial sampling points are generated within the three-dimensional spatial sampling grid. Calculate the local density distribution of the initial spatial sampling points, and adjust the position of the spatial sampling points according to the local density distribution to generate uniformly distributed spatial sampling points. The uniformly distributed spatial sampling points are projected onto a multi-level feature mapping space, and the feature vectors at the projection positions are extracted. The three-dimensional coordinate offset of the spatial sampling points is calculated based on the feature vectors. The three-dimensional coordinate offset is superimposed onto the coordinates of the uniformly distributed spatial sampling points to generate the three-dimensional coordinate values ​​of the spatial sampling points. The three-dimensional coordinates of the spatial sampling points and the joint feature representation are used to generate semantic features of the spatial sampling points through feature aggregation operations. Based on the semantic features and the scene space feature representation, a probability distribution function is constructed to calculate the semantic category probability distribution at the location of the spatial sampling point and generate a semantic probability distribution. The three-dimensional coordinates of the spatial sampling points and the semantic probability distribution are combined to construct a spatial semantic field.

5. The method according to claim 1, characterized in that, The spatial sampling points in the spatial semantic field are clustered into a spatial sampling point set using an attention weight matrix. Consistency enhancement is applied to the semantic probability distribution within the spatial sampling point set to obtain an enhanced semantic probability distribution. Converting the enhanced semantic probability distribution into deterministic semantic labels includes: Project the spatial sampling points in the spatial semantic field onto the feature space corresponding to the attention weight matrix, and calculate the structural representation vector of the spatial sampling points. The structural representation matrix of the spatial sampling points is calculated based on the structural representation vector. Spatial continuity constraints are introduced into the structural representation matrix, and matrix decomposition is performed to divide the spatial sampling points into multiple sets of spatial sampling points. Calculate the spatial distance distribution and semantic probability distribution within the set of spatial sampling points, construct probability propagation weights based on the spatial distance distribution, and combine the probability propagation weights with the semantic probability distribution to generate a probability update matrix; The semantic probability distribution is updated in multiple rounds using a probability update matrix. Spatial distance constraints are applied in each round of iteration, and the enhanced semantic probability distribution is output. Spatial structural features are extracted from the structural representation matrix, and semantic distribution features are extracted from the enhanced semantic probability distribution. The spatial structural features and semantic distribution features are combined to construct a feature transformation matrix. The enhanced semantic probability distribution is mapped to deterministic semantic labels using the feature transformation matrix.

6. The method according to claim 1, characterized in that, A semantically guided geometric optimization objective function is constructed based on deterministic semantic labels and 3D coordinate values. The 3D coordinate values ​​of spatial sampling points are adjusted by minimizing the geometric optimization objective function, resulting in optimized 3D coordinate values ​​including: Semantic structural components and spatial structural components are decomposed from deterministic semantic labels. A semantic topology matrix is ​​calculated based on the semantic structural components, and a spatial geometry matrix is ​​calculated based on the spatial structural components. The semantic topology matrix and the spatial geometry matrix are combined to generate a structural constraint matrix. Calculate the distance matrix between three-dimensional coordinate values, and obtain the neighborhood coordinate difference based on the distance matrix; combine the neighborhood coordinate difference with the structural constraint matrix to obtain weight coefficients, use the weight coefficients to construct spatial constraint terms, and combine the spatial constraint terms with semantic structural components to generate a semantically guided geometric optimization objective function; Calculate the gradient vector of the semantically guided geometric optimization objective function with respect to the 3D coordinate values, calculate the coordinate update step size based on the distance matrix, and multiply the coordinate update step size by the gradient vector to obtain the coordinate offset. The coordinate offset is superimposed on the three-dimensional coordinate value, and the iteration is repeated until the semantically guided geometric optimization objective function value is less than the preset convergence threshold, thus obtaining the optimized three-dimensional coordinate value.

7. The method according to claim 1, characterized in that, Optimized 3D coordinates of continuous spatial sampling points with the same deterministic semantic labels are extracted to form object boundaries. A scene topology graph is constructed based on the inclusion and adjacency relationships between object boundaries. The scene functional structure is inferred from the scene topology graph. Navigation paths are generated based on the object boundaries and the scene functional structure, including: Calculate the density distribution vector of continuous spatial sampling points with the same deterministic semantic labels, construct a multi-scale density field, extract the density gradient direction in the multi-scale density field and convert it into the boundary growth direction; Adaptive region expansion is performed on the optimized 3D coordinate values ​​along the boundary growth direction to form the object boundary and extract the spatial contact points and spatial nesting points between the object boundaries. Spatial contact points and spatial nested points are mapped to topological nodes, and containment relationship connecting edges and adjacency relationship connecting edges are constructed. Based on the topological nodes, containment relationship connecting edges and adjacency relationship connecting edges, a scene topology graph is constructed, and the topological node connection patterns are extracted from the scene topology graph to infer the scene functional structure. The object boundary is converted into a spatial cost matrix, the scene functional structure is converted into a motion constraint matrix, the motion constraint matrix is ​​superimposed on the spatial cost matrix, and path search is performed on the superimposed spatial cost matrix to generate a navigation path.

8. A scene reconstruction and semantic understanding system based on a large visual model, used to implement the method of any one of claims 1-7, characterized in that, include: The first unit is used to acquire multi-view image data of the target scene, input the multi-view image data into a pre-trained large visual model, and output joint feature representation and attention weight matrix. The second unit is used to generate the three-dimensional coordinates and semantic probability distribution of spatial sampling points in three-dimensional space based on the joint feature representation, and to construct a spatial semantic field. The third unit is used to cluster spatial sampling points in the spatial semantic field into a set of spatial sampling points using an attention weight matrix, perform consistency enhancement on the semantic probability distribution within the set of spatial sampling points to obtain an enhanced semantic probability distribution, and convert the enhanced semantic probability distribution into a deterministic semantic label. The fourth unit is used to construct a semantically guided geometric optimization objective function based on deterministic semantic labels and three-dimensional coordinate values. The three-dimensional coordinate values ​​of the spatial sampling points are adjusted by minimizing the geometric optimization objective function to obtain the optimized three-dimensional coordinate values. The fifth unit is used to extract the optimized 3D coordinate values ​​of continuous spatial sampling points with the same deterministic semantic labels to form object boundaries, construct scene topology graphs based on the inclusion and adjacency relationships between object boundaries, infer scene functional structure based on scene topology graphs, and generate navigation paths based on object boundaries and scene functional structure.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images