Real scene three-dimensional data scene understanding and cognition method, device, medium and product

Abstract

The application discloses a real scene three-dimensional data scene understanding and cognition method, equipment, medium and product, relates to the real scene three-dimensional data processing and three-dimensional point cloud semantic segmentation technical field, and the method comprises the following steps: acquiring original real scene three-dimensional point cloud data under several frames of different scenes and the semantic category label corresponding to each frame of original real scene three-dimensional point cloud data; the original real scene three-dimensional point cloud data is preprocessed, and the subsampling point cloud and KDTree index are obtained, and the local input block is formed; the local block point coordinates in the local input block are input into the point cloud semantic segmentation network, and the point-by-point semantic segmentation result is obtained; the point cloud semantic segmentation network is trained and optimized, and the trained scene understanding model is obtained; the target scene point cloud is input into the trained scene understanding model; the scene cognition language model is used to generate the scene cognition result. The application simultaneously considers sampling efficiency and geometric details, and improves the segmentation precision.

Description

Technical Field

[0001] This application relates to the fields of real-scene 3D data processing and 3D point cloud semantic segmentation technology, and in particular to a method, device, medium and product for understanding and recognizing real-scene 3D data scenes. Background Technology

[0002] With the development of LiDAR, oblique photogrammetry, motion mapping, and high-precision map acquisition technologies, the scale and update frequency of real-scene 3D data continue to increase. This type of data typically manifests as disordered, unstructured, sparse, non-uniform point clouds or grid data with large scale spans. How to achieve refined semantic understanding of massive amounts of 3D points while maintaining high efficiency, and further form scene cognition results that can be directly utilized by humans or higher-level applications, is a crucial issue in the current field of intelligent real-scene 3D processing.

[0003] Traditional 3D point cloud semantic segmentation methods can be broadly categorized into voxelization methods, multi-view projection methods, graph convolution methods, and direct point processing methods. Voxelization methods typically require discretizing the point cloud into a regular voxel grid. While this facilitates convolution computation, it introduces quantization errors, loss of detail, and high memory overhead. Multi-view projection methods rely on 2D projection, which can easily lead to the loss of 3D geometric relationships. Although graph convolution methods and complex sampling methods can express local topological information, they usually require expensive graph construction, far-point sampling, or density estimation processes, making them difficult to directly apply to real-time or near-real-time processing of millions of point clouds.

[0004] Existing lightweight semantic segmentation methods for large-scale point clouds typically reduce the size of the point cloud through downsampling and combine local neighborhood feature aggregation mechanisms to achieve efficient point-by-point semantic reasoning. These methods have certain advantages in terms of computational efficiency and engineering applicability, but they still have the following shortcomings in real-world 3D scene understanding and cognitive tasks: it is difficult to balance sampling efficiency and geometric detail preservation, resulting in poor segmentation accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, medium, and product for understanding and recognizing real-world 3D data scenes, which can simultaneously take into account sampling efficiency and geometric details, thereby improving segmentation accuracy.

[0006] To achieve the above objectives, this application provides the following solution.

[0007] Firstly, this application provides a method for understanding and recognizing real-world 3D data scenes, including: Obtain the dataset; the dataset includes several frames of original real-world 3D point cloud data in different scenes and the semantic category label corresponding to each frame of original real-world 3D point cloud data. Preprocess the original real-scene 3D point cloud data to obtain subsampled point clouds and KDTree indexes; Based on the subsampled point cloud and KDTree index, the center point and nearest neighbor set in the original real scene 3D point cloud data are determined to form a local input block; The coordinates of local block points in the local input block are input into a point cloud semantic segmentation network for point-by-point semantic category prediction, resulting in point-by-point semantic segmentation of the local input block. The point cloud semantic segmentation network includes an encoder, a decoder, and a classification head. The encoder is used to: perform multi-layer neighborhood mapping and geometric perception sampling on the local input block to generate multi-layer neighborhood structure information; perform extended local spatial encoding on the center point and its neighboring points in the multi-layer neighborhood structure information, and aggregate it with the point features obtained by sharing the MLP mapping to obtain the final encoded features; the decoder is used to decode the final encoded features to obtain decoded features; and the classification head is used to predict the point-by-point semantic category based on the decoded features to obtain the point-by-point semantic segmentation result. The point cloud semantic segmentation network is trained to obtain a trained scene understanding model; The target scene point cloud is input into the trained scene understanding model to obtain the point-by-point semantic segmentation result of the target scene point cloud; Using a scene cognition language model, scene cognition results are generated based on the point-by-point semantic segmentation results of the target scene point cloud.

[0008] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for understanding and recognizing real-world 3D data scenes.

[0009] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method for understanding and recognizing real-world 3D data scenes.

[0010] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for understanding and recognizing real-world 3D data scenes.

[0011] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method, device, medium, and product for understanding and recognizing real-world 3D data scenes. It employs a point cloud semantic segmentation network for scene understanding. The encoder of the point cloud semantic segmentation network is used to: perform multi-layer neighborhood mapping and geometric perception sampling on local input blocks to generate multi-layer neighborhood structure information, and perform extended local spatial encoding on the multi-layer neighborhood structure information. By using multi-layer neighborhood mapping and geometric perception sampling, the problem of easily losing geometric key points in pure random sampling is solved, while taking into account both sampling efficiency and geometric details. Point-by-point semantic segmentation through a trained scene understanding model can improve segmentation accuracy. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is an application environment diagram of a real-scene 3D data scene understanding and cognition method according to an embodiment of this application; Figure 2 A flowchart illustrating a method for understanding and recognizing real-world 3D data scenes, provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the process of semantic segmentation of real-scene 3D data scene according to an embodiment of this application; Figure 4 A schematic diagram of an encoder and decoder provided in one embodiment of this application; Figure 5 This is a schematic diagram of the local aggregation feature aggregation and expansion residual process provided in an embodiment of this application; Figure 6 This is a schematic diagram of a scene recognition language description generation process provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] The real-scene 3D data scene understanding and cognition method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the dataset and target scene point cloud to server 104. After receiving the dataset and target scene point cloud, server 104 uses the dataset to train and optimize the point cloud semantic segmentation network to obtain a trained scene understanding model. The target scene point cloud is then input into the trained scene understanding model to obtain the point-by-point semantic segmentation result of the target scene point cloud. The scene recognition language model then generates a scene recognition result based on the point-by-point semantic segmentation result of the target scene point cloud. Server 104 can then feed back the trained scene understanding model and the scene recognition result to terminal 102. In addition, in some embodiments, the real-scene 3D data scene understanding and cognition method can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly perform model training on the dataset or perform scene understanding and cognition on the target scene point cloud. Alternatively, the server 104 can obtain the dataset and the target scene point cloud from the data storage system and perform model training on the dataset or perform scene understanding and cognition on the target scene point cloud.

[0016] The terminal 102 can be, but is not limited to, various desktop computers and laptops. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers, or it can be a cloud server.

[0017] In one exemplary embodiment, such as Figure 2 As shown, a method for understanding and recognizing real-world 3D data scenes is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 207.

[0018] Step 201: Obtain the dataset; the dataset includes several frames of original real-world 3D point cloud data in different scenes and the semantic category label corresponding to each frame of original real-world 3D point cloud data.

[0019] Step 202: Preprocess the original real-scene 3D point cloud data to obtain subsampled point clouds and KDTree indexes.

[0020] Step 203: Based on the subsampled point cloud and KDTree index, determine the center point and nearest neighbor set in the original real-world 3D point cloud data to form a local input block.

[0021] Step 204: Input the coordinates of local block points in the local input block into the point cloud semantic segmentation network for point-by-point semantic category prediction to obtain the point-by-point semantic segmentation result of the local input block. The point cloud semantic segmentation network includes an encoder, a decoder, and a classification head. The encoder is used to: perform multi-layer neighborhood mapping and geometric perception sampling on the local input block to generate multi-layer neighborhood structure information; perform extended local spatial encoding on the center point and its neighboring points in the multi-layer neighborhood structure information, and aggregate it with the point features obtained by sharing the MLP mapping to obtain the final encoded features; the decoder is used to decode the final encoded features to obtain decoded features; and the classification head is used to perform point-by-point semantic category prediction based on the decoded features to obtain the point-by-point semantic segmentation result.

[0022] Step 205: Train the point cloud semantic segmentation network to obtain a trained scene understanding model.

[0023] Step 206: Input the target scene point cloud into the trained scene understanding model to obtain the point-by-point semantic segmentation result of the target scene point cloud.

[0024] Step 207: Using the scene cognition language model, generate scene cognition results based on the point-by-point semantic segmentation results of the target scene point cloud.

[0025] By implementing steps 201 to 207 above, multi-layer neighborhood mapping and geometrically perceptual sampling solve the problem of easily losing geometric key points in pure random sampling, while balancing sampling efficiency and geometric details. Point-by-point semantic segmentation using a trained scene understanding model improves segmentation accuracy. This real-scene 3D data scene understanding and cognition method is applicable to scenarios such as LiDAR, mobile surveying, digital twin cities, natural resource surveys, remote sensing, and autonomous driving. Based on a large 3D geographic model (scene cognition language model), it achieves point-by-point semantic understanding, scene structure cognition, and natural language-level scene description output for large-scale 3D point cloud data.

[0026] First, a large-scale point cloud (raw real-world 3D point cloud data) is acquired as input. This large-scale point cloud is then preprocessed, specifically through voxel subsampling and index construction, resulting in subsampled point clouds and KDTree indices. Local input blocks are then constructed. These local input blocks are fed into a point cloud semantic segmentation network (which includes an encoder, decoder, and classifier). The network is then trained using a joint loss function to obtain a trained scene understanding model. The real-world 3D data scene semantic segmentation process is as follows: Figure 3 As shown.

[0027] Obtain raw real-world 3D point cloud data from different scenarios and divide the dataset into training, validation, and test sets. The specific process of step 202 is as follows: read the 3D coordinate information of each frame of raw real-world 3D point cloud data; remap the semantic category labels of the data with semantic category labels according to the learning mapping table; then, subsample the sub-point clouds using voxel meshes and construct a KDTree index for each frame of subsampled point clouds; for the validation and test sets, further establish projection indexes from the original points to the subsampled points.

[0028] In a specific example, the dataset could be the SemanticKITTI dataset, and the object of processing would be a sequence of point cloud data of a road scene using a vehicle-mounted LiDAR.

[0029] In this embodiment, the voxel mesh side length is set to 0.06m. This parameter is chosen because: if the mesh size is too small, the subsequent computational scale cannot be effectively reduced; if the mesh size is too large, geometric details of road boundaries, rod-shaped objects, and small-scale targets will be lost. A subsampling scale of 0.06m achieves a good balance between computational efficiency and geometric fidelity.

[0030] Through the above step 202, the subsampled point cloud, the corresponding KDTree index, and the projection index from the original point to the subsampled point can be obtained simultaneously, thus providing a unified data organization foundation for subsequent training sample construction, inference reprojection, and full-resolution result restoration.

[0031] Based on the subsampled point cloud of the training set and its KDTree index obtained in step 202, neighborhood retrieval and center point-driven local pruning are performed to construct a local input block for training the scene understanding model. That is, the center point is selected from each frame of original real scene 3D point cloud data, and the corresponding KDTree is called to query the nearest neighbor set to form a local input block with a fixed number of points.

[0032] The number of input points per instance is denoted as N. In this embodiment, N = 4096 × 11 = 45056. Using a local block-based approach instead of direct whole-frame input allows for the segmentation of large-scale scenes into local training samples adapted to GPU memory, while also preserving the spatial continuity of the original 3D coordinates. The output of step 203 will serve as the direct input for multi-layer neighborhood mapping and network forward propagation.

[0033] Specifically, the encoder is used to: perform multi-layer neighborhood mapping and geometrically perceptual sampling on the local input block to generate multi-layer neighborhood structure information; the multi-layer neighborhood structure information includes the coordinate sets of points in each layer, neighborhood index, downsampling index, and upsampling interpolation index; based on the coordinate sets of points in each layer and the neighborhood index, perform extended local spatial coding on the center point and its neighboring points to obtain local geometric coding; perform shared MLP mapping on the point coordinates in the local input block to obtain point features; based on the point features, local geometric coding, and neighborhood index, perform geometric conditional attention pooling and dilated residual feature aggregation to obtain the final encoded features.

[0034] The encoder comprises several dilated residual blocks. After each layer of dilated residual blocks completes local aggregation feature aggregation, it downsamples the point set of the current layer. A geometrically perceptual sampling mechanism is introduced in the first few layers of dilated residual blocks to downsample the point set in the local input block with a set sampling probability, resulting in several layers of point sets and neighborhood relationships. The downsampling method in the later layers of dilated residual blocks is random sampling. For example... Figure 4 As shown, the encoder includes four layers of dilated residual blocks ( Figure 4 The expansion residual blocks are divided into expansion residual blocks 1, expansion residual block 2, expansion residual block 3 and expansion residual block 4. The first two expansion residual blocks (expansion residual block 1 and expansion residual block 2) introduce a geometrically sensed sampling mechanism, while the latter two expansion residual blocks (expansion residual block 3 and expansion residual block 4) are downsampled by random sampling.

[0035] In a specific example, the encoder has 4 network layers, each layer has 16 K-nearest neighbors, the number of input points in a single input is 45056, the downsampling ratio is [4,4,4,4], the number of encoder output channels is [16,64,128,256], and the classification head outputs 19 semantic probability distributions.

[0036] like Figure 4 As shown, in this embodiment, the encoder includes four layers of dilated residual blocks. The first two layers, after completing local feature aggregation, introduce a geometrically perceptual sampling unit to downsample the current layer's point set. The latter two layers, after completing local feature aggregation, use a random sampling unit to downsample the current layer's point set. Each layer of dilated residual blocks includes a shared MLP, a local spatial coding unit, a geometrically conditional attention pooling unit, and an aggregation unit. The shared MLP is used to perform point-by-point nonlinear mapping on the input point coordinates or input point features, mapping the original low-dimensional features to a high-dimensional feature space suitable for subsequent neighborhood feature aggregation. "Shared" means that each point in the local input block uses a multilayer perceptron with the same parameters for feature transformation, thereby achieving feature channel enhancement, initial geometric / semantic feature extraction, and feature dimension adaptation for subsequent attention pooling without changing the disorder of the point cloud or the spatial relationship between points.

[0037] Local aggregation feature aggregation and expansion residual process as follows Figure 5 As shown.

[0038] For the local input block obtained in step 203, neighborhood relationships and sampling indices are constructed layer by layer according to the network hierarchy. For the first... For each layer of points, a shared MLP is first used to perform feature transformation on each point. Then, a K-nearest neighbor search is used to obtain the neighborhood index. Finally, the point set is compressed according to a set downsampling ratio. To overcome the problem that pure random sampling is prone to losing geometric key points, a geometry-aware sampling mechanism is introduced in the first few layers.

[0039] The sampling probability is calculated as follows: (1); (2); in, For point Set the sampling probability; This is the geometric weighting coefficient, which can be taken as 0.4 in this embodiment; This represents the number of points in the local input block; Normalized points The geometric score; For point The geometric score is usually higher when a point is located in a boundary region, a region of curvature change, or a sparse small target region. Normalized points The geometric score; It is a very small positive number, used to avoid the denominator being 0 and to enhance numerical stability; It represents the variance of the distance set and measures the degree of fluctuation in the local distance distribution. A larger variance usually indicates that the local geometry is more uneven (such as boundaries, abrupt curvature changes, and areas of density variation). Point With The set of distances between neighboring points; the first term in the geometric score formula. The second term reflects the fluctuations in local distance distribution. Reflects the degree of directional bias in the neighborhood; This represents the number of nearest neighbors (the number of neighbors in KNN). Point The The coordinates of the neighboring points are obtained using KNN; Indicates the first The three-dimensional coordinates of the points , It is the set of real numbers; This represents the L2 norm.

[0040] Subsequently, based on the set sampling probability, a subset of points is extracted without replacement from the current layer point set, and corresponding pooling indices and upsampling interpolation indices are generated simultaneously. Through this step, the set of point coordinates, neighborhood index set, downsampling index set, and upsampling index set for each layer can be obtained.

[0041] The local spatial coding unit performs extended local spatial coding on the point features (center point and its neighboring points) in the local input block to obtain a local geometric code that integrates distance, relative position, orientation, and height information. That is, for each generated point set and neighborhood relationship, a local geometric code is constructed. For the center point... and its first neighborhood points Constructing local geometric coding (i.e. Figure 5 The 14-dimensional extended local space encoding in [the code] is as follows: (3); in, Local geometric coding; Center point Its first Euclidean distance between neighboring points; Center point Its first The relative coordinates of the neighboring points; The x, y, and z coordinates of the center point Center point The The x, y, and z coordinates of the neighboring points; Center point Its first The direction cosine similarity of the neighboring points; Features of height difference; For distance ratio features; The 14-dimensional local geometric encoding incorporates local average distance features. This encoding adds information on orientation consistency, height variation, and scale awareness, enhancing the model's ability to represent road elevation differences, ground undulations, poles, boundary regions, and sparse small targets.

[0042] A geometric conditional attention pooling unit is constructed based on point features, local geometric encoding, and neighborhood indexing. The geometric conditional attention pooling unit computes the attention response from both the semantic feature branch and the positional encoding branch. (4); (5); (6); (7); in, and These represent the attention responses generated by the semantic feature branch and the positional encoding branch, respectively. and These represent the learnable mapping parameters for the semantic feature branch and the positional encoding branch, respectively; Indicates features of neighboring points With local geometric coding Enhanced neighborhood features formed by splicing; For the first The first center point corresponds to the Features of each neighboring point; This represents the normalized neighborhood attention weights; for Activation function; Indicates the first Local aggregation features obtained by aggregating multiple center points; For the first The number of neighborhood points of a center point; This indicates element-wise weighting; This indicates feature splicing.

[0043] Two levels of local spatial encoding units are connected in series with two levels of geometrically conditional attention pooling units, and dilated residual blocks are formed through residual connections. Since the second-level aggregation receives the features after the first-level aggregation, its equivalent receptive field is further expanded, enabling the acquisition of contextual information over a larger neighborhood with lower computational cost.

[0044] The encoder structure is formed by stacking dilated residual blocks to extract and downsample features from the input local point cloud layer by layer. At the same time, nearest neighbor interpolation is used in the decoding stage, and the upsampled decoded features are fused with the shallow features of the corresponding resolution in the encoding stage through cross-layer skip connections to gradually restore the point-level resolution. Finally, the classification head outputs the point-by-point semantic segmentation results, including the semantic category probability distribution of each point.

[0045] Specifically, the initial features of the input points are three-dimensional coordinates, which are mapped to a low-dimensional feature space through shared convolution and then fed into the first coding unit of the encoder (i.e., the first dilated residual block). After the local feature aggregation is completed in each layer of the encoder's dilated residual block, the point set is compressed to the next scale using the downsampling index. Finally, the encoded features (i.e., the deepest features) are upsampled layer by layer by the decoder and fused with the shallow features by skip connections to obtain the point-by-point semantic segmentation result of the local input block.

[0046] The point cloud semantic segmentation network is trained to obtain a trained scene understanding model. Specifically, this involves: training the point cloud semantic segmentation network using a joint loss function to obtain a trained scene understanding model; and constructing a joint loss function based on the point-by-point semantic segmentation results and a pre-defined physical prior to train and optimize the point cloud semantic segmentation network, resulting in a trained scene understanding model.

[0047] The joint loss function is expressed as follows: (8); in, For the joint loss function; Category-weighted cross-entropy loss; , , , These are gravity constraint losses. Spatial consistency loss Geometric regularity loss High prior loss The weighting coefficients are as follows: Gravity constraint loss is used to limit unreasonable predictions that ground class points are located above non-ground class points; Spatial consistency loss suppresses local semantic jitter by using the differences in the distribution of neighboring point classes; Geometric regularity loss estimates the planarity of structural class targets by using the eigenvalues ​​of the local neighborhood covariance matrix; Height prior loss is used to constrain ground class points to be located at local lower positions.

[0048] The category-weighted cross-entropy loss is represented as follows: (9); (10); in, For the first The center point belongs to the first The class probability; the probability of all points belonging to each class constitutes the point-by-point semantic segmentation result, that is, the point-by-point semantic segmentation result includes the semantic class probability distribution of each point; The number of points; The total number of categories; For the first Class weights; For one-hot tags, when the first... The point belongs to the th point The value is 1 if the condition is met, otherwise it is 0. For the first The point belongs to the th point The network outputs the score for this class; For the first The point belongs to the th point The network output score for this class.

[0049] Let the first The first point in the neighborhood of the nth point The elevation of each neighboring point is The elevation of the center point is Define a soft indicator that the neighboring points are located below the center point: (11); in, For neighborhood points Located at the center point The soft indicator below; For the Sigmoid function, This is the scaling factor.

[0050] The gravity constraint loss is expressed as follows: (12); (13); in, Indicates the center point Its neighboring points The set of adjacent pairs formed; This indicates the number of adjacency pairs in the set of adjacency pairs; Indicates the first The predicted probability that each center point belongs to a ground-type target; It is a collection of ground-type targets, including various ground-type targets; Indicates the center point neighborhood points Belongs to the ground class collection The predicted probability.

[0051] Neighboring point pairs First, define the distance weights: (14); in, Center point Its neighboring points Distance weights between them; The base is the natural number; and The center point Its neighboring points The coordinates; Available points Adaptive estimation of the mean squared distance of the neighborhood; To prevent extremely small constants with a denominator of 0.

[0052] Let the set of probabilities of the center point categories be denoted as . , and The center point Belongs to category 1 and category 2 Class probability, set of neighborhood point class probabilities , and Neighboring points Belongs to category 1 and category 2 The class probability; and The Jensen-Shannon divergence is then: (15); in, Jensen-Shannon divergence; express and Kullback-Leibler divergence between them; express and The Kullback-Leibler divergence between them.

[0053] Spatial consistency loss is expressed as follows: (16).

[0054] For the center point local neighborhood First, calculate the neighborhood centroid: (17); in, The coordinates of the neighborhood centroid; Center point neighborhood points The coordinates; Center point The number of neighboring points in the local neighborhood.

[0055] Next, calculate the covariance matrix: (18); in, Here is the covariance matrix; superscript This indicates transpose.

[0056] Suppose that its eigenvalues ​​satisfy Then local planarity can be defined as: (19); in, It is a local planarity; , , For the covariance matrix The three eigenvalues ​​obtained after eigenvalue decomposition are sorted in descending order and satisfy the following conditions: .in, This represents the variance of the local neighborhood point cloud along the first principal direction. This represents the variance of the local neighborhood point cloud along the second principal direction. This represents the variance of the local neighborhood point cloud along the third principal direction.

[0057] The geometric regularity loss is expressed as follows: (20); (twenty one); in, Indicates the center point The predicted probability of belonging to the structural class of targets; It is a collection of structure classes, including various structure class targets.

[0058] Set the center point The minimum and maximum elevations within the neighborhood are as follows: (twenty two); in, and The center point Minimum and maximum elevations within the neighborhood; Center point The elevation of the neighboring points.

[0059] Then the center point The normalized relative height is defined as: (twenty three); in, Center point The normalized relative height.

[0060] The high prior loss is represented as follows: (twenty four).

[0061] To avoid excessive physical priors interfering with backbone network convergence during early training, and to achieve better physical consistency and spatial continuity while maintaining semantic discriminative ability, a warm-up and linear ramp-up mechanism is employed, namely, weight coefficients... , , , The expression for the dynamic adjustment coefficient of the physical constraint loss, which varies with the training rounds, is as follows: (25); in, The dynamic adjustment coefficient representing the physical constraint loss; Indicates the current training round; Indicates the preheating cycle; This indicates a linear growth cycle. When no physical constraints are introduced, that is, the joint loss function only includes the joint loss function; in At that time, the physical constraint weights increase linearly with the number of training rounds; in At this stage, physical constraints participate in training with full weights. This mechanism is used to avoid excessive physical priors interfering with the convergence of the backbone network in the early stages of training.

[0062] The target scene point cloud is input into a trained scene understanding model to obtain the point-by-point semantic segmentation result of the target scene point cloud. Specifically, the target scene point cloud is input into a trained scene understanding model, and the point-by-point semantic segmentation result of the target scene point cloud is obtained by using rolling overlay inference and probabilistic voting fusion.

[0063] After preprocessing the target scene point cloud according to step 202, several target local blocks are obtained. The coordinates of the local block points in the target local blocks are input into the trained scene understanding model. During the inference stage, a rolling coverage local block trimming strategy is adopted, and multiple prediction results are fused by probability voting to obtain the point-by-point semantic segmentation result of the target scene. The point-by-point semantic segmentation result is the full-resolution semantic segmentation result. The point-by-point semantic segmentation result includes the semantic category of each point in the target scene point cloud.

[0064] Specifically, a visit probability value is maintained for each point. Local culling is performed starting from the point with the lowest current probability. After one forward inference, the visit probability value of each point in the selected target local block is updated based on the distance from each point in the selected target local block to the center point. This process is repeated until all points in the target scene point cloud of the entire frame have been fully visited.

[0065] (26); in, This represents the updated semantic category probability distribution obtained after this probability fusion for the current point; This represents the historical semantic category probability distribution accumulated for the current point before this fusion. After the scene understanding model has been trained and forward reasoned for the current local input block, the probability distribution of the current semantic category of the output point is predicted. The smoothing coefficient is set to 0.98 in this embodiment. Exponential smoothing fusion is used because different local block center positions can cause prediction fluctuations; repeating the process multiple times and accumulating probabilities significantly improves the stability of the results. When the user needs the original resolution result, the sub-sampling point probabilities are mapped back to the original point cloud based on the established projection index, thus obtaining the point-by-point semantic segmentation result at the original resolution.

[0066] The pointwise semantic segmentation results obtained in step 206 are input into the scene cognition language description module. Statistical analysis, scene type inference, and natural language generation are performed on each frame or several frames at fixed sampling intervals. The scene cognition results are output, including scene description text and structured statistical results.

[0067] The process of generating scene cognitive language description is as follows: Figure 6 As shown. Specifically, the target scene point-by-point semantic labels (the point-by-point semantic segmentation results obtained in step 206) after the fusion of rolling coverage reasoning and probabilistic voting are input. First, the number and proportion of each semantic category in the point cloud of the target scene in the current frame are counted, and a label statistical vector is formed. Then, each semantic category is grouped and statistically analyzed according to five major categories: dynamic targets, road facilities, built environment, natural environment, and traffic facilities. Based on the proportion of categories such as vehicles, pedestrians, buildings, and vegetation and the density of dynamic targets, it is inferred whether the current scene belongs to scene types such as urban road scene, natural scene, or mixed scene, and the traffic complexity level is determined at the same time.

[0068] With point cloud coordinates provided, dynamic target distance, average distance, and scene spatial range attributes can be further calculated to enhance the spatial interpretability of the scene cognition results. Subsequently, the aforementioned statistical information, scene attributes, and optional spatial information are organized into prompt words based on a structured prompt template and input into the scene cognition language model (scene cognition large language model) to generate 2 to 3 natural language scene descriptions. This scene cognition language model can adopt a rule-based template mode, an external large language model interface mode, or a locally deployed model mode.

[0069] For example, when roads, buildings, and vehicles dominate a point cloud frame, and a small number of pedestrians are detected, the language model can output the following scene recognition result: "

Frame01

[0070] Finally, the scene description text (natural language scene description) and structured statistical results are saved as text files and JSON files, respectively. Through step 207, this application realizes the transformation from point-by-point semantic segmentation results to scene-level cognitive representation, enabling real-world 3D data to not only output classification labels, but also cognitive descriptions that can be directly invoked by human-computer interaction and upper-level tasks.

[0071] To verify the effectiveness of this application, a comparative experiment was conducted between the method of this application and existing methods under the same hardware and software environment and dataset (SemanticKITTI). The results are shown in Table 1.

[0072] Table 1 Comparison results between the method of this application and existing methods

[0073] As shown in Table 1, under the same hardware and software environment and the SemanticKITTI dataset, the method in this application achieves an mIoU of 53.3%, significantly outperforming the comparison methods PointNet, TangentConv, and DarkNet. PointNet's mIoU is 13.7%, indicating that it primarily relies on point-level independent feature learning and lacks sufficient modeling ability for complex spatial neighborhood structures, limiting its segmentation performance in tasks with complex geometry and uneven class distribution, such as road scenes. TangentConv improves its mIoU to 41.2%, demonstrating its enhanced ability to characterize neighborhood geometric information through local tangent plane convolution, but it still falls short in handling multi-scale contextual dependencies and long-distance semantic associations in large-scale scenes. DarkNet achieves an mIoU of 47.8%, demonstrating stronger feature extraction capabilities compared to the other two, but it primarily focuses on two-dimensional convolutional feature representation, and its adaptability to irregular distributions and spatial topological relationships in three-dimensional point clouds remains limited.

[0074] In comparison, the method presented in this application achieved the highest mIoU of 53.3% on the SemanticKITTI road scene, an improvement of 39.6 percentage points over PointNet, 12.1 percentage points over TangentConv, and 5.5 percentage points over DarkNet. This indicates that the method presented in this application can more effectively mine local geometric structure information and global semantic association information in point cloud data, improving the ability to identify and segment typical scene targets such as roads, vehicles, and buildings. The reasons for this are as follows: Firstly, this application constructs a point cloud semantic segmentation network for real-world 3D scenes, which can robustly model the characteristics of sparse, irregular, and highly variable point cloud density; secondly, by enhancing the expression of local neighborhood features and the aggregation of multi-level semantic information, the model's ability to distinguish boundary regions, occluded regions, and small target categories in complex road environments is improved.

[0075] The experimental results fully demonstrate that the method proposed in this application has good effectiveness and advancement in the semantic segmentation task of real-scene 3D point cloud, and can obtain more stable and accurate point-by-point segmentation output, providing a reliable data foundation for subsequent scene understanding, relationship cognition and semantic description.

[0076] To further verify the contribution of each component module to the overall performance improvement, ablation experiments were conducted under the same hardware and software environment and SemanticKITTI dataset, using the complete model of this application as a baseline, and key modules were removed one by one. The key modules include: the geometrically aware sampling unit, the local spatial encoding unit, the geometrically conditional attention pooling unit, the physically constrained joint training module, and the rolling over inference module. The experimental results are shown in Table 2.

[0077] Table 2 Experimental Results

[0078] The complete model in this application refers to the real-world 3D data scene understanding and cognition method provided in this application.

[0079] As shown in Table 2, removing the key modules from this application resulted in varying degrees of performance degradation on the SemanticKITTI dataset, indicating that each module positively contributes to the final performance improvement and exhibits good synergistic enhancement among multiple modules. Removing the geometry-aware sampling unit reduced mIoU from 53.3% to 51.9%, demonstrating that this module helps preserve key points in boundaries, sparse regions, and geometrically abrupt regions, improving the discriminative power of the input point set. Removing the local spatial encoding unit reduced mIoU to 51.4%, indicating that extended geometric description enhances the ability to express differences in complex local structures. Removing the geometrically conditional attention pooling unit resulted in the largest reduction in mIoU, decreasing to 50.8%, indicating that this module is crucial for adaptive aggregation of neighborhood features. Removing the physical constraint joint training module reduced mIoU to 52.3%, demonstrating that physical priors help improve the reasonableness and interpretability of the results. Removing the rolling over inference module reduced mIoU to 52.6%, indicating that this module improves prediction stability.

[0080] Ablation experiments demonstrate that the proposed modules—geometric perception sampling, local spatial coding unit, geometric conditional attention pooling, physical constraint joint training, and rolling overlay inference—all contribute to performance improvement. Among these, the geometric conditional attention pooling unit and the extended local spatial coding module make the most significant contributions, indicating that this application possesses strong advantages in local geometric representation enhancement and neighborhood adaptive aggregation. The combined effect of these modules enables this application to achieve higher accuracy, stronger robustness, and better physical consistency in semantic segmentation and scene recognition output in complex real-world 3D scenes.

[0081] This application provides a method for understanding and recognizing real-world 3D data based on large-scale 3D geographic models. By coupling an efficient point cloud segmentation network with a scene recognition language model, and combining original point cloud preprocessing and index construction, geometric perception sampling, extended local spatial encoding, geometric conditional attention pooling, physical constraint joint training, and a rolling overlay inference mechanism, a complete processing chain from point-by-point semantic segmentation to scene-level language recognition of large-scale real-world 3D data is realized. This method combines high efficiency, geometric detail fidelity, physical consistency, and natural language interpretability, making it suitable for applications in fields such as large-scale 3D geographic models, autonomous driving, digital twin cities, intelligent surveying and mapping, and spatial intelligent analysis.

[0082] Existing lightweight semantic segmentation methods for large-scale point clouds typically reduce the point cloud size through downsampling and combine local neighborhood feature aggregation mechanisms to achieve efficient point-by-point semantic reasoning. These methods have certain advantages in terms of computational efficiency and engineering applicability, but they still have the following shortcomings in real-world 3D scene understanding and cognitive tasks: 1. When using the basic downsampling strategy, key points in geometric boundaries, sparse targets, and regions of structural abrupt changes are easily ignored, resulting in insufficient representation of important targets and local details.

[0083] 2. Existing local spatial feature modeling methods have limited ability to characterize geometric priors such as height differences, local scale changes, and directional consistency. In complex urban, road, and terrain environments, the ability to identify fine-grained structures still needs to be improved.

[0084] 3. Existing point-by-point semantic segmentation methods typically only output discrete category results, lacking further organization and abstraction of scene-level semantic relationships, making it difficult to directly support application needs such as 3D geographic models, intelligent surveying and mapping, autonomous driving cognitive decision-making, and natural language interaction.

[0085] 4. Prediction results that rely purely on data-driven approaches are prone to inconsistencies with the physical laws of the scene. For example, there may be abnormal spatial relationships between ground and non-ground categories, structural targets may not meet the required geometric regularity, and semantic predictions may be discontinuous between neighboring points. This can affect the interpretability and reliability of the results.

[0086] This application has the following beneficial effects: 1. By fusing geometric perception sampling and random sampling, the ability to retain boundary points, sparse points and key structural points is improved while maintaining the efficiency of large-scale point cloud processing.

[0087] 2. By using 14-dimensional extended local spatial encoding and geometric conditional attention pooling mechanism, the model's ability to express local geometric relationships, height changes, scale changes, and orientation consistency is enhanced, thereby improving the semantic segmentation accuracy and stability in complex real-world 3D scenes.

[0088] 3. By using the dilated residual local feature aggregation module, the receptive field is effectively expanded without introducing highly complex graph construction and kernel convolution operations, making it suitable for large-scale point clouds.

[0089] 4. By using gravity constraint loss, spatial consistency loss, geometric regularity loss, and height prior loss, the physical rationality of semantic prediction results is corrected, thereby improving the interpretability and engineering reliability of the segmentation results.

[0090] 5. By fusing rolling coverage reasoning and probabilistic voting, the coverage balance and reasoning robustness of large-scale scenarios during the testing phase are improved.

[0091] 6. By introducing a scene language description module, the point-by-point segmentation results are transformed into natural language scene cognition results. This enables the application to output not only the semantic category of each point, but also the scene type, main target composition, and risk warning information, making it more suitable for use in large-scale 3D geographic models, human-computer interaction, and upper-level business applications.

[0092] This application also provides an application scenario in which the above-mentioned real-scene 3D data scene understanding and cognition method is applied. Specifically, the real-scene 3D data scene understanding and cognition method provided in this embodiment can be applied to autonomous driving scenarios. An autonomous driving scenario includes a content production stage, a scene understanding and cognition link, and an autonomous driving stage. The dataset and target scene point cloud enter the scene understanding and cognition link from the content production stage, obtain scene cognition results through human-machine collaboration, and then enter the downstream autonomous driving stage. The real-scene 3D data scene understanding and cognition method provided in this embodiment belongs to the scene understanding and cognition link. Specifically, in the scene understanding and cognition link process for the target scene point cloud, the dataset can be used to train and optimize the point cloud semantic segmentation network to obtain a trained scene understanding model. The target scene point cloud is then input into the trained scene understanding model to obtain the point-by-point semantic segmentation results of the target scene point cloud. The scene cognition language model is then used to generate scene cognition results based on the point-by-point semantic segmentation results of the target scene point cloud.

[0093] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data for understanding and recognizing real-world 3D scenes. The I / O interfaces allow the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for understanding and recognizing real-world 3D scenes.

[0094] Figure 7 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0095] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0096] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0097] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0098] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0099] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for understanding and recognizing real-world 3D data scenes, characterized in that, The method includes: Obtain the dataset; the dataset includes several frames of original real-world 3D point cloud data in different scenes and the semantic category label corresponding to each frame of original real-world 3D point cloud data. Preprocess the original real-scene 3D point cloud data to obtain subsampled point clouds and KDTree indexes; Based on the subsampled point cloud and KDTree index, the center point and nearest neighbor set in the original real scene 3D point cloud data are determined to form a local input block; The coordinates of local block points in the local input block are input into a point cloud semantic segmentation network for point-by-point semantic category prediction, resulting in point-by-point semantic segmentation of the local input block. The point cloud semantic segmentation network includes an encoder, a decoder, and a classification head. The encoder is used to: perform multi-layer neighborhood mapping and geometrically perceptual sampling on the local input block to generate multi-layer neighborhood structure information; perform extended local spatial encoding on the center point and its neighboring points in the multi-layer neighborhood structure information, and aggregate it with the point features obtained from the shared MLP mapping to obtain the final encoded features; the decoder is used to decode the final encoded features to obtain decoded features; the classification head is used to perform point-by-point semantic category prediction based on the decoded features to obtain point-by-point semantic segmentation results. The encoder includes several dilated residual blocks. After completing the local aggregation features, each layer of the dilated residual block downsamples the point set of the current layer; a geometrically perceptual sampling mechanism is introduced in the first few layers of dilated residual blocks to downsample the point set in the local input block with a set sampling probability, obtaining several layers of point sets and neighborhood relationships; the downsampling method for the last few layers of dilated residual blocks is random sampling; the calculation method for setting the sampling probability is as follows: ； ； in, For point Set the sampling probability; These are geometric weighting coefficients; This represents the number of points in the local input block; Normalized points The geometric score; For point The geometric score; Normalized points The geometric score; It is a positive number; Represents the variance of the distance set; Point With The set of distances between neighboring points; Point The Coordinates of neighboring points; Indicates the first The three-dimensional coordinates of each point; The point cloud semantic segmentation network is trained to obtain a trained scene understanding model; The target scene point cloud is input into the trained scene understanding model to obtain the point-by-point semantic segmentation result of the target scene point cloud; Using a scene cognition language model, scene cognition results are generated based on the point-by-point semantic segmentation results of the target scene point cloud.

2. The method for understanding and recognizing real-world 3D data scenes according to claim 1, characterized in that, The local geometric coding is represented as follows: ； in, Local geometric coding; Center point Its first Euclidean distance between neighboring points; Center point Its first The relative coordinates of the neighboring points; The x, y, and z coordinates of the center point Center point The The x, y, and z coordinates of the neighboring points; Center point Its first The direction cosine similarity of the neighboring points; Features of height difference; For distance ratio features; This represents the local average distance feature.

3. The method for understanding and recognizing real-world 3D data scenes according to claim 1, characterized in that, The dilated residual block includes a geometrically conditional attention pooling unit; the geometrically conditional attention pooling unit computes the attention response from both the semantic feature branch and the positional encoding branch. ； ； ； ； in, and These represent the attention responses generated by the semantic feature branch and the positional encoding branch, respectively. and These represent the learnable mapping parameters for the semantic feature branch and the positional encoding branch, respectively; Indicates features of neighboring points With local geometric coding Enhanced neighborhood features formed by splicing; For the first The first center point corresponds to the Features of each neighboring point; This represents the normalized neighborhood attention weights; for Activation function; Indicates the first Local aggregation features obtained by aggregating multiple center points; For the first The number of neighborhood points of a center point; This indicates element-wise weighting; This indicates feature splicing.

4. The method for understanding and recognizing real-world 3D data scenes according to claim 1, characterized in that, Training a point cloud semantic segmentation network to obtain a trained scene understanding model specifically includes: training a point cloud semantic segmentation network using a joint loss function to obtain a trained scene understanding model. The joint loss function is expressed as follows: ； ； ； ； ； ； in, For the joint loss function; Category-weighted cross-entropy loss; , , , These are gravity constraint losses. Spatial consistency loss Geometric regularity loss High prior loss Weighting coefficients; For the first The center point belongs to the first The class probability; the probability of each point belonging to each class constitutes the point-by-point semantic segmentation result; The number of points; The total number of categories; For the first Class weights; For one-hot tags, when the first... The point belongs to the th point The value is 1 if the condition is met, otherwise it is 0. Indicates the center point Its neighboring points The set of adjacent pairs formed; This indicates the number of adjacency pairs in the set of adjacency pairs; For neighborhood points Located at the center point The soft indicator below; Indicates the first The predicted probability that each center point belongs to a ground-type target; Indicates the center point neighborhood points Belongs to the ground class collection The predicted probability; Center point Its neighboring points Distance weights between them; Jensen-Shannon divergence; set of class probabilities for center points , and The center point Belongs to category 1 and category 2 Class probability; set of neighborhood point class probabilities , and Neighboring points Belongs to category 1 and category 2 The probability of a class category; Indicates the center point The predicted probability of belonging to the structural class of targets; It is a local planarity; Center point The normalized relative height.

5. The method for understanding and recognizing real-world 3D data scenes according to claim 1, characterized in that, The target scene point cloud is input into a trained scene understanding model to obtain point-by-point semantic segmentation results of the target scene point cloud, specifically including: The target scene point cloud is input into the trained scene understanding model, and the point-by-point semantic segmentation result of the target scene point cloud is obtained by using rolling overlay reasoning and probabilistic voting fusion.

6. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the real-scene three-dimensional data scene understanding and cognition method according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the real-world 3D data scene understanding and cognition method as described in any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the real-world 3D data scene understanding and cognition method as described in any one of claims 1-5.

Images