Robot three-dimensional point cloud semantic segmentation method and device

Abstract

The application relates to the technical field of robots, and provides a robot three-dimensional point cloud semantic segmentation method and device, which comprises the following steps: extracting multi-scale point cloud features, initializing a Gaussian cell and iteratively updating parameters thereof, fusing neighbor point features through distance gating attention, finally obtaining a Gaussian semantic field and outputting a point cloud semantic segmentation result. The method provided in the embodiment of the application unifies the expression of geometric and semantic information in continuous three-dimensional space by constructing a Gaussian semantic field, designs a multi-round cascading optimization mechanism to promote the deep interaction of discrete multi-scale point cloud features and continuous Gaussian cells, and realizes adaptive feature fusion through a distance gating mechanism, so that the accurate spatial correspondence relationship between the point cloud geometric structure and the Gaussian continuous attribute is effectively established, the three-dimensional perception robustness of the robot when facing the object edges, small structures and occluded areas in a complex scene is significantly improved, and finally a stable and high-precision point cloud semantic segmentation result is output.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method and apparatus for semantic segmentation of 3D point clouds for robots. Background Technology

[0002] With the widespread adoption of 3D sensor technology, efficient and accurate semantic segmentation of 3D point clouds has become a crucial step for robots performing downstream tasks such as environmental understanding and grasping operations. Existing point cloud semantic segmentation methods are mainly divided into three categories: voxel-based methods discretize the point cloud into a regular voxel mesh and then use 3D convolution for segmentation; projection-based methods project the point cloud onto a 2D plane for segmentation and then backproject it back into 3D space; and point-based methods directly classify the original point cloud through point-by-point feature extraction and aggregation. All of these methods rely on discrete spatial representations, making it difficult to realistically depict the continuous 3D geometric structure in real-world scenes, resulting in low segmentation accuracy at object edges, small structures, or severely occluded areas. In recent years, 3D Gaussian splashing technology has demonstrated significant advantages in continuous modeling in scene reconstruction. It expresses the scene using a set of Gaussian primitives with continuous characteristics, enabling the rendering of high-quality images from any viewpoint.

[0003] However, existing Gaussian semantic field methods only focus on the interaction between Gaussian primitives and the rendered image, without considering the accurate correspondence with the point cloud geometry, making them unsuitable for point cloud semantic segmentation tasks. Therefore, how to integrate the continuity of Gaussian primitives into the point cloud semantic segmentation framework to achieve adaptive optimization of point cloud semantics is a problem worthy of in-depth research. Summary of the Invention

[0004] This invention provides a method and apparatus for semantic segmentation of 3D point clouds for robots, which solves the problem that existing point cloud segmentation based on discrete spatial representation lacks spatial continuity and that continuous Gaussian fields cannot accurately correspond to the geometric structure of point clouds, resulting in low segmentation accuracy of complex regions.

[0005] This invention provides a method for semantic segmentation of 3D point clouds for robots, comprising: Extract multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene; The Gaussian parameters obtained by initializing Gaussian primitives based on the original input point cloud are used as the current Gaussian parameters for the current round, and the Gaussian query features corresponding to the current Gaussian parameters are extracted. The center coordinates of each Gaussian element are retrieved to the nearest neighbor points in the original input point cloud, and the associated point cloud features corresponding to the nearest neighbor points are calculated based on the multi-scale point cloud features. Based on the Gaussian query features and the associated point cloud features, the original attention score map is calculated, and a distance gating tensor is constructed based on the spatial distance between the center coordinates and the nearest neighbor points. The original attention score map is weighted based on the distance gate tensor to obtain an adjusted attention map, and a Gaussian fusion mixing vector is obtained based on the adjusted attention map. The current Gaussian parameters are updated based on the Gaussian fusion mixing vector to obtain the updated Gaussian parameters. The updated Gaussian parameters are used as the current Gaussian parameters in the next round. The step of extracting the Gaussian query features corresponding to the current Gaussian parameters is returned and executed iteratively until the preset cutoff condition is met. The updated Gaussian parameters in the last round are used as the Gaussian semantic field parameter set. Based on the Gaussian semantic field parameter set, the fused semantic vector of each point in the original input point cloud is calculated, and the semantic segmentation result of the original input point cloud is obtained based on the fused semantic vector of each point.

[0006] According to the present invention, a semantic segmentation method for 3D point clouds of a robot is provided, wherein calculating the associated point cloud features corresponding to the nearest neighbor points based on the multi-scale point cloud features includes: Retrieve the nearest neighbor points and generate a nearest neighbor index tensor; Based on the nearest neighbor index tensor, the multi-scale point cloud features are sliced ​​and recombined to extract the multi-scale point cloud feature vectors corresponding to the nearest neighbor points, and the local multi-scale point cloud feature slices are reconstructed. Obtain the spatial coordinates of the nearest neighbor points and extract the position vectors of the nearest neighbor points; After concatenating the local multi-scale point cloud feature slices with the nearest neighbor point location vectors, feature mapping processing is performed on the concatenated vectors to obtain the associated point cloud features corresponding to the nearest neighbor points.

[0007] According to the present invention, a semantic segmentation method for 3D point clouds of a robot is provided, wherein the step of constructing a distance-gated tensor based on the spatial distance between the center coordinates and the nearest neighbor points includes: Calculate the Euclidean distance between the center coordinates of each Gaussian element and the nearest neighbor index tensor; The Euclidean distance is calculated using a preset first learnable parameter and a second learnable parameter to obtain the distance-gated tensor.

[0008] According to the present invention, a semantic segmentation method for 3D point clouds of a robot is provided, wherein the calculation of the original attention score map based on the Gaussian query features and the associated point cloud features includes: Feature extraction is performed on the Gaussian query features, and the spatial dimension is expanded to obtain the query tensor; Feature mapping is performed on the associated point cloud features to obtain key tensors and value tensors respectively; Calculate the inner product of the query tensor and the transpose of the key tensor, and divide the inner product by a preset scaling factor to obtain the original attention score map.

[0009] According to the present invention, a semantic segmentation method for robot 3D point clouds is provided, wherein extracting the Gaussian query features corresponding to the current Gaussian parameters includes: Map the center coordinates in the current Gaussian parameters to the set three-dimensional voxel space to obtain the voxel index; The voxel index and the Gaussian mixture vector in the current Gaussian parameters are mapped and sparse convolution is performed to obtain the first Gaussian feature map. After mapping the current Gaussian parameters to a unified feature dimension, element-wise addition is performed to obtain the second Gaussian feature map. The first Gaussian feature map and the second Gaussian feature map are concatenated along the channel dimension to obtain the third Gaussian feature map; The third Gaussian feature map is fused and processed to obtain the Gaussian query feature corresponding to the current Gaussian parameter.

[0010] According to the present invention, a semantic segmentation method for robot 3D point clouds includes updating the current Gaussian parameters based on the Gaussian fusion mixture vector to obtain updated Gaussian parameters, comprising: The Gaussian fusion mixing vector is processed to predict the center coordinate residual, rotation quaternion residual, scale residual, weight residual and semantic residual; The center coordinates in the current Gaussian parameters are updated by gated incremental update with the center coordinate residuals to obtain the updated center coordinates. The rotated quaternion residual is multiplied by the rotated quaternion in the current Gaussian parameters to obtain the updated rotated quaternion; Based on the preset scale adjustment constraints, the scale parameters in the current Gaussian parameters are incrementally superimposed and updated with the scale residuals to obtain the updated scale parameters. The weight residuals and semantic residuals are processed using activation functions to obtain updated Gaussian weights and updated semantic vectors. The updated center coordinates, rotation quaternions, scale parameters, Gaussian weights, and semantic vectors are used as the updated Gaussian parameters.

[0011] According to the present invention, a semantic segmentation method for 3D point clouds of a robot is provided, wherein calculating the fused semantic vector of each point in the original input point cloud based on the Gaussian semantic field parameter set includes: The convergent center coordinates in the Gaussian semantic field parameter set are mapped to a pre-divided three-dimensional voxel space to establish a three-dimensional spatial index. For the target point in the original input point cloud, a local candidate set is retrieved in the local neighborhood where the target point is located based on the three-dimensional spatial index; Calculate the Mahalanobis distance between the target point and the center coordinates of each Gaussian element in the local candidate set; Using a preset semantic truncation threshold, the Mahalanobis distance is filtered to obtain a set of Gaussian elements that meet the conditions. For the set of Gaussian elements that meet the conditions, a probability density decay term is calculated. The probability density decay term is multiplied by the Gaussian weights and the semantic vector and then accumulated to obtain the fused semantic vector of the target point.

[0012] According to the present invention, a semantic segmentation method for robot 3D point clouds is provided, wherein obtaining the semantic segmentation result of the original input point cloud based on the fused semantic vectors of the points includes: The fused semantic vectors of the various points are concatenated to form the overall semantic probability tensor; For the overall semantic probability tensor, extract the category index corresponding to the maximum value along the category dimension, output the semantic prediction value vector, and use the semantic prediction value vector as the semantic segmentation result of the original input point cloud.

[0013] According to the present invention, a method for semantic segmentation of 3D point clouds for robots includes extracting multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene, comprising: The original input point cloud is downsampled multiple times to obtain multiple sets of downsampled point clouds with different resolutions; Feature extraction is performed on each downsampled point cloud set to generate multi-scale point cloud features at different scales. The multi-scale point cloud features of different scales are upsampled and mapped back to the spatial location of the original input point cloud to obtain multiple upsampled point cloud features. The multi-scale point cloud features are obtained by stitching together and performing channel dimensionality reduction on multiple upsampled point cloud features.

[0014] The present invention also provides a robot 3D point cloud semantic segmentation device, comprising: The point cloud feature extraction unit extracts multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene; The Gaussian parameter feature extraction unit takes the Gaussian parameters obtained by initializing Gaussian primitives based on the original input point cloud as the current Gaussian parameters of the current round and extracts the Gaussian query features corresponding to the current Gaussian parameters. The associated point cloud feature extraction unit retrieves the nearest neighbor points of the center coordinates of each Gaussian element in the original input point cloud, and calculates the associated point cloud features corresponding to the nearest neighbor points based on the multi-scale point cloud features. The distance gating construction unit calculates the original attention score map based on the Gaussian query features and the associated point cloud features, and constructs a distance gating tensor based on the spatial distance between the center coordinates and the nearest neighbor points. The weighted adjustment unit adjusts the weights of the original attention score map based on the distance gate tensor to obtain the adjusted attention map, and obtains the Gaussian fusion mixing vector based on the adjusted attention map. The Gaussian parameter iterative update unit updates the current Gaussian parameters based on the Gaussian fusion mixing vector to obtain the updated Gaussian parameters. The updated Gaussian parameters are used as the current Gaussian parameters in the next round. The unit returns to the step of extracting the Gaussian query features corresponding to the current Gaussian parameters. The unit iteratively executes the steps until the preset cutoff condition is met. The updated Gaussian parameters in the last round are used as the Gaussian semantic field parameter set. The semantic segmentation unit calculates the fused semantic vector of each point in the original input point cloud based on the Gaussian semantic field parameter set, and obtains the semantic segmentation result of the original input point cloud based on the fused semantic vector of each point.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot 3D point cloud semantic segmentation method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot 3D point cloud semantic segmentation method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot 3D point cloud semantic segmentation method as described above.

[0018] The robot 3D point cloud semantic segmentation method and apparatus provided by this invention constructs a Gaussian semantic field to uniformly express geometric and semantic information in continuous 3D space, designs a multi-round cascaded optimization mechanism to promote deep interaction between discrete multi-scale point cloud features and continuous Gaussian primitives, and achieves adaptive feature fusion with a distance gating mechanism. This not only effectively establishes an accurate spatial correspondence between the point cloud geometric structure and Gaussian continuous attributes, but also significantly improves the robot's 3D perception robustness when facing object edges, small structures and occluded areas in complex scenes, and finally outputs stable and high-precision point cloud semantic segmentation results. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the robot 3D point cloud semantic segmentation method provided by the present invention; Figure 2 This is a schematic diagram of the structure of the robot 3D point cloud semantic segmentation device provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] It should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of the present invention, "multiple" means two or more, unless otherwise explicitly specified. Furthermore, all actions involving the acquisition of signals, information, or data in this invention are performed in accordance with the applicable data protection regulations and policies of the locality and with permission granted by the owner of the relevant device.

[0023] To address the problem that existing point cloud segmentation technologies based on discrete spatial representation lack spatial continuity, and that continuous Gaussian fields cannot accurately correspond to point cloud geometry, resulting in low segmentation accuracy for complex regions, this invention provides a robot 3D point cloud semantic segmentation method. This method uniformly expresses geometric and semantic information in continuous 3D space, thereby improving the accuracy of point cloud semantic segmentation. It should be noted that the robot 3D point cloud semantic segmentation method provided in this embodiment can construct a point cloud semantic segmentation model. The point cloud semantic segmentation model can be a pre-constructed robot point cloud semantic segmentation network containing a multi-scale point cloud feature extractor, a Gaussian semantic field initialization module, a Gaussian semantic field optimization module, and a Gaussian semantic field splashing module. That is, the multi-scale point cloud feature extractor can be used to perform step 110, the Gaussian semantic field initialization module can be used to obtain the current Gaussian parameters in step 120, the Gaussian semantic field optimization module can be used to extract the Gaussian query features corresponding to the current Gaussian parameters in step 120, and steps 130 to 160, and the Gaussian semantic field splashing module can be used to perform step 170. Figure 1 This is a flowchart illustrating the robot 3D point cloud semantic segmentation method provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step 110: Extract multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene.

[0024] Here, "robot" refers to an intelligent entity that performs downstream tasks such as environmental understanding and grasping operations in a real physical environment. The current scene refers to the robot's current 3D physical working environment. Therefore, the original input point cloud here is a set of discrete points containing real physical information such as 3D coordinates, collected by a 3D vision sensor in this working environment, used for subsequent semantic segmentation. Furthermore, the multi-scale point cloud features here aim to characterize the spatial geometric structure and deep global semantic information of the aforementioned discrete point cloud under different receptive fields.

[0025] Specifically, the original input point cloud can be iterated multiple times along the dimension containing the points to perform farthest distance retrieval and downsampling operations, resulting in multiple sets of downsampled point clouds at different resolutions. Then, features are extracted from each set of point clouds at different resolutions, and the extracted multi-scale features are mapped back to the spatial locations of the original input point cloud using inverse distance weighted interpolation algorithms. Finally, these mapped features are fused through channel concatenation and linear dimensionality reduction to obtain multi-scale point cloud features that possess both macroscopic vision and microscopic detail representation capabilities.

[0026] Step 120: Use the Gaussian parameters obtained by initializing Gaussian elements based on the original input point cloud as the current Gaussian parameters for the current round, and extract the Gaussian query features corresponding to the current Gaussian parameters.

[0027] Here, Gaussian elements are the fundamental structures with continuous properties used to express scene geometry and continuous semantics in three-dimensional space. Gaussian parameters are the set of variables that define and describe the physical properties and feature distribution of each Gaussian element, such as center point, covariance, weights, and semantic distribution. The current Gaussian parameters in the current iteration refer to the intermediate Gaussian parameter variables received and processed by the current level during the multi-level cascaded iterative optimization process. Furthermore, the Gaussian query features here represent the features that Gaussian elements actively retrieve spatial and semantic information from external discrete point cloud features in subsequent feature interaction mechanisms.

[0028] Specifically, based on the original input point cloud, local sampling and covariance calculation are performed to obtain initialized Gaussian parameters, which are then used as the current Gaussian parameters in the first iteration of the cascaded loop. Subsequently, the coordinates in the current Gaussian parameters of the current iteration are mapped to a specific 3D voxel space to perform sparse convolution feature extraction. Combined with the linear mapping results of the parameters themselves, channel-level feature concatenation and multilayer perceptron fusion processing are performed to output the Gaussian query features corresponding to the current Gaussian parameters.

[0029] Step 130: Retrieve the nearest neighbor points of the center coordinates of each Gaussian element in the original input point cloud, and calculate the associated point cloud features corresponding to the nearest neighbor points based on the multi-scale point cloud features.

[0030] Here, the center coordinates represent the core position vector of each Gaussian element in a continuous three-dimensional space. The nearest neighbor refers to the set of discrete point clouds in the original input point cloud that are physically closest to the center coordinates of a specific Gaussian element. Furthermore, the associated point cloud features are feature combinations extracted from slices of the aforementioned multi-scale point cloud features, used to characterize the local geometry and contextual relationships of these specific nearest neighbors.

[0031] Specifically, the spatial Euclidean distance between the center coordinates of each Gaussian unit and each discrete point in the original input point cloud can be calculated, and a predetermined number of nearest points can be retrieved to generate a nearest neighbor index. Then, this nearest neighbor index is used to extract slices in the multi-scale point cloud features, and these slices are recombined and concatenated with the spatial location vectors of the nearest points. Subsequently, a multilayer perceptron is used to perform nonlinear feature mapping to calculate the associated point cloud features containing both location and deep semantics.

[0032] Step 140: Based on the Gaussian query features and the associated point cloud features, calculate the original attention score map, and construct the distance gating tensor based on the spatial distance between the center coordinates and the nearest neighbor points.

[0033] Here, the original attention score map reflects the initial cross-modal feature matching response between the Gaussian query features and the associated point cloud features. Here, the distance-gated tensor is a non-linear decaying weight constructed based on the real physical spatial distance between Gaussian units and their nearest neighbors, serving as a spatial geometric constraint switch in the feature fusion process.

[0034] Specifically, the Gaussian query features and associated point cloud features can be linearly transformed and then subjected to inner product and scaling operations to obtain the original attention score map. Simultaneously, the spatial distance between the center coordinates and the corresponding nearest neighbors is calculated, and a nonlinear activation mapping is performed using learnable scaling and bias parameters to construct a distance-gated tensor.

[0035] Step 150: Adjust the weights of the original attention score map based on the distance gating tensor to obtain the adjusted attention map, and obtain the Gaussian fusion mixing vector based on the adjusted attention map.

[0036] The adjusted attention map here is a high-confidence attention weight matrix after filtering out spatially biased interference terms through distance constraints. Furthermore, the Gaussian fusion mixing vector here is the final comprehensive feature representation that aggregates the continuous attributes of Gaussian primitives and the discrete features of the point cloud.

[0037] Specifically, the normalized original attention score map can be adjusted element-wise by multiplying using the distance-gated tensor. Then, the adjusted attention map is multiplied and dimension-reduced by matrix multiplication with the value vectors of associated point cloud features to output a Gaussian fusion mixture vector.

[0038] It should be noted that by introducing a cross-attention mechanism with a spatial distance gating function during the feature interaction process, adaptive dynamic fusion of discrete multi-scale point cloud features and continuous Gaussian features is achieved, effectively eliminating feature interference caused by positional deviations of associated points and ensuring the accuracy of geometric feature transfer.

[0039] Step 160: Update the current Gaussian parameters based on the Gaussian fusion mixing vector to obtain the updated Gaussian parameters. Use the updated Gaussian parameters as the current Gaussian parameters for the next round. Return to the step of extracting the Gaussian query features corresponding to the current Gaussian parameters. Iterate until the preset cutoff condition is met. Use the updated Gaussian parameters in the last round as the Gaussian semantic field parameter set.

[0040] Here, the updated Gaussian parameters are a set of parameters that have been adaptively optimized by fusing local geometric and deep semantic features of the point cloud. The Gaussian semantic field parameter set is a combination of parameters that has finally converged after multiple rounds of cascaded iterations and can accurately represent the bidirectional continuity of the geometry and semantics of the entire 3D scene.

[0041] Specifically, firstly, Gaussian fusion mixing vectors can be used to generate prediction residuals for center coordinates, rotation quaternions, scale, weights, and semantics through different prediction branches. Then, by combining constraints such as progressively gated incremental formulas or direct assignment of activation functions, the various attributes in the current Gaussian parameters can be superimposed, updated, or replaced to obtain updated Gaussian parameters. Subsequently, these updated Gaussian parameters are assigned as input parameters to the next round, and the above series of steps—feature extraction, spatial query, and fusion update—is repeated until a preset maximum number of cascaded rounds is reached, or the updated Gaussian parameters meet a preset convergence condition. At this point, the parameters output from the final round after convergence are fixed as the final set of Gaussian semantic field parameters.

[0042] Step 170: Based on the Gaussian semantic field parameter set, calculate the fused semantic vector of each point in the original input point cloud, and obtain the semantic segmentation result of the original input point cloud based on the fused semantic vector of each point.

[0043] Here, the fused semantic vector refers to the continuous probability density representation tensor obtained by aggregating the relevant Gaussian semantic attributes within the local neighborhood of each discrete 3D point in the original input point cloud. The semantic segmentation result is the specific scene classification label assigned to each point after discretizing the aforementioned continuous probability density.

[0044] Specifically, a 3D spatial index can be established using the center coordinates of the converged Gaussian semantic field parameter set to accelerate spatial retrieval efficiency. For each target point in the original input point cloud, relevant Gaussian primitives are retrieved within its local neighborhood, and primitives with mismatched shapes and positions are filtered out using a Mahalanobis distance threshold. Then, for the set of retained Gaussian primitives, the corresponding probability density decay term is calculated, multiplied by Gaussian weights and semantic attributes, and accumulated one by one to obtain the fused semantic vector for each target point. Finally, the category index corresponding to the maximum probability value of the fused semantic vector of all points is extracted along the category channel dimension, serving as the final semantic segmentation result of the original input point cloud.

[0045] The method provided in this invention constructs a Gaussian semantic field to uniformly express geometric and semantic information in a continuous three-dimensional space, designs a multi-round cascaded optimization mechanism to promote deep interaction between discrete multi-scale point cloud features and continuous Gaussian primitives, and achieves adaptive feature fusion with a distance gating mechanism. This not only effectively establishes an accurate spatial correspondence between the geometric structure of the point cloud and the continuous Gaussian properties, but also significantly improves the robustness of the robot's three-dimensional perception when facing object edges, small structures and occluded areas in complex scenes, and finally outputs stable and high-precision point cloud semantic segmentation results.

[0046] Based on any of the above embodiments, in step 130, calculating the associated point cloud features corresponding to the nearest points based on the multi-scale point cloud features includes: Retrieve the nearest neighbor points and generate a nearest neighbor index tensor; Based on the nearest neighbor index tensor, the multi-scale point cloud features are sliced ​​and recombined to extract the multi-scale point cloud feature vectors corresponding to the nearest neighbor points, and the local multi-scale point cloud feature slices are reconstructed. Obtain the spatial coordinates of the nearest neighbor points and extract the position vectors of the nearest neighbor points; After concatenating the local multi-scale point cloud feature slices with the nearest neighbor point location vectors, feature mapping processing is performed on the concatenated vectors to obtain the associated point cloud features corresponding to the nearest neighbor points.

[0047] Specifically, firstly, nearest neighbor points are retrieved, and a nearest neighbor index tensor is generated. Here, the nearest neighbor index tensor is a two-dimensional matrix that records the position indices of several three-dimensional discrete points in the original input point cloud that are closest to each Gaussian cell center in the global data. It is used to quickly locate locally relevant points in massive point clouds.

[0048] In practice, the pairwise three-dimensional Euclidean distance between the center coordinates of each Gaussian element and all discrete points in the original input point cloud can be calculated. The preset number of local neighboring points closest to each Gaussian center can be retrieved by sorting, and the position numbers of these neighboring points can be recorded to generate a nearest neighbor index tensor.

[0049] Then, based on the nearest neighbor index tensor, the multi-scale point cloud features are sliced ​​and recombined to extract the multi-scale point cloud feature vectors corresponding to the nearest neighbors, and the local multi-scale point cloud feature slices are reconstructed. Here, the local multi-scale point cloud feature slices are three-dimensional feature blocks formed by combining the features of all nearest neighbors belonging to the same Gaussian local neighborhood.

[0050] In practice, based on the nearest neighbor index tensor generated above, feature index slicing can be performed on the fused multi-scale point cloud features to accurately extract the corresponding multi-scale point cloud feature vectors of these nearest neighbors. Subsequently, these extracted discrete feature vectors are reconstructed and their dimensions adjusted according to the hierarchical relationship between Gaussian units and nearest neighbors to generate local multi-scale point cloud feature slices.

[0051] Next, the spatial coordinates of the nearest points are obtained, and the nearest point position vector is extracted. Here, the spatial coordinates refer to the three-dimensional coordinate values ​​of the nearest points in the real three-dimensional physical world. Here, the nearest point position vector is a local spatial geometric representation composed of these specific physical coordinate values, used to explicitly provide the network with absolute position information.

[0052] In practice, based on the nearest neighbor index tensor generated above, the coordinate data of the original input point cloud are accurately indexed and the three-dimensional spatial coordinates of these nearest neighbors are obtained. These coordinates are then extracted and combined to form the nearest neighbor position vector.

[0053] Finally, the local multi-scale point cloud feature slices are concatenated with the nearest neighbor point location vectors, and feature mapping is performed on the concatenated vectors to obtain the associated point cloud features corresponding to the nearest neighbors. Here, the associated point cloud features are high-level local context features that deeply combine the absolute spatial geometric position of the point cloud with multi-scale semantic priors.

[0054] In practice, along the last data dimension, the local multi-scale point cloud feature slice containing semantic priors is concatenated with the nearest neighbor position vector containing absolute physical geometric information. Subsequently, the concatenated high-dimensional vector is input into a multi-layer perceptron (MLP) for non-linear feature mapping and depth processing, and finally the associated point cloud features corresponding to the nearest neighbor are calculated.

[0055] The method provided in this invention uses a local feature slicing mechanism based on nearest neighbor index to explicitly splice and nonlinearly map and fuse high-order multi-scale semantic features with underlying absolute spatial physical coordinates. This successfully constructs a locally associated point cloud feature that contains both rich semantic priors and precise spatial location constraints, providing important local geometric guidance for high-precision cross-modal attention matching between Gaussian query features and point cloud features.

[0056] Based on any of the above embodiments, in step 140, constructing a distance-gated tensor based on the spatial distance between the center coordinates and the nearest neighbor points includes: Calculate the Euclidean distance between the center coordinates of each Gaussian element and the nearest neighbor index tensor; The Euclidean distance is calculated using a preset first learnable parameter and a second learnable parameter to obtain the distance gate tensor; Specifically, first, the Euclidean distance between the center coordinates of each Gaussian element and the corresponding nearest neighbor index tensor is calculated. Then, using preset first and second learnable parameters, the Euclidean distance is calculated to obtain the distance-gated tensor.

[0057] Here, Euclidean distance refers to the absolute straight-line distance between the center of a Gaussian unit and its nearest discrete point cloud in three-dimensional physical space. The first and second learnable parameters are adaptively optimized weight variables obtained through network training, used to control the sensitivity and truncation boundary of distance decay, respectively. The distance-gated tensor is a set of spatially constrained weight matrices, after spatial distance decay mapping, used to characterize the spatial matching confidence of each nearest neighbor point.

[0058] In practice, the three-dimensional Euclidean distance between the center coordinates of each Gaussian element and the corresponding nearest neighbor index tensor is calculated. Then, a preset first learnable parameter is used... With the second learnable parameter Same as the above Euclidean distance The differences are multiplied, and a sigmoid activation function (Sigmoid) is applied for nonlinear mapping calculation to obtain the distance-gated tensor. For example, it can be obtained through the formula... The distance-gated tensor was calculated. .

[0059] In step 150, the original attention score map is weighted based on the distance-gated tensor to obtain the adjusted attention map, including: The original attention score map is normalized, and the normalized original attention score map is multiplied element-wise with the distance gating tensor to obtain the adjusted attention map.

[0060] Specifically, the original attention score map can be normalized by applying the Softmax Activation Function (Softmax). Then, the normalized original attention score map is multiplied element-wise with the distance-gated tensor generated in the previous steps to obtain the final adjusted attention map.

[0061] The method provided in this invention constructs a smooth gating constraint based on spatial physical location by calculating Euclidean distance and combining it with learnable parameters for adaptive nonlinear mapping. Then, the constraint tensor is used to adjust the weights of the attention score through element-wise multiplication, which accurately suppresses semantic matching interference caused by spatially distant but similar features, and significantly enhances the fidelity of local geometric alignment during feature fusion.

[0062] Based on any of the above embodiments, in step 140, the original attention score map is calculated based on the Gaussian query features and the associated point cloud features, including: Feature extraction is performed on the Gaussian query features, and the spatial dimension is expanded to obtain the query tensor; Feature mapping is performed on the associated point cloud features to obtain key tensors and value tensors respectively; Calculate the inner product of the query tensor and the transpose of the key tensor, and divide the inner product by a preset scaling factor to obtain the original attention score map.

[0063] Specifically, firstly, feature extraction is performed on the Gaussian query features, and the spatial dimension is expanded to obtain the query tensor. Here, the Gaussian query features refer to the vectors representing the Gaussian elements actively retrieving information from the local point cloud.

[0064] In practice, the Gaussian query features are input into a linear layer for feature extraction and spatial transformation. Then, the data dimension is added to a specific spatial dimension using the dimension expansion operation, thereby obtaining the shape-matching query tensor.

[0065] Then, feature mapping is performed on the associated point cloud features to extract key and value tensors respectively. Here, the key tensor is a reference feature used for similarity matching with the query tensor, while the value tensor contains the local semantic content information that actually needs to be extracted and aggregated after a successful match. In practice, the associated point cloud features can be fed into two independent parallel linear layers for feature mapping and processing, thereby extracting and generating the corresponding key and value tensors respectively.

[0066] Finally, the inner product of the query tensor and the transpose of the key tensor is calculated, and then divided by a preset scaling factor to obtain the original attention score map. Here, the transpose tensor refers to the data representation after matrix row and column interchange of the key tensor. The scaling factor is a numerical smoothing parameter set to prevent the network gradient from vanishing or the distribution from becoming extreme due to excessively large values ​​of high-dimensional feature dot products. In specific implementation, batch matrix multiplication operations are performed to calculate the inner product between the query tensor and the transpose of the key tensor, and then the result of the inner product is divided by the preset scaling factor to obtain the original attention score map representing the initial matching degree.

[0067] The method provided in this invention, by introducing independent feature mappings for queries, keys, and values, as well as a dot product attention mechanism with scaling factors, effectively and stably measures the deep feature similarity between the retrieval intent of continuous Gaussian units and discrete local point clouds. This provides a basic attention response map for subsequent high-precision cross-modal feature adaptive aggregation by combining distance gating.

[0068] Based on any of the above embodiments, step 120, extracting the Gaussian query features corresponding to the current Gaussian parameters, includes: Map the center coordinates in the current Gaussian parameters to the set three-dimensional voxel space to obtain the voxel index; The voxel index and the Gaussian mixture vector in the current Gaussian parameters are mapped and sparse convolution is performed to obtain the first Gaussian feature map. After mapping the current Gaussian parameters to a unified feature dimension, element-wise addition is performed to obtain the second Gaussian feature map. The first Gaussian feature map and the second Gaussian feature map are concatenated along the channel dimension to obtain the third Gaussian feature map; The third Gaussian feature map is fused and processed to obtain the Gaussian query feature corresponding to the current Gaussian parameter.

[0069] Specifically, firstly, the center coordinates in the current Gaussian parameters are mapped to a defined 3D voxel space to obtain voxel indices. Then, the voxel indices are mapped to the Gaussian mixture vector in the current Gaussian parameters, and sparse convolution is performed to obtain the first Gaussian feature map. Here, the 3D voxel space is a 3D grid discretized at a set resolution. Here, the voxel index is the discrete coordinate position identifier of the Gaussian element in this grid system. Here, the Gaussian mixture vector is the implicit high-dimensional feature representation carried by the Gaussian element. Thus, the first Gaussian feature map is the extracted feature representation containing local spatial structure information of the Gaussian elements.

[0070] In practice, the center coordinates in the current Gaussian parameters can be mapped to a 3D voxel space with a set side length and resolution, and the corresponding voxel index can be calculated. Then, the voxel index is associated with the Gaussian mixture vector in the current Gaussian parameters, and the corresponding data is fed into the sparse convolutional sub-network to perform sparse convolution processing to extract the first Gaussian feature map.

[0071] Then, after mapping the current Gaussian parameters to a unified feature dimension, element-wise addition is performed to obtain the second Gaussian feature map. Here, the second Gaussian feature map is a feature combination representation that integrates all explicit physical properties of the Gaussian primitive itself.

[0072] In practice, multiple parallel linear layers are used to perform non-linear feature mapping on various attributes of the current Gaussian parameters, such as center coordinates, covariance, Gaussian weights, and semantic vectors, projecting them onto a unified feature dimension. Subsequently, element-wise addition is performed on the mapped features to obtain a second Gaussian feature map that integrates multiple explicit attributes.

[0073] Furthermore, the first Gaussian feature map and the second Gaussian feature map are concatenated along the channel dimension to obtain the third Gaussian feature map. Here, the third Gaussian feature map is a comprehensive feature combination that simultaneously includes local topological features in space and the explicit physical properties of the Gaussian unit itself.

[0074] In practice, a channel concatenation operation can be used to concatenate the first Gaussian feature map extracted from the sparse network with the second Gaussian feature map that incorporates explicit attributes along the last channel dimension to obtain a high-dimensional third Gaussian feature map.

[0075] Finally, the third Gaussian feature map is fused and processed to obtain the Gaussian query feature corresponding to the current Gaussian parameters. In specific implementation, the concatenated third Gaussian feature map is fed into a multi-layer perceptron (MLP) to perform non-linear feature fusion and channel dimensionality reduction processing, and finally outputs the extracted Gaussian query feature corresponding to the current Gaussian parameters.

[0076] The method provided in this invention combines spatial topological features extracted by sparse convolution with Gaussian explicit physical properties extracted by parallel linear mapping, and uses a multilayer perceptron for deep splicing and fusion, successfully constructing a comprehensive Gaussian query feature with extremely rich expressive power, thereby improving the accuracy of subsequent retrieval and matching.

[0077] In one embodiment, the Gaussian semantic field optimization module includes indivual( There can be three sequentially connected semantic optimization units, each of which contains a Gaussian feature extraction subnetwork, an adaptive Gaussian-point cloud fusion subnetwork, and a Gaussian update subnetwork.

[0078] The Gaussian feature extraction subnetwork comprises a sparse convolutional subnetwork (SparseConv), a feature enhancement layer, and a feature concatenation layer. The SparseConv subnetwork can have 64 output channels. The feature enhancement layer consists of four parallel linear layers: Linear_mu, Linear_sigma, Linear_pi, and Linear_vs, configured as follows: the four linear layers use weight matrices of sizes 3×64, 6×64, 1×64, and 20×64, respectively, with the weight matrix parameters in the format of input feature number × output feature number. All four linear layers have learnable bias parameters, implemented using the `nn.linear` function in the PyTorch library. The feature concatenation layer consists of channel feature concatenation and a multilayer perceptron (MLP_fuse). Channel feature concatenation is implemented using the torch.cat function in PyTorch. The MLP_fuse uses a 3×64 weight matrix, with the weight matrix parameters in the format of input feature number × output feature number, and ReLU activation, implemented using the nn.Linear and nn.ReLU functions in PyTorch.

[0079] The Gaussian feature extraction subnetwork is used to extract the Gaussian query features corresponding to the current Gaussian parameters. The specific process includes: The central coordinates of each Gaussian element Mapped to voxel edge length is (e.g., 10mm), resolution is A three-dimensional voxel space (which can be set to 100×100×100) is used to obtain the voxel index. and voxel indexing Gaussian mixture vector The corresponding data is then fed into the sparse convolutional sub-network SparseConv for processing, resulting in the first Gaussian feature map. The size is .

[0080] Meanwhile, the current Gaussian parameters (center coordinates) Covariance matrix Gaussian weights semantic vectors The second Gaussian feature map is obtained by mapping the data through four linear layers (Linear_mu, Linear_sigma, Linear_pi, and Linear_vs) to 64 dimensions and then performing element-wise addition. The size is Then, and The data is fed into the feature concatenation layer, and the torch.cat function in PyTorch is used to concatenate the data along the channel dimension to obtain the third Gaussian feature map. The size is , After processing by a multilayer perceptron (MLP_fuse), the Gaussian query features are output. The size is .

[0081] Additionally, the adaptive Gaussian-point cloud fusion subnetwork can consist of a spatial location retrieval unit, a feature index slicing unit, a feature concatenation layer, a multilayer perceptron (MLP_assoc), a linear layer (Linear_Q), a linear layer (Linear_K), a linear layer (Linear_V), and a first learnable parameter. Second learnable parameter The system consists of a sigmoid activation function, a softmax normalization layer, a matrix multiplication unit, and a Squeeze dimensionality reduction unit. The multilayer perceptron MLP_assoc (weight matrix size 259×64), the first linear layer Linear_Q (weight matrix size 64×64), the second linear layer Linear_K (weight matrix size 64×64), and the third linear layer Linear_V (weight matrix size 64×64) all have learnable bias vectors, implemented using the nn.Linear function in the PyTorch library; the first learnable parameter... Second learnable parameter The following functions are implemented using the `nn.Parameter` function from the PyTorch library: the Sigmoid activation function, the `nn.Sigmoid` function from the `torch` library, the Softmax normalization layer, the `nn.Softmax` function from the PyTorch library, the matrix multiplication unit, the `torch.bmm` function from the PyTorch library, the feature concatenation layer, the `torch.cat` function from the PyTorch library, and the Squeeze dimensionality reduction unit from the `torch.squeeze` function from the `torch` library.

[0082] Understandably, the adaptive Gaussian-point cloud fusion subnetwork is used to perform steps 130 to 150. Specifically, the processing flow of the adaptive Gaussian-point cloud fusion subnetwork includes: receiving Gaussian query features through the adaptive Gaussian-point cloud fusion subnetwork. (size is) First, the center coordinates of the Gaussian element can be calculated using the spatial location retrieval unit. (size is) ) and the original input point cloud The pairwise Euclidean distance between points (of size M×3) is used to retrieve the K nearest neighbor points (K=16 in this embodiment), generating a nearest neighbor index tensor with size M×3. Then, the fused multi-scale point cloud features can be processed using feature index slicing units based on the nearest neighbor index tensor. (Size M×256) Perform feature index slicing and reconstruction, extract the multi-scale point cloud feature vectors of the corresponding K nearest neighbors, and reconstruct them into local multi-scale point cloud feature slices, size M×256. Additionally, the spatial coordinates of the corresponding K nearest neighbors are extracted to form a nearest neighbor position vector (with a size of [missing information]). Next, the local multi-scale point cloud feature slices and the nearest point location vectors can be concatenated along the last dimension using a feature concatenation layer, and then processed by a multilayer perceptron (MLP_assoc) to obtain the associated point cloud features. The size is .

[0083] Subsequently, Gaussian query features After processing by the first linear layer (Linear_Q) and expanding its spatial dimension, we obtain the query tensor Q, with a size of [size missing]. Additionally, associated point cloud features After processing through a second linear layer (Linear_K) and a third linear layer (Linear_V), we obtain the key tensor K and the value tensor V, both with the same size. Then, the torch.bmm function can be used to calculate the inner product of the query tensor Q and the transpose of the key tensor K, and divided by the scaling factor 8 to obtain the original attention score map with a size of [size missing]. .

[0084] Simultaneously, the nearest neighbor index tensor and the center coordinates of the corresponding Gaussian elements are calculated. Euclidean distance between Using the first learnable parameter Second learnable parameter and calculation formula Obtain the distance-gated tensor The size is Furthermore, the original attention score map is processed through a Softmax normalization layer and then compared with the distance-gated tensor. Element-wise multiplication yields the adjusted attention map. The size is Finally, the adjusted attention map can be calculated using the torch.bmm function. Matrix multiplication with the value tensor V generates the fifth Gaussian feature map. The size is Fifth Gaussian feature map After removing the spatial dimension of 1 by the Squeeze dimensionality reduction unit, the output is a Gaussian blending vector. The size is .

[0085] Based on any of the above embodiments, in step 160, updating the current Gaussian parameters based on the Gaussian fusion mixing vector to obtain the updated Gaussian parameters includes: The Gaussian fusion mixing vector is processed to predict the center coordinate residual, rotation quaternion residual, scale residual, weight residual and semantic residual; The center coordinates in the current Gaussian parameters are updated by gated incremental update with the center coordinate residuals to obtain the updated center coordinates. The rotated quaternion residual is multiplied by the rotated quaternion in the current Gaussian parameters to obtain the updated rotated quaternion; Based on the preset scale adjustment constraints, the scale parameters in the current Gaussian parameters are incrementally superimposed and updated with the scale residuals to obtain the updated scale parameters. The weight residuals and semantic residuals are processed using activation functions to obtain updated Gaussian weights and updated semantic vectors. The updated center coordinates, rotation quaternions, scale parameters, Gaussian weights, and semantic vectors are used as the updated Gaussian parameters.

[0086] Specifically, firstly, the Gaussian fusion mixture vector is processed to predict the center coordinate residual, rotation quaternion residual, scale residual, weight residual, and semantic residual. Here, the center coordinate residual, rotation quaternion residual, scale residual, weight residual, and semantic residual refer to the numerical increments or adjustment biases predicted by the network for each attribute of the Gaussian element in the current cascaded iteration, such as its 3D spatial position, rotation orientation, size, rendering opacity, and semantic category distribution.

[0087] In practice, the Gaussian fusion hybrid vector output from the current round can be input into multiple parallel linear layers for feature dimension mapping and processing, thereby independently predicting the center coordinate residual, rotation quaternion residual, scale residual, weight residual, and semantic residual.

[0088] Next, the corresponding current Gaussian parameters are updated using the predicted values ​​to obtain the updated Gaussian parameters. Specifically, for the update of the center coordinates, a gated incremental update is performed between the center coordinates in the current Gaussian parameters of the current round and the center coordinate residuals to obtain the updated center coordinates.

[0089] Here, the updated center coordinates are the new Gaussian unit spatial positions obtained after applying this amplitude constraint. In practice, the maximum allowable adjustment amplitude of the center coordinates and the sigmoid activation function (Sigmoid) can be combined to apply a nonlinear amplitude constraint to the center coordinate residuals. Then, the constrained residual terms are incrementally added to the center coordinates in the current Gaussian parameters of the current round to obtain the updated center coordinates.

[0090] To update the rotation quaternion, the rotation quaternion residual can be multiplied by the rotation quaternion in the current Gaussian parameters of the current iteration to obtain the updated rotation quaternion. Here, the rotation quaternion is a mathematical variable used to express the rotational attitude of Gaussian elements in continuous three-dimensional space. The updated rotation quaternion is a new rotational attitude expression that incorporates attitude residual compensation.

[0091] In practice, the rotation quaternion residual is first normalized using a normalization operation, then multiplied by the rotation quaternion in the current Gaussian parameters of the current cycle, and finally normalized again to obtain the updated rotation quaternion.

[0092] To adjust the scale parameters, based on preset scale adjustment constraints, the scale parameters in the current Gaussian parameters of the current round are incrementally superimposed and updated with the scale residuals to obtain the updated scale parameters. Here, the scale adjustment constraints are a set of upper and lower limits for extreme values ​​pre-set to prevent Gaussian elements from exhibiting abnormal deformation, infinite expansion, or extreme shrinkage during the optimization process.

[0093] In practice, based on the scale adjustment constraints formed by the preset upper and lower limits of scale adjustment, the scale parameters in the current Gaussian parameters of the current round and the scale residual can be incrementally superimposed under nonlinear constraints using the S-shaped activation function and the preset scale adjustment range to obtain the updated scale parameters.

[0094] To update the Gaussian weights and semantic vectors, activation functions can be used to process the weight residuals and semantic residuals respectively, yielding updated Gaussian weights and updated semantic vectors. Here, the updated Gaussian weights and updated semantic vectors represent the optimized primitive opacity attribute and the final classification probability density expression, respectively.

[0095] In practice, the sigmoid activation function is used to process the weight residuals and directly assign values ​​to obtain the updated Gaussian weights. Additionally, the normalized exponential function is used to process the semantic residuals and directly assign values ​​to obtain the updated semantic vector.

[0096] Finally, the updated center coordinates, rotation quaternions, scale parameters, Gaussian weights, and semantic vectors obtained through the separate update mechanisms are recombined and used as the updated Gaussian parameters for the current round output.

[0097] The method provided in this invention predicts the residuals of various parameters in parallel and designs adaptive update strategies such as gated increments, quaternion multiplication, scale constraints, and direct assignment of activation maps for the spatial position, rotational orientation, size, and semantic attributes of Gaussian elements. This effectively avoids parameter divergence and geometric distortion in the Gaussian semantic field during multi-round iterative optimization and ensures the convergence quality of scene geometric continuity and semantic features with extremely stable performance.

[0098] In one embodiment, the Gaussian update subnetwork in the semantic optimization unit consists of a first linear prediction layer (Linear_du), a second linear prediction layer (Linear_dr), a third linear prediction layer (Linear_ds), a fourth linear prediction layer (Linear_dp), a fifth linear prediction layer (Linear_dv), a Sigmoid activation function, a Softmax activation function, and a parameter progressive update unit. Specifically, the first linear prediction layer (Linear_du) has a weight matrix size of 64×3, the second linear prediction layer (Linear_dr) has a weight matrix size of 64×4, the third linear prediction layer (Linear_ds) has a weight matrix size of 64×3, the fourth linear prediction layer (Linear_dp) has a weight matrix size of 64×1, and the fifth linear prediction layer (Linear_dv) has a weight matrix size of 64×1. C , C (where is the number of semantic categories, which can be 20) All of them have learnable bias vectors and are implemented using the nn.Linear function in the PyTorch library; the sigmoid activation function is implemented using the torch.sigmoid function in the PyTorch library; the softmax activation function is implemented using the torch.softmax function in the PyTorch library.

[0099] The Gaussian update subnetwork is used to update the current Gaussian parameters. The specific process includes: receiving the Gaussian fusion mixture vector through the Gaussian update subnetwork. (size is) The results are then fed into five parallel linear prediction layers for processing. Specifically, the center coordinate residuals are output after passing through the first linear prediction layer (Linear_du). (size is) After passing through the second linear prediction layer (Linear_dr), the output is a rotated quaternion residual. (size is) The output scale residuals after passing through the third linear prediction layer, Linear_ds. (size is) The weighted residuals are output after the fourth linear prediction layer, Linear_dp. (size is) After passing through the fifth linear prediction layer (Linear_dv), the semantic residual is output. (size is) ).

[0100] Next, the aforementioned center coordinate residuals can be received through the parameter asymptotic update unit. Rotational quaternion residuals Scale residuals Weighted residuals Semantic residuals Perform the following incremental update and direct assignment update operations. That is, for the updated center coordinates... Based on the progressive gating formula Perform incremental updates, where These are the center coordinates in the current Gaussian parameters for the current round, which can also be understood as the coordinates output in the previous update. The maximum allowable adjustment range for the preset center coordinates is set to 0.05 in this embodiment, and the coordinate size remains unchanged after the update. .

[0101] For the updated rotation quaternion First, the normalization operation, implemented by the `torch.nn.functional.normalize` function in the PyTorch library, is used to normalize the rotated quaternion residuals. Normalize it, then combine it with the rotation quaternion from the previous unit. (i.e., the rotated quaternions in the current Gaussian parameters) undergo quaternion multiplication, and finally normalization is performed again using the normalization operation. The size is .

[0102] For the updated scale parameters Based on the formula Perform incremental superposition updates under nonlinear constraints, with a size of ,in, Indicates the previous scale parameter; This represents the magnitude of the Gaussian scaling adjustment. and These are the upper and lower limits of the Gaussian scaling adjustment, respectively.

[0103] For the updated Gaussian weights The Sigmoid activation function is used to process the weight residuals for direct assignment updates, i.e. The size is Additionally, regarding the updated semantic vector... The semantic residuals can be processed using the Softmax activation function for direct assignment updates, i.e. The size is .

[0104] Understandably, the entire Gaussian semantic field optimization module includes (In this embodiment, there are 3) semantic optimization units. By directly using the update parameters output by the previous semantic optimization unit as the input of the next optimization unit, cascade optimization is performed in this sequential chain manner. Finally, the last semantic optimization unit outputs a converged set of Gaussian semantic field parameters. ( = ,in, Indicates the first The convergence parameter set corresponding to each Gaussian element includes the center coordinates of convergence. ( = ), size is ; convergent covariance matrix ( = ), size is Converging Gaussian weights ( = ), size is ; and convergent semantic vectors ( = ), size is ( (where 20 is the number of semantic categories) and the convergent Gaussian fusion mixture vector. ( ), size is ; Indicates the index number of the Gaussian unit. This represents the total number of Gaussian elements.

[0105] Based on any of the above embodiments, in step 170, calculating the fused semantic vector of each point in the original input point cloud based on the Gaussian semantic field parameter set includes: The convergent center coordinates in the Gaussian semantic field parameter set are mapped to a pre-divided three-dimensional voxel space to establish a three-dimensional spatial index. For the target point in the original input point cloud, a local candidate set is retrieved in the local neighborhood where the target point is located based on the three-dimensional spatial index; Calculate the Mahalanobis distance between the target point and the center coordinates of each Gaussian element in the local candidate set; Using a preset semantic truncation threshold, the Mahalanobis distance is filtered to obtain a set of Gaussian elements that meet the conditions. For the set of Gaussian elements that meet the conditions, a probability density decay term is calculated. The probability density decay term is multiplied by the Gaussian weights and the semantic vector and then accumulated to obtain the fused semantic vector of the target point.

[0106] Specifically, firstly, the convergent center coordinates of the Gaussian semantic field parameter set are mapped to a pre-defined 3D voxel space to establish a 3D spatial index. Here, the convergent center coordinates of the Gaussian semantic field parameter set refer to the Gaussian element position vectors that, after multiple rounds of iterative optimization, no longer change drastically and are precisely aligned with the physical geometric surface. The pre-defined 3D voxel space refers to a regular grid system formed by discretizing the continuous 3D physical working environment according to fixed physical dimensions. Therefore, this 3D spatial index is a mapping data structure between the voxel grid and the corresponding Gaussian element list, established to accelerate spatial retrieval queries.

[0107] In practice, the three-dimensional continuous physical space can be divided into a three-dimensional voxel space with a set side length and a specific resolution. Then, the convergent center coordinates of the Gaussian semantic field parameter set are mapped to the corresponding voxel grid in the three-dimensional voxel space, thereby establishing a three-dimensional spatial index corresponding to the voxel and the Gaussian element list.

[0108] Then, for the target point in the original input point cloud, a local candidate set is retrieved within its local neighborhood based on a 3D spatial index. Here, the target point refers to the discrete 3D point in the original input point cloud whose predicted class probability is currently to be calculated. The local neighborhood refers to a 3D spatial region extending outwards from the voxel containing the target point. Therefore, the local candidate set is the set of all Gaussian units falling within this spatial region that may have a semantic impact on the target point.

[0109] In practice, the influence radius of a single Gaussian element can be calculated by determining the maximum eigenvalue of each covariance matrix in the converged covariance matrix and combining it with a preset scaling factor. This allows for the determination of the range span of the voxel neighborhood expansion. For any target point in the original input point cloud, the corresponding local neighborhood is accessed based on this range span, centered on the voxel it belongs to. All Gaussian elements contained within this neighborhood are then merged to generate a local candidate set.

[0110] Furthermore, the Mahalanobis distance between the target point and the center coordinates of each Gaussian element in the local candidate set is calculated. Then, using a preset semantic truncation threshold, the Mahalanobis distances are filtered to obtain a set of Gaussian elements that meet the conditions.

[0111] Here, Mahalanobis distance is an isotropic spatial distance metric that takes into account the covariance of Gaussian elements. The semantic truncation threshold is an empirical upper limit on the distance used to filter out Gaussian elements that do not match in shape, pose, or position.

[0112] In practice, the Mahalanobis distance between the target point and the center coordinates of each Gaussian element in the local candidate set can be calculated one by one. Then, the calculated Mahalanobis distances are numerically judged using a preset semantic truncation threshold. Gaussian elements whose Mahalanobis distance is greater than the semantic truncation threshold are directly eliminated, while Gaussian elements whose Mahalanobis distance is less than or equal to the threshold are retained, forming the final set of Gaussian elements that meet the conditions.

[0113] Finally, for the set of Gaussian elements that meet the conditions, a probability density decay term is calculated. This term is then multiplied by the Gaussian weights and the semantic vector, and the results are summed to obtain the fused semantic vector of the target point. Here, the probability density decay term is an exponentially decreasing weight coefficient calculated based on Mahalanobis distance, used to simulate the spatial decay characteristics of the Gaussian distribution. The fused semantic vector is a comprehensive category probability expression calculated for the target point, integrating the continuous probability density distributions of multiple effective Gaussian elements in the neighborhood.

[0114] In practice, for each Gaussian element in the set of Gaussian elements that meet the conditions, an exponentially decaying probability density term is calculated based on its distance from the target point. Subsequently, this probability density decay term is multiplied by the converged Gaussian weights and the converged semantic vector, and after being divided by the relevant normalization coefficients, it is accumulated one by one to finally obtain the fused semantic vector corresponding to the target point.

[0115] The method provided in this invention greatly improves the retrieval efficiency of Gaussian primitives by constructing a voxelized three-dimensional spatial index, and introduces Mahalanobis distance and semantic truncation threshold to strictly screen effective primitives. It combines probability density decay term to discretize and accumulate continuous semantics, i.e. semantic splashing, to achieve efficient and accurate mapping of continuous Gaussian semantic field to discrete point cloud, and significantly ensures the clarity of semantic segmentation boundary at point cloud edge and complex geometric structure.

[0116] Based on any of the above embodiments, in step 170, the semantic segmentation result of the original input point cloud is obtained based on the fused semantic vectors of the points, including: The fused semantic vectors of the various points are concatenated to form the overall semantic probability tensor; For the overall semantic probability tensor, extract the category index corresponding to the maximum value along the category dimension, output the semantic prediction value vector, and use the semantic prediction value vector as the semantic segmentation result of the original input point cloud.

[0117] Specifically, firstly, the fused semantic vectors of each point are concatenated to form an overall semantic probability tensor. Here, the overall semantic probability tensor is a global feature matrix formed by combining the predicted probabilities of all discrete points in the scene, and its dimension is usually the total number of point clouds multiplied by the total number of semantic categories.

[0118] In practice, for each point in the original input point cloud, its corresponding fused semantic vector is extracted, and all these vectors are spliced ​​and aggregated in a specific dimension according to the spatial or input order of the point cloud to form a semantic probability tensor covering the entire input point cloud.

[0119] Then, for the overall semantic probability tensor, the category index corresponding to the maximum value is extracted along the category dimension, and the semantic prediction value vector is output. The semantic prediction value vector is used as the semantic segmentation result of the original input point cloud.

[0120] Here, the category dimension refers to the data dimension in the probability tensor that represents different semantic classification labels. The category index is the position number of the element with the highest probability value in this dimension, which directly corresponds to the specific semantic category, such as desktop, chair, etc. Here, the semantic prediction vector is a one-dimensional data sequence containing the final discrete classification labels of all points.

[0121] In practice, the system receives the semantic probability tensor of the whole system and uses tensor processing functions, such as the argmax operation, to traverse along the category dimension and extract the category index with the highest probability value. Then, it combines all the extracted category indices and outputs a semantic prediction value vector. This vector is then used as the semantic segmentation result of the original input point cloud as the final output of the robot vision system for downstream grasping or navigation tasks.

[0122] The method provided in this invention globally concatenates the continuous probability density distribution tensors of each point and efficiently extracts the maximum probability index along the category dimension. This enables the rapid and accurate transformation of the continuous probability features predicted by the deep network into discrete scene classification labels that can be directly understood by the robot. It bridges the final step from the continuous Gaussian semantic field to the actual application output on the edge, ensuring the complete closed loop and reliable output of the point cloud semantic segmentation task.

[0123] In one embodiment, the Gaussian semantic field splashing module consists of a spatial index construction unit, a local candidate set generation unit, a Mahalanobis distance filtering unit, a semantic splashing accumulation unit, and a category index extraction unit. Original input point cloud Total points; This represents the total number of Gaussian elements; The number of semantic categories is 20 in this embodiment, for example. The matrix inversion and matrix multiplication in the Mahalanobis distance filtering unit are implemented using the `torch.inverse` and `torch.matmul` functions from the PyTorch library, respectively; the exponentiation operation in the semantic splash accumulation unit is implemented using the `torch.exp` function from the PyTorch library.

[0124] The Gaussian semantic field splashing module is mainly used to execute step 170. The specific method flow includes: receiving the original input point cloud through the Gaussian semantic field splashing module. (size is) and the convergent Gaussian semantic field parameter set First, the three-dimensional space can be divided into voxel sides of length using spatial indexing units. (e.g., 10mm), resolution is In a three-dimensional voxel space (e.g., 100×100×100), the convergent center coordinates Mapping to the corresponding voxels in 3D voxel space, a 3D spatial index of the "voxel-Gaussian list" is established. Then, the convergent covariance matrix can be calculated using local candidate set generation units. ( = The largest eigenvalue of each covariance matrix in ) This can be achieved using the `torch.linalg.eigvalsh` function in the PyTorch library. Combined with a scaling factor... For example, if the value is 3, calculate the influence radius of a single Gaussian element. And then through the formula Find the range span of the voxel neighborhood expansion ,in, This indicates rounding up, implemented by the torch.ceil function in the PyTorch library.

[0125] For the original input point cloud The first in Points With its voxel as the center, the access size is The local neighborhood is used to merge all Gaussian elements contained within that neighborhood into a local candidate set. Subsequently, points can be calculated using Mahalanobis distance to filter the units. With the center coordinates of convergence The central coordinates of each Gaussian element Mahalanobis distance and remove The basic unit, here, For a given semantic truncation threshold, for example, 1, primitive pairs that satisfy the condition are retained. Beneficial Gaussian set .

[0126] Then, semantic splashing accumulation units can be used to target the set. In the Gaussian element, calculate the probability density decay term. , and and Multiply and then divide by the normalization coefficient. Then, the points are accumulated one by one to obtain the original input point cloud. The first in Points Fusion semantic vector ,all The semantic vectors of each point are concatenated to form the overall semantic probability tensor, with a size of [size missing]. .

[0127] Finally, the semantic probability tensor of the whole can be received through the category index extraction unit, along the semantic category dimension. Execute the `torch.argmax` operation in the PyTorch library to extract the category index corresponding to the maximum value, and output the size as follows: The semantic prediction vector is the final output of the robot point cloud semantic segmentation network.

[0128] Based on any of the above embodiments, step 110 includes: The original input point cloud is downsampled multiple times to obtain multiple sets of downsampled point clouds with different resolutions; Feature extraction is performed on each downsampled point cloud set to generate multi-scale point cloud features at different scales. The multi-scale point cloud features of different scales are upsampled and mapped back to the spatial location of the original input point cloud to obtain multiple upsampled point cloud features. The multi-scale point cloud features are obtained by stitching together and performing channel dimensionality reduction on multiple upsampled point cloud features.

[0129] Specifically, firstly, the original input point cloud is downsampled multiple times to obtain multiple sets of downsampled point clouds at different resolutions. In practice, iterative Farthest Point Sampling (FPS) operations are performed on the original input point cloud along the dimension containing the number of points. In each iteration, based on several preset downsampling factors, such as 8x, 16x, and 32x, the target number of points to be sampled is calculated, thus sequentially obtaining multiple sets of downsampled point clouds at different resolutions.

[0130] Then, feature extraction is performed on each downsampled point cloud set to generate multi-scale point cloud features at different scales. In specific implementation, the various downsampled point cloud sets with different resolutions generated above can be sequentially fed into the feature extraction backbone network for processing, extracting geometric and contextual information at each resolution layer by layer, thereby generating corresponding multi-scale point cloud features at multiple different scales.

[0131] Next, the multi-scale point cloud features at different scales are upsampled and mapped back to the spatial positions of the original input point cloud, resulting in multiple upsampled point cloud features. Specifically, spatial interpolation algorithms such as Inverse Distance Weighting (IDW) can be used to interpolate and map the generated multi-scale point cloud features at different scales back to the true spatial positions of each point in the original input point cloud based on the inverse relationship of spatial distance, thus obtaining multiple high-resolution upsampled point cloud features with the same number of points as the original input point cloud.

[0132] Finally, multiple upsampled point cloud features are concatenated and subjected to channel dimensionality reduction to obtain multi-scale point cloud features. Specifically, these multiple upsampled point cloud features are concatenated along the last channel dimension to form a high-dimensional concatenated multi-scale feature. This concatenated multi-scale feature is then fed into a linear layer with learnable biases for channel dimensionality reduction and deep feature fusion, ultimately outputting the dimensionality-reduced fused multi-scale point cloud feature.

[0133] The method provided in this invention combines farthest point sampling, multi-branch feature extraction, and inverse distance weighted interpolation and channel dimensionality reduction techniques to effectively mine the hierarchical geometric topology of point clouds at different spatial resolutions, and finally aggregates and generates multi-scale point cloud features that combine macroscopic scene semantics and microscopic local details.

[0134] In one embodiment, the multi-scale point cloud feature extractor consists of a farthest point sampling layer FPS_1, a feature extraction backbone network PointNet, and a feature fusion unit. The farthest point sampling layer FPS_1 receives the original input point cloud with a size of M×3 (number of points × coordinate dimension). Perform iterative maximum distance retrieval along the dimension containing the points, with the target point count set sequentially as follows: , , Where L1, L2, and L3 are preset downsampling factors, for example, L1=8, L2=16, L3=32. This indicates rounding down to the nearest integer, resulting in the set of downsampled point clouds. S Then, the downsampled point cloud set can be... S The point cloud data is sequentially fed into the PointNet feature extraction backbone network for feature extraction, generating multi-scale point cloud features. (Feature tensor size is) ), (Feature tensor size is) ), (Feature tensor size is) ).

[0135] Next, the multi-scale point cloud features , , The data is fed into the feature fusion unit, where an inverse distance-weighted interpolation algorithm is first used to combine the multi-scale point cloud features. , , Each of the M upsampled point cloud features is mapped back to its spatial location, resulting in three features of size M×256. Then, the `torch.cat` function from the PyTorch library is used to concatenate these three upsampled point cloud features along the last channel dimension, yielding a concatenated multi-scale feature of size M×768. Subsequently, this concatenated multi-scale feature undergoes channel dimensionality reduction and deep feature fusion through a feature fusion linear layer, ultimately outputting a fused multi-scale point cloud feature of size M×256. The feature fusion linear layer here has 768 input channels and 256 output channels, implemented using the nn.Linear function in the PyTorch library, and includes a learnable bias.

[0136] In one embodiment, the Gaussian semantic field initialization module utilizes the farthest point sampling algorithm to initialize the original input point cloud. Mid-sampling Points (in this embodiment of the invention) ) as the central coordinate of the Gaussian unit The size is The initial covariance matrix is ​​generated by calculating the local point cloud covariance of each center point using the K-nearest neighbor algorithm. The size is It includes 3 scaling parameters and 3 rotation parameters. Gaussian weights. The size is Further random initialization of semantic vectors. (size , (The number of semantic categories, for example, 20) and Gaussian mixture vectors (size Then, the original input point cloud... The initial center coordinates of the Gaussian element Initial covariance matrix Gaussian weights semantic vectors Gaussian mixture vector and fused multi-scale point cloud features It is then fed into the Gaussian semantic field optimization module for subsequent operations.

[0137] Based on any of the above embodiments, this invention provides a point cloud semantic segmentation model to execute the above-described robot 3D point cloud semantic segmentation method and achieve point cloud semantic segmentation. During the training phase of the point cloud semantic segmentation model, the point cloud semantic segmentation network used to construct the model can be trained based on pre-constructed training samples and a pre-defined loss function to obtain the specific weight parameters of each module. The training method includes training using ScanNet v2 and TO-Scene datasets with 3D semantic labels for indoor or desktop scenes. During training, a 3D point cloud sequence containing coordinates and colors is obtained for each scene and registered and aligned as the network input data, with the corresponding point-by-point semantic labels serving as supervision signals. The network training process uses the Adam optimizer. The initial learning rate of the network weights is set to 0.001, and the weight decay coefficient is set to 0.0001. The maximum number of training epochs is set to 200. Batch training is used, with the batch size parameter for a single training session set to 8.

[0138] Multi-task loss function during training The mathematical expression is shown below, which aims to simultaneously ensure the bidirectional consistency of semantics and geometry during the optimization process of Gaussian semantic fields: ; In the formula, the weighting coefficient and To balance the hyperparameters of the loss, the embodiments of the present invention can be set to 1.0 and 1.0. For semantic supervision loss; The geometric bijective reconstruction loss designed for this invention; The Lovász-Softmax loss is used to directly optimize the intersection-union ratio (IU) of discrete classes, improving the segmentation accuracy of long-tail classes.

[0139] in, For semantic supervision loss, the standard cross-entropy loss function is used to constrain the predicted probability distribution of the Gaussian semantic field splash output to the true label, which can be expressed by the following formula, as shown in the following equation: ; In the formula, This is an indicator function; it outputs 1 if the condition is met, and 0 otherwise. Represents the original input point cloud The first in Points The fused semantic vector has a feature dimension equal to the number of semantic categories. In this embodiment , for The true semantic tags; This represents the total number of points in the original input point cloud. This indicates the index number of the original input point cloud.

[0140] The geometric bijective reconstruction loss is used to maintain the bidirectional fit between Gaussian elements and the local geometry of the point cloud, and to suppress the disordered positional drift of Gaussian centers. Its expression is: ; In the formula, Indicates the first The center coordinates of each Gaussian element; Represents the first point in the original input point cloud. The three-dimensional spatial position of the point; the other parameters are the same as explained above, and will not be repeated here.

[0141] Therefore, the trained robot point cloud semantic segmentation network can be applied to 3D point cloud segmentation methods. In the application phase, firstly, the point cloud of the current scene, i.e., the original input point cloud, is acquired using a 3D vision sensor mounted on the robot. Here, the 3D vision sensor can be a Mech-Eye PRO S. Then, the point cloud of the current scene is input into the trained robot point cloud semantic segmentation network, outputting the robot's 3D point cloud semantic segmentation result. Specifically, this includes: inputting the point cloud of the current scene into the multi-scale point cloud feature extractor of the trained robot point cloud semantic segmentation network to extract the fused multi-scale point cloud features; inputting the point cloud of the current scene into the Gaussian semantic field initialization module of the trained robot point cloud semantic segmentation network to extract the initial center coordinates of Gaussian primitives, the initial covariance matrix, Gaussian weights, semantic vectors, and Gaussian mixture vectors; inputting the point cloud of the current scene, the fused multi-scale point cloud features, the initial center coordinates of Gaussian primitives, the initial covariance matrix, Gaussian weights, semantic vectors, and Gaussian mixture vectors into the Gaussian semantic field optimization module of the trained robot point cloud semantic segmentation network to obtain a converged set of Gaussian semantic field parameters; and inputting the point cloud of the current scene and the converged set of Gaussian semantic field parameters into the Gaussian semantic field splashing module of the trained robot point cloud semantic segmentation network for processing, and outputting the robot's 3D point cloud semantic segmentation result.

[0142] It should be noted that the method provided in this embodiment of the invention, by constructing a Gaussian semantic field, uniformly expresses geometric and semantic information in a continuous three-dimensional space. Furthermore, multiple cascaded semantic optimization units are designed, and through a cross-attention mechanism with a distance gating function, adaptive fusion of discrete multi-scale point cloud features and continuous Gaussian features is achieved, eliminating feature interference caused by positional deviations of associated points. In addition, a geometric bijective reconstruction loss is introduced during network training, establishing bidirectional geometric consistency between the point cloud and the Gaussian metacenter, enabling stable and high-precision three-dimensional point cloud semantic segmentation in complex scenes, providing a perceptual foundation for robot hand-eye manipulation.

[0143] Based on any of the above embodiments Figure 2 This is a schematic diagram of the structure of the robot 3D point cloud semantic segmentation device provided by the present invention, as shown below. Figure 2 As shown, the device includes: Point cloud feature extraction unit 210 extracts multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene; The Gaussian parameter feature extraction unit 220 uses the Gaussian parameters obtained by initializing Gaussian primitives based on the original input point cloud as the current Gaussian parameters of the current round, and extracts the Gaussian query features corresponding to the current Gaussian parameters. The associated point cloud feature extraction unit 230 retrieves the nearest neighbor points of the center coordinates of each Gaussian element in the original input point cloud, and calculates the associated point cloud features corresponding to the nearest neighbor points based on the multi-scale point cloud features. The distance gating construction unit 240 calculates the original attention score map based on the Gaussian query features and the associated point cloud features, and constructs a distance gating tensor based on the spatial distance between the center coordinates and the nearest neighbor points. The weighted adjustment unit 250 adjusts the weights of the original attention score map based on the distance gate tensor to obtain an adjusted attention map, and obtains a Gaussian fusion mixing vector based on the adjusted attention map. The Gaussian parameter iterative update unit 260 updates the current Gaussian parameters based on the Gaussian fusion mixing vector to obtain the updated Gaussian parameters. The updated Gaussian parameters are used as the current Gaussian parameters in the next round. The unit returns to the step of extracting the Gaussian query features corresponding to the current Gaussian parameters. The unit iteratively executes the steps until the preset cutoff condition is met. The updated Gaussian parameters in the last round are used as the Gaussian semantic field parameter set. The semantic segmentation unit 270 calculates the fused semantic vector of each point in the original input point cloud based on the Gaussian semantic field parameter set, and obtains the semantic segmentation result of the original input point cloud based on the fused semantic vector of each point.

[0144] The device provided in this invention constructs a Gaussian semantic field to uniformly express geometric and semantic information in a continuous three-dimensional space, designs a multi-round cascaded optimization mechanism to promote deep interaction between discrete multi-scale point cloud features and continuous Gaussian primitives, and achieves adaptive feature fusion with a distance gating mechanism. This not only effectively establishes an accurate spatial correspondence between the geometric structure of the point cloud and the continuous Gaussian properties, but also significantly improves the robustness of the robot's three-dimensional perception when facing object edges, small structures and occluded areas in complex scenes, and finally outputs stable and high-precision point cloud semantic segmentation results.

[0145] Based on any of the above embodiments, the associated point cloud feature extraction unit is specifically used for: Retrieve the nearest neighbor points and generate a nearest neighbor index tensor; Based on the nearest neighbor index tensor, the multi-scale point cloud features are sliced ​​and recombined to extract the multi-scale point cloud feature vectors corresponding to the nearest neighbor points, and the local multi-scale point cloud feature slices are reconstructed. Obtain the spatial coordinates of the nearest neighbor points and extract the position vectors of the nearest neighbor points; After concatenating the local multi-scale point cloud feature slices with the nearest neighbor point location vectors, feature mapping processing is performed on the concatenated vectors to obtain the associated point cloud features corresponding to the nearest neighbor points.

[0146] Based on any of the above embodiments, the distance gating construction unit is specifically used for: Calculate the Euclidean distance between the center coordinates of each Gaussian element and the nearest neighbor index tensor; The Euclidean distance is calculated using a preset first learnable parameter and a second learnable parameter to obtain the distance gate tensor; The weighted adjustment unit is specifically used for: The original attention score map is normalized, and the normalized original attention score map is multiplied element-wise with the distance gating tensor to obtain the adjusted attention map.

[0147] Based on any of the above embodiments, the distance gating construction unit is further specifically used for: Feature extraction is performed on the Gaussian query features, and the spatial dimension is expanded to obtain the query tensor; Feature mapping is performed on the associated point cloud features to obtain key tensors and value tensors respectively; Calculate the inner product of the query tensor and the transpose of the key tensor, and divide the inner product by a preset scaling factor to obtain the original attention score map.

[0148] Based on any of the above embodiments, the Gaussian parameter feature extraction unit is specifically used for: Map the center coordinates in the current Gaussian parameters to the set three-dimensional voxel space to obtain the voxel index; The voxel index and the Gaussian mixture vector in the current Gaussian parameters are mapped and sparse convolution is performed to obtain the first Gaussian feature map. After mapping the current Gaussian parameters to a unified feature dimension, element-wise addition is performed to obtain the second Gaussian feature map. The first Gaussian feature map and the second Gaussian feature map are concatenated along the channel dimension to obtain the third Gaussian feature map; The third Gaussian feature map is fused and processed to obtain the Gaussian query feature corresponding to the current Gaussian parameter.

[0149] Based on any of the above embodiments, the Gaussian parameter iteration update unit is specifically used for: The Gaussian fusion mixing vector is processed to predict the center coordinate residual, rotation quaternion residual, scale residual, weight residual and semantic residual; The center coordinates in the current Gaussian parameters are updated by gated incremental update with the center coordinate residuals to obtain the updated center coordinates. The rotated quaternion residual is multiplied by the rotated quaternion in the current Gaussian parameters to obtain the updated rotated quaternion; Based on the preset scale adjustment constraints, the scale parameters in the current Gaussian parameters are incrementally superimposed and updated with the scale residuals to obtain the updated scale parameters. The weight residuals and semantic residuals are processed using activation functions to obtain updated Gaussian weights and updated semantic vectors. The updated center coordinates, rotation quaternions, scale parameters, Gaussian weights, and semantic vectors are used as the updated Gaussian parameters.

[0150] Based on any of the above embodiments, the semantic segmentation unit is specifically used for: The convergent center coordinates in the Gaussian semantic field parameter set are mapped to a pre-divided three-dimensional voxel space to establish a three-dimensional spatial index. For the target point in the original input point cloud, a local candidate set is retrieved in the local neighborhood where the target point is located based on the three-dimensional spatial index; Calculate the Mahalanobis distance between the target point and the center coordinates of each Gaussian element in the local candidate set; Using a preset semantic truncation threshold, the Mahalanobis distance is filtered to obtain a set of Gaussian elements that meet the conditions. For the set of Gaussian elements that meet the conditions, a probability density decay term is calculated. The probability density decay term is multiplied by the Gaussian weights and the semantic vector and then accumulated to obtain the fused semantic vector of the target point.

[0151] Based on any of the above embodiments, the semantic segmentation unit is further specifically used for: The fused semantic vectors of the various points are concatenated to form the overall semantic probability tensor; For the overall semantic probability tensor, extract the category index corresponding to the maximum value along the category dimension, output the semantic prediction value vector, and use the semantic prediction value vector as the semantic segmentation result of the original input point cloud.

[0152] Based on any of the above embodiments, the point cloud feature extraction unit is specifically used for: The original input point cloud is downsampled multiple times to obtain multiple sets of downsampled point clouds with different resolutions; Feature extraction is performed on each downsampled point cloud set to generate multi-scale point cloud features at different scales. The multi-scale point cloud features of different scales are upsampled and mapped back to the spatial location of the original input point cloud to obtain multiple upsampled point cloud features. The multi-scale point cloud features are obtained by stitching together and performing channel dimensionality reduction on multiple upsampled point cloud features.

[0153] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a robot 3D point cloud semantic segmentation method. This method includes: extracting multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene; using the Gaussian parameters obtained by initializing Gaussian units based on the original input point cloud as the current Gaussian parameters for the current round; and extracting Gaussian query features corresponding to the current Gaussian parameters; retrieving the nearest neighbor points of the center coordinates of each Gaussian unit in the original input point cloud; and calculating the associated point cloud features corresponding to the nearest neighbor points based on the multi-scale point cloud features; calculating the original attention score map based on the Gaussian query features and the associated point cloud features; and constructing an original attention score map based on the spatial distance between the center coordinates and the nearest neighbor points. A distance-gated tensor is constructed; the original attention score map is weighted based on the distance-gated tensor to obtain an adjusted attention map, and a Gaussian fusion mixing vector is obtained based on the adjusted attention map; the current Gaussian parameters are updated based on the Gaussian fusion mixing vector to obtain updated Gaussian parameters, and the updated Gaussian parameters are used as the current Gaussian parameters in the next round. The step of extracting the Gaussian query features corresponding to the current Gaussian parameters is returned to the previous round until the updated Gaussian parameters in the last round are used as the Gaussian semantic field parameter set; based on the Gaussian semantic field parameter set, the fused semantic vector of each point in the original input point cloud is calculated, and the semantic segmentation result of the original input point cloud is obtained based on the fused semantic vector of each point.

[0154] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot 3D point cloud semantic segmentation method provided by the above methods. The method includes: extracting multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene; using the Gaussian parameters obtained by initializing Gaussian units based on the original input point cloud as the current Gaussian parameters for the current round; and extracting the Gaussian query features corresponding to the current Gaussian parameters; retrieving the nearest neighbor points of the center coordinates of each Gaussian unit in the original input point cloud; and calculating the associated point cloud features corresponding to the nearest neighbor points based on the multi-scale point cloud features; and calculating the original annotation based on the Gaussian query features and the associated point cloud features. The attention score map is generated, and a distance-gated tensor is constructed based on the spatial distance between the center coordinates and the nearest neighbor points. The original attention score map is then weighted based on the distance-gated tensor to obtain an adjusted attention map, and a Gaussian fusion mixing vector is obtained based on the adjusted attention map. The current Gaussian parameters are updated based on the Gaussian fusion mixing vector to obtain updated Gaussian parameters. The updated Gaussian parameters are used as the current Gaussian parameters for the next round, and the process of extracting the Gaussian query features corresponding to the current Gaussian parameters is repeated until the updated Gaussian parameters in the last round are used as the Gaussian semantic field parameter set. Based on the Gaussian semantic field parameter set, the fused semantic vector of each point in the original input point cloud is calculated, and the semantic segmentation result of the original input point cloud is obtained based on the fused semantic vector of each point.

[0156] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the robot 3D point cloud semantic segmentation method provided by the above methods. This method includes: extracting multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene; using Gaussian parameters obtained by initializing Gaussian units based on the original input point cloud as the current Gaussian parameters for the current round; and extracting Gaussian query features corresponding to the current Gaussian parameters; retrieving the nearest neighbor points of the center coordinates of each Gaussian unit in the original input point cloud; and calculating associated point cloud features corresponding to the nearest neighbor points based on the multi-scale point cloud features; calculating an original attention score map based on the Gaussian query features and the associated point cloud features; and calculating an original attention score map based on the Gaussian query features and the associated point cloud features. A distance-gated tensor is constructed by the spatial distance between the center coordinates and the nearest neighbor points. The original attention score map is then weighted based on this distance-gated tensor to obtain an adjusted attention map. A Gaussian fusion mixing vector is then obtained based on the adjusted attention map. The current Gaussian parameters are updated based on this Gaussian fusion mixing vector to obtain updated Gaussian parameters. These updated Gaussian parameters are then used as the current Gaussian parameters for the next round. The process of extracting the Gaussian query features corresponding to the current Gaussian parameters is repeated until the updated Gaussian parameters from the last round are used as the Gaussian semantic field parameter set. Based on the Gaussian semantic field parameter set, the fused semantic vector of each point in the original input point cloud is calculated. The semantic segmentation result of the original input point cloud is then obtained based on the fused semantic vectors of each point.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic segmentation method for 3D point clouds of a robot, characterized in that, include: Extract multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene; The Gaussian parameters obtained by initializing Gaussian primitives based on the original input point cloud are used as the current Gaussian parameters for the current round, and the Gaussian query features corresponding to the current Gaussian parameters are extracted. The center coordinates of each Gaussian element are retrieved to the nearest neighbor points in the original input point cloud, and the associated point cloud features corresponding to the nearest neighbor points are calculated based on the multi-scale point cloud features. Based on the Gaussian query features and the associated point cloud features, the original attention score map is calculated, and a distance gating tensor is constructed based on the spatial distance between the center coordinates and the nearest neighbor points. The original attention score map is weighted based on the distance gate tensor to obtain an adjusted attention map, and a Gaussian fusion mixing vector is obtained based on the adjusted attention map. The current Gaussian parameters are updated based on the Gaussian fusion mixing vector to obtain the updated Gaussian parameters. The updated Gaussian parameters are used as the current Gaussian parameters in the next round. The step of extracting the Gaussian query features corresponding to the current Gaussian parameters is returned and executed iteratively until the preset cutoff condition is met. The updated Gaussian parameters in the last round are used as the Gaussian semantic field parameter set. Based on the Gaussian semantic field parameter set, the fused semantic vector of each point in the original input point cloud is calculated, and the semantic segmentation result of the original input point cloud is obtained based on the fused semantic vector of each point.

2. The robot 3D point cloud semantic segmentation method according to claim 1, characterized in that, The calculation of the associated point cloud features corresponding to the nearest points based on the multi-scale point cloud features includes: Retrieve the nearest neighbor points and generate a nearest neighbor index tensor; Based on the nearest neighbor index tensor, the multi-scale point cloud features are sliced ​​and recombined to extract the multi-scale point cloud feature vectors corresponding to the nearest neighbor points, and the local multi-scale point cloud feature slices are reconstructed. Obtain the spatial coordinates of the nearest neighbor points and extract the position vectors of the nearest neighbor points; After concatenating the local multi-scale point cloud feature slices with the nearest neighbor point location vectors, feature mapping processing is performed on the concatenated vectors to obtain the associated point cloud features corresponding to the nearest neighbor points.

3. The robot 3D point cloud semantic segmentation method according to claim 2, characterized in that, The distance-gated tensor constructed based on the spatial distance between the center coordinates and the nearest neighbor points includes: Calculate the Euclidean distance between the center coordinates of each Gaussian element and the nearest neighbor index tensor; The Euclidean distance is calculated using a preset first learnable parameter and a second learnable parameter to obtain the distance-gated tensor.

4. The robot 3D point cloud semantic segmentation method according to any one of claims 1 to 3, characterized in that, The calculation of the original attention score map based on the Gaussian query features and the associated point cloud features includes: Feature extraction is performed on the Gaussian query features, and the spatial dimension is expanded to obtain the query tensor; Feature mapping is performed on the associated point cloud features to obtain key tensors and value tensors respectively; Calculate the inner product of the query tensor and the transpose of the key tensor, and divide the inner product by a preset scaling factor to obtain the original attention score map.

5. The robot 3D point cloud semantic segmentation method according to any one of claims 1 to 3, characterized in that, The step of extracting the Gaussian query features corresponding to the current Gaussian parameters includes: Map the center coordinates in the current Gaussian parameters to the set three-dimensional voxel space to obtain the voxel index; The voxel index and the Gaussian mixture vector in the current Gaussian parameters are mapped and sparse convolution is performed to obtain the first Gaussian feature map. After mapping the current Gaussian parameters to a unified feature dimension, element-wise addition is performed to obtain the second Gaussian feature map. The first Gaussian feature map and the second Gaussian feature map are concatenated along the channel dimension to obtain the third Gaussian feature map; The third Gaussian feature map is fused and processed to obtain the Gaussian query feature corresponding to the current Gaussian parameter.

6. The robot 3D point cloud semantic segmentation method according to any one of claims 1 to 3, characterized in that, The step of updating the current Gaussian parameters based on the Gaussian fusion mixing vector to obtain the updated Gaussian parameters includes: The Gaussian fusion mixing vector is processed to predict the center coordinate residual, rotation quaternion residual, scale residual, weight residual and semantic residual; The center coordinates in the current Gaussian parameters are updated by gated incremental update with the center coordinate residuals to obtain the updated center coordinates. The rotated quaternion residual is multiplied by the rotated quaternion in the current Gaussian parameters to obtain the updated rotated quaternion; Based on the preset scale adjustment constraints, the scale parameters in the current Gaussian parameters are incrementally superimposed and updated with the scale residuals to obtain the updated scale parameters. The weight residuals and semantic residuals are processed using activation functions to obtain updated Gaussian weights and updated semantic vectors. The updated center coordinates, rotation quaternions, scale parameters, Gaussian weights, and semantic vectors are used as the updated Gaussian parameters.

7. The robot 3D point cloud semantic segmentation method according to any one of claims 1 to 3, characterized in that, The step of calculating the fused semantic vector of each point in the original input point cloud based on the Gaussian semantic field parameter set includes: The convergent center coordinates in the Gaussian semantic field parameter set are mapped to a pre-divided three-dimensional voxel space to establish a three-dimensional spatial index. For the target point in the original input point cloud, a local candidate set is retrieved in the local neighborhood where the target point is located based on the three-dimensional spatial index; Calculate the Mahalanobis distance between the target point and the center coordinates of each Gaussian element in the local candidate set; Using a preset semantic truncation threshold, the Mahalanobis distance is filtered to obtain a set of Gaussian elements that meet the conditions. For the set of Gaussian elements that meet the conditions, a probability density decay term is calculated. The probability density decay term is multiplied by the Gaussian weights and the semantic vector and then accumulated to obtain the fused semantic vector of the target point.

8. The robot 3D point cloud semantic segmentation method according to any one of claims 1 to 3, characterized in that, The semantic segmentation result of the original input point cloud obtained by fusing the semantic vectors of each point includes: The fused semantic vectors of the various points are concatenated to form the overall semantic probability tensor; For the overall semantic probability tensor, extract the category index corresponding to the maximum value along the category dimension, output the semantic prediction value vector, and use the semantic prediction value vector as the semantic segmentation result of the original input point cloud.

9. The robot 3D point cloud semantic segmentation method according to any one of claims 1 to 3, characterized in that, The extraction of multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene includes: The original input point cloud is downsampled multiple times to obtain multiple sets of downsampled point clouds with different resolutions; Feature extraction is performed on each downsampled point cloud set to generate multi-scale point cloud features at different scales. The multi-scale point cloud features of different scales are upsampled and mapped back to the spatial location of the original input point cloud to obtain multiple upsampled point cloud features. The multi-scale point cloud features are obtained by stitching together and performing channel dimensionality reduction on multiple upsampled point cloud features.

10. A robot 3D point cloud semantic segmentation device, characterized in that, include: The point cloud feature extraction unit extracts multi-scale point cloud features based on the original input point cloud acquired by the robot in the current scene; The Gaussian parameter feature extraction unit takes the Gaussian parameters obtained by initializing Gaussian primitives based on the original input point cloud as the current Gaussian parameters of the current round and extracts the Gaussian query features corresponding to the current Gaussian parameters. The associated point cloud feature extraction unit retrieves the nearest neighbor points of the center coordinates of each Gaussian element in the original input point cloud, and calculates the associated point cloud features corresponding to the nearest neighbor points based on the multi-scale point cloud features. The distance gating construction unit calculates the original attention score map based on the Gaussian query features and the associated point cloud features, and constructs a distance gating tensor based on the spatial distance between the center coordinates and the nearest neighbor points. The weighted adjustment unit adjusts the weights of the original attention score map based on the distance gate tensor to obtain the adjusted attention map, and obtains the Gaussian fusion mixing vector based on the adjusted attention map. The Gaussian parameter iterative update unit updates the current Gaussian parameters based on the Gaussian fusion mixing vector to obtain the updated Gaussian parameters. The updated Gaussian parameters are used as the current Gaussian parameters in the next round. The unit returns to the step of extracting the Gaussian query features corresponding to the current Gaussian parameters. The unit iteratively executes the steps until the preset cutoff condition is met. The updated Gaussian parameters in the last round are used as the Gaussian semantic field parameter set. The semantic segmentation unit calculates the fused semantic vector of each point in the original input point cloud based on the Gaussian semantic field parameter set, and obtains the semantic segmentation result of the original input point cloud based on the fused semantic vector of each point.

Images