A multi-modal semantic mapping method and system based on master-slave architecture

Abstract

The application belongs to the technical field of point cloud processing, and discloses a multi-modal semantic mapping method and system based on a master-slave architecture. The method comprises the following steps: S1, extracting feature points in point cloud data and calculating the covariance matrix of the neighborhood point cloud of each feature point; S2, solving the optimal pose corresponding to the minimum of the target function; S3, converting the obtained point cloud data into a world coordinate system using the optimal pose, fusing all frame point cloud data in the world coordinate system to form a three-dimensional point cloud map, projecting the three-dimensional point cloud map to a camera coordinate system, and inputting the three-dimensional point cloud map projected to the camera coordinate system into a semantic segmentation neural network to obtain the semantic label corresponding to each point cloud in the three-dimensional point cloud map, thereby achieving semantic mapping. Through the application, the problems of weak dynamic adaptability, insufficient model generalization ability and long-term running precision decay of the traditional three-dimensional semantic mapping method in a dynamic complex scene are solved.

Description

Technical Field

[0001] This invention belongs to the technical field of point cloud processing, and more specifically, relates to a multimodal semantic mapping method and system based on a master-slave architecture. Background Technology

[0002] In recent years, with the continuous iteration of unmanned system technology, how to enable intelligent agents to accurately perceive their surroundings in complex scenarios has gradually become a research hotspot. Multimodal semantic mapping based on a master-slave architecture is an innovative method for achieving intelligent agent perception, and this technology can be applied to several key areas such as autonomous driving and robot autonomous navigation.

[0003] Currently, traditional cumulative mapping methods are susceptible to interference from moving objects in dynamic environments, leading to significant accumulation of pose estimation errors and map distortion. Furthermore, fixed model weights cannot adapt to sudden environmental changes, resulting in decreased semantic segmentation accuracy and increased feature noise, thereby reducing the reliability of the semantic map. Finally, single-sensor methods struggle to handle complex scenarios, necessitating reliance on multimodal complementarity to improve system perception accuracy and robustness through cross-modal constraints.

[0004] Furthermore, in the process of providing pose estimation for semantic mapping, traditional point cloud registration methods only consider geometric feature matching, ignoring the consistency of local structural information in the point cloud. This leads to a significant decrease in registration accuracy in feature-sparse scenarios, resulting in obvious deviations in pose estimation and ultimately causing distortion in the constructed point cloud map. Meanwhile, in terms of dynamic object detection and processing, most existing methods rely on motion information from point clouds between consecutive frames to identify dynamic objects. While this method is effective for obviously moving objects, it lacks the ability to handle objects that are currently stationary but have the potential to move. If these potential dynamic objects are incorrectly included in the static map construction process in the initial stage, they will generate a large number of abnormal point clouds in subsequent frames, not only misleading the pose estimation process but also polluting the map data. Here, we use LiDAR echo intensity to construct a grayscale consistency detection model. By comparing changes in grayscale variance, we achieve efficient and accurate removal of dynamic point clouds. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a multimodal semantic mapping method and system based on a master-slave architecture, which solves the problems of weak dynamic adaptability, insufficient model generalization ability, and long-term accuracy decay in traditional 3D semantic mapping methods under dynamic and complex scenarios.

[0006] To achieve the above objectives, according to one aspect of the present invention, a multimodal semantic graph construction method based on a master-slave architecture is provided, the method comprising the following steps:

[0007] S1 extracts corner and planar features from point cloud data, calculates the local curvature of points in the corner and planar features, and uses points with local curvature greater than a preset threshold as feature points. It then calculates the covariance matrix of the neighborhood point cloud for each feature point.

[0008] S2 fits the corner features into edge lines and the planar features into a two-dimensional plane. The sum of the Euclidean distance residual from the point to the edge line, the Euclidean distance residual from the point to the two-dimensional plane, and the covariance matrix difference is used as the objective function to solve for the optimal pose corresponding to the minimum objective function.

[0009] S3 uses the optimal pose to convert the acquired point cloud data into the world coordinate system, merges the point cloud data of all frames in the world coordinate system to form a three-dimensional point cloud map, projects the three-dimensional point cloud map onto the camera coordinate system, and inputs the three-dimensional point cloud map projected onto the camera coordinate system into the semantic segmentation neural network to obtain the semantic label corresponding to each point cloud in the three-dimensional point cloud map, thereby realizing semantic mapping.

[0010] More preferably, in step S2, the formula for the objective function is as follows:

[0011]

[0012] Where ξ is the Lie algebra representation of the pose. The three-dimensional coordinates of the i-th corner feature in the current frame; To be in the previous frame with The corresponding straight line; d l This represents the distance error between points and lines in inter-frame matching. The three-dimensional coordinates of the j-th corner feature in the current frame; To be in the previous frame with The corresponding plane; d p This represents the point-to-surface distance error in inter-frame matching. Let be the covariance matrix of the local neighborhood point cloud of the k-th feature point in the current frame; In the previous frame, the covariance matrix of the local neighborhood point cloud corresponding to the feature point is K. l K represents the set of matching point-line feature pairs between the current frame and the previous frame; p K represents the set of point-to-surface feature pairs that match in the current frame and the previous frame; s It represents the set of covariance matrix feature pairs that match in the current frame and the previous frame.

[0013] More preferably, in step S1, the formula for calculating the local curvature is as follows:

[0014]

[0015] Where C is the local curvature of the feature point, and λ1, λ2, and λ3 are the three eigenvalues ​​corresponding to the covariance matrix of the local neighborhood point cloud of the feature point.

[0016] More preferably, converting the acquired point cloud data to the world coordinate system using the optimal pose is achieved by multiplying the point cloud data coordinates by the optimal pose.

[0017] More preferably, after step S3, noise reduction is performed according to the following steps:

[0018] Calculate the k-neighborhood distance of each point in the 3D point cloud map projected onto the camera coordinate system and evaluate the local point cloud density;

[0019] The clustering radius is set according to the assessed point cloud density, and the point cloud is clustered according to the clustering radius;

[0020] Points that are not clustered are deleted using Euclidean point cloud clustering.

[0021] More preferably, the formula for setting the cluster radius is as follows:

[0022] R(P i )=k·d(P i N pts )+R0

[0023] Among them, P i Let d be the point cloud being considered for clustering, and d be the distance from the point cloud to its N nodes. pts The average distance between N nearest neighbors, pts The number of points considered when calculating nearest neighbors is k, which is the scaling factor, and R0 is the minimum cluster radius offset.

[0024] More preferably, before step S3, dynamic points in the point cloud data are removed according to the following steps:

[0025] Obtain the echo intensity of each point in the point cloud data and convert the echo intensity into a grayscale value;

[0026] Calculate the grayscale variance of each point at different times;

[0027] Points with a grayscale variance greater than the preset dynamic threshold are not dynamic points and are deleted.

[0028] More preferably, the dynamic threshold is set according to the following formula:

[0029] T(P) = T0 + k D ·D(P)+k σ ·σ(P)

[0030] Where P is the point cloud for dynamic detection; T(P) is the dynamic intensity difference threshold calculated for point P; k D D(P) is the distance influence coefficient; D(P) is the distance from point P to the sensor; k σ σ(P) is the standard deviation influence coefficient; σ(P) is the standard deviation of the echo intensity at the location of point P.

[0031] More preferably, after step S3, the semantic tags are further optimized according to the following steps:

[0032] Calculate the confidence entropy of each semantic label. Points with confidence entropy greater than a preset threshold are designated as hard example features. Use these hard example features to train the semantic segmentation neural network, thereby updating the semantic segmentation neural network.

[0033] The formula for calculating confidence entropy is as follows:

[0034]

[0035] Among them, H i p is the confidence entropy of the i-th detected target; ic Let C be the Softmax probability that the target belongs to category c; C is the total number of semantic categories.

[0036] According to another aspect of the present invention, a multimodal semantic mapping system based on a master-slave architecture is provided, the system including an executor for executing the aforementioned multimodal semantic mapping method based on a master-slave architecture.

[0037] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0038] 1. This invention employs a feature intensity screening method to extract more discriminative corner and planar features. It combines a structural consistency constraint descriptor generated from the constructed local point cloud covariance matrix in the objective function, and jointly optimizes the difference between point-line / point-plane distance residuals and the covariance matrix. This significantly improves the accuracy and robustness of pose estimation, especially in scenarios with sparse features or repetitive structures, effectively reducing false matches. Addressing the issue of long-term accuracy decay, it effectively controls the accumulation of pose estimation errors by reducing the false match rate in feature matching. This fundamentally solves the problem of map distortion caused by the continuous accumulation of errors over time in traditional methods, ensuring the system maintains high-precision mapping capabilities during long-term operation.

[0039] 2. In this invention, an adaptive density point cloud clustering algorithm is adopted to address semantic misassociations caused by projection errors. The clustering radius is dynamically adjusted according to the local density of the point cloud to effectively remove abnormal semantic points. At the same time, a grayscale consistency detection model is constructed using LiDAR echo intensity, and dynamic point clouds are efficiently filtered out through dynamic thresholds. This solution addresses the dynamic adaptability problem and can effectively identify and remove various dynamic interferences, including stationary but potentially moving objects. This allows the system to accurately construct static maps even in complex dynamic environments, significantly improving the accuracy and stability of semantic maps in complex dynamic scenarios and ensuring the long-term usability of the map.

[0040] 3. This invention calculates the confidence entropy of each semantic label. Points with a confidence entropy greater than a preset threshold are designated as hard example features. These hard example features are used to train the semantic segmentation network separately, updating the weights within the network. This achieves online asynchronous updates to the semantic segmentation model, enabling real-time updates without interrupting the current task. Addressing the issue of insufficient model generalization ability, this targeted training with hard example features allows the system to quickly learn feature patterns in new scenarios, improving its adaptability to unknown environments and significantly enhancing its generalization capabilities, ensuring real-time recognition performance. The asynchronous weight update mechanism enables continuous model optimization, avoiding the performance degradation caused by environmental changes in traditional fixed-weight models. It effectively solves the technical challenge of gradually decreasing semantic recognition accuracy over long-term operation, ensuring the system maintains high-precision semantic understanding capabilities throughout extended periods. Attached Figure Description

[0041] Figure 1 This is a flowchart of a multimodal semantic graphing method based on a master-slave architecture constructed according to a preferred embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the point cloud registration principle with structural consistency constraints constructed according to a preferred embodiment of the present invention, wherein (a) is a schematic diagram of the point cloud of the current frame and (b) is a schematic diagram of the point cloud of the previous frame.

[0043] Figure 3 This is a schematic diagram of the semantic segmentation neural network constructed according to a preferred embodiment of the present invention.

[0044] Figure 4 This is a schematic diagram of the asynchronous model weight loading process constructed according to a preferred embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0046] like Figure 1 As shown, a multimodal semantic graph construction method based on a master-slave architecture includes the following steps:

[0047] S1. Obtain spatial point cloud information of the surrounding environment of the LiDAR, extract corner features and planar features, and then perform a feature intensity filtering operation based on PCA, that is, filter out features with local curvature greater than a preset threshold in corner features and planar features.

[0048] Corner feature: A set of points in a point cloud that represents a dramatic change in geometric structure, with its local neighborhood point cloud exhibiting a sharp, angular distribution.

[0049] Planar features: The point set in a point cloud that represents a continuous smooth surface has a local neighborhood point cloud distribution that approximates a plane.

[0050] Preferably, the feature intensity screening operation in step S1 specifically includes the following steps:

[0051] S11: Calculate the covariance matrix of the local neighborhood point cloud of the extracted feature points, perform eigenvalue decomposition on the covariance matrix, and obtain the three eigenvalues ​​λ1, λ2, and λ3 corresponding to the covariance matrix.

[0052] S12: Based on the local curvature calculation formula, select the feature points corresponding to the larger value of the local curvature between the corner feature and the planar feature to participate in subsequent steps. The local curvature calculation formula is:

[0053]

[0054] Where C is the local curvature of the feature point, and λ1, λ2, and λ3 are the three eigenvalues ​​corresponding to the covariance matrix of the local neighborhood point cloud of the feature point.

[0055] Based on the feature points extracted in the current frame, assuming that the lidar moves at approximately a constant speed in a short period of time, the end computing node uses the lidar pose of the previous frame as the initial value of the pose of the current frame, and then constructs the local point cloud covariance matrix in the feature points of the current frame and the neighboring point clouds of the feature points of the adjacent previous frames.

[0056] S2. The goal is to obtain an accurate pose estimate by minimizing the difference between the Euclidean distance residuals and the covariance matrix of points / lines and points / surfaces; for example... Figure 2As shown, the optimal pose of the lidar in the current frame is calculated when the target function is minimized using the covariance matrix of the current frame and the previous frame.

[0057] Line: Edge line fitted by corner feature, constraining the alignment of sharp structures.

[0058] Surface: A two-dimensional plane fitted with planar features, constraining the fitting of flat regions.

[0059] The objective function is:

[0060]

[0061] Where ξ is the Lie algebra representation of the pose. The three-dimensional coordinates of the i-th corner feature in the current frame; To be in the previous frame with The corresponding straight line; d l This represents the distance error between points and lines in inter-frame matching. The three-dimensional coordinates of the j-th corner feature in the current frame; To be in the previous frame with The corresponding plane; d p This represents the point-to-surface distance error in inter-frame matching. Let be the covariance matrix of the local neighborhood point cloud of the k-th feature point in the current frame; The covariance matrix of the local neighborhood point cloud of the corresponding feature point in the previous frame.

[0062] Furthermore, a grayscale consistency detection model is constructed based on the lidar echo intensity. According to the scene echo intensity distribution, the set threshold is dynamically adjusted. By comparing the echo intensity changes at the same position in adjacent frames, efficient removal of dynamic point clouds is achieved.

[0063] The echo intensity information returned by the lidar in different scanning cycles is treated as a grayscale image and processed. The point cloud is then compared with the grayscale changes between adjacent frames to determine whether the object is a dynamic object. The specific steps are as follows:

[0064] First, when the lidar collects point cloud data, the returned point cloud data includes the three-dimensional spatial coordinate information and echo intensity information of each point. The point cloud data is projected onto a two-dimensional polar coordinate system according to the time series to form pixels. The echo intensity of each pixel forms an echo intensity image. The echo intensity of each pixel is converted into a grayscale value. Specifically, for each point cloud data point, the system extracts its echo intensity value I and converts it into a standard grayscale range [0,255] through linear mapping. The variance of each point after conversion is calculated. Then, a grayscale consistency detection algorithm is used to calculate the grayscale variance of each pixel in the time dimension. If the variance exceeds the set dynamic threshold, it is considered that there is a dynamic object at that location. Finally, the point cloud data that is judged to be dynamic is removed from the overall point cloud.

[0065] The linear mapping conversion formula from echo intensity to grayscale value is as follows:

[0066]

[0067] Where G is the grayscale value of the current point after conversion, and I is the echo intensity of the current point. max The maximum acceptable echo intensity for lidar.

[0068] The dynamic threshold formula is as follows:

[0069] T(P) = T0 + k D ·D(P)+k σ ·σ(P)

[0070] Where P is the point cloud for dynamic detection; T(P) is the dynamic intensity difference threshold calculated for point P; k D D(P) is the distance influence coefficient; D(P) is the distance from point P to the sensor; k σ σ is the standard deviation influence coefficient; σ(P) is the standard deviation of the echo intensity at point P.

[0071] S3, such as Figure 3 As shown, the point cloud data obtained in each frame is multiplied with the optimal pose corresponding to the current frame to obtain the point cloud of the current frame in the world coordinate system. The point clouds in the world coordinate system of all frames are fused to form a three-dimensional point cloud map in the world coordinate system. The three-dimensional point cloud map is projected onto the camera imaging plane through the calibration matrix.

[0072] Then, based on the semantic mask output by the semantic segmentation neural network, the segmentation accuracy of the object edge is enhanced by the boundary optimization module, and an optimized semantic mask is generated. The points projected from the point cloud map onto the image plane are matched with the optimized semantic mask to give the point cloud the corresponding semantic meaning, thus completing the semantic association of the point cloud.

[0073] As a preferred embodiment, the steps performed by the semantic segmentation neural network boundary optimization module are as follows:

[0074] S31: Obtain the output value of the semantic segmentation neural network, upsample it to the resolution of the shallow feature map, concatenate the upsampled prediction value with the shallow feature map channels, and input it into the lightweight convolutional neural network module.

[0075] S32: A lightweight convolutional neural network learns gradient information in the direction of the boundary normal and restores the gradient information to the original prediction value. The Sigmoid function is then activated to generate a refined mask.

[0076] S33: Compare the generated refined mask with the ground truth, and optimize the segmentation results by combining binary cross-entropy and Dice loss to eliminate the blurred transition region.

[0077] The joint optimization objective function is:

[0078]

[0079] Among them, M p For the boundary-optimized segmentation mask; M gt For real masks; Binary cross-entropy loss; is the Dice loss; w1 and w2 are the weighting coefficients.

[0080] The aforementioned module utilizes a lightweight convolutional neural network to learn boundary gradient information, generate a refined mask, and performs joint optimization using multiple losses. This effectively solves the problems of blurred object boundaries and loss of detail in traditional segmentation, generating high-quality 2D semantic information. Subsequently, a calibration matrix is ​​used to accurately align it with the 3D point cloud, assigning accurate semantic labels to the point cloud and improving the quality of the semantic map.

[0081] Furthermore, to address the noise problem caused by projection errors in S3, namely, the deviation in the projection position of the point cloud causing it to be associated with incorrect semantic labels and forming noisy data, an adaptive density point cloud clustering algorithm is adopted. The threshold is dynamically adjusted according to the local point cloud density to separate outliers and remove abnormal point cloud data with semantic misassociation.

[0082] Due to projection errors, a certain amount of noisy point cloud data may be introduced. This noise not only affects subsequent processing efficiency but may also lead to semantic association errors. Therefore, an adaptive density point cloud clustering algorithm is used to optimize the point cloud data. This algorithm automatically adjusts the clustering parameters based on the local density of the point cloud, effectively eliminating erroneously associated point cloud data.

[0083] First, calculate the k-neighbor distance of each point and evaluate its local density to assess the density of the point cloud in the clustered region. Then, based on the density of the point cloud, set the clustering radius according to a preset formula, remove the un-clustered points through Euclidean point cloud clustering, and finally retain the high-density point cloud region after clustering as the result of accurate semantic association.

[0084] The formula for the cluster radius is:

[0085] R(P i )=k·d(P i N pts )+R0

[0086] Among them, P iThe point cloud is being considered for clustering; d represents the distance from the point cloud to its N... pts The average distance between N nearest neighbors; pts The number of points considered when calculating nearest neighbors; k is the scaling factor; R0 is the minimum cluster radius offset.

[0087] Furthermore, such as Figure 4 As shown, new training model weights are assigned according to environmental changes. Based on the asynchronous weight loading mechanism, while continuously processing point cloud data, the background receives and loads new model weights, and then performs semantic re-segmentation on the point cloud data to achieve real-time updates of recognition results.

[0088] New training model weights are assigned based on environmental changes. Based on the asynchronous weight loading mechanism, while continuously processing point cloud data, the background receives and loads new model weights, and then performs semantic re-segmentation on the point cloud data to achieve real-time updates of recognition results.

[0089] Furthermore, the specific process is as follows: Figure 4 As shown, the identification and update steps are as follows:

[0090] (1) Calculate the confidence entropy of each detected target in the semantic segmentation model in real time, filter out difficult samples with high uncertainty by setting a threshold, that is, difficult samples with confidence exceeding the threshold, and transmit them to the host after efficient encoding and compression via Socket protocol.

[0091] (2) After the number of difficult sample samples reaches the batch threshold, the original difficult sample samples are augmented by adding random noise to obtain augmented difficult sample samples. Specifically, Gaussian noise with a mean of 0 and a standard deviation of σ is added to the three-dimensional coordinates of the difficult sample point cloud, and the entire difficult sample point cloud is randomly rotated within the range of [-15°, 15°]. After the above processing, each original difficult sample generates multiple augmented difficult sample samples. These augmented difficult sample samples are input into the feature extraction neural network, which outputs the data-augmented difficult sample features in the high-dimensional space. All the difficult sample features form a training set, and then the semantic segmentation model parameters are updated based on the elastic incremental learning framework, that is, only the parameters related to the difficult samples in the model are updated, a new weight file is generated and pushed to the slave buffer via the Socket protocol;

[0092] (3) Run an independent weight management thread, which periodically checks the weight temporary storage area. Once a new weight file is detected, the following asynchronous loading process is started.

[0093] (4) Load new weights in the background without interrupting the current segmentation effect. After loading is complete, smoothly switch to the new model to achieve real-time recognition effect updates.

[0094] The formula for calculating the confidence entropy of the detected target is as follows:

[0095]

[0096] Among them, H i p is the confidence entropy of the i-th detected target; ic Let C be the Softmax probability that the target belongs to category c; C is the total number of semantic categories.

[0097] The loss function of the elastic incremental learning framework is as follows:

[0098]

[0099] In the formula, λ is the detection loss of the original model; λ is the regularization coefficient, which controls the degree of preservation of the old model; F w For the diagonal elements of the Fisher information matrix with parameter w; θ w θ represents the current model parameters. wo These are the parameters for the old model.

[0100] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal semantic graph construction method based on a master-slave architecture, characterized in that, The method includes the following steps: S1 extracts corner and planar features from point cloud data, calculates the local curvature of points in the corner and planar features, and uses points with local curvature greater than a preset threshold as feature points. It then calculates the covariance matrix of the neighborhood point cloud for each feature point. S2 fits the corner features into edge lines and the planar features into a two-dimensional plane. The sum of the Euclidean distance residual from the point to the edge line, the Euclidean distance residual from the point to the two-dimensional plane, and the covariance matrix difference is used as the objective function to solve for the optimal pose corresponding to the minimum objective function. S3 uses the optimal pose to convert the point cloud data of the current frame into the world coordinate system, merges the point cloud data of all frames in the world coordinate system to form a three-dimensional point cloud map, projects the three-dimensional point cloud map onto the camera coordinate system, and inputs the three-dimensional point cloud map projected onto the camera coordinate system into the semantic segmentation neural network to obtain the semantic label corresponding to each point cloud in the three-dimensional point cloud map, thereby realizing semantic mapping. In step S2, the formula for the objective function is as follows: in, Let Lie algebra represent the pose. The three-dimensional coordinates of the i-th corner feature in the current frame; To be in the previous frame with The corresponding straight line; This represents the distance error between points and lines in inter-frame matching. The three-dimensional coordinates of the j-th corner feature in the current frame; To be in the previous frame with The corresponding plane; This represents the point-to-surface distance error in inter-frame matching. Let be the covariance matrix of the local neighborhood point cloud of the k-th feature point in the current frame; The covariance matrix of the local neighborhood point cloud of the corresponding feature point in the previous frame It represents the set of point-line feature pairs that match in the current frame and the previous frame; It represents the set of point-surface feature pairs that match in the current frame and the previous frame; It represents the set of covariance matrix feature pairs that match in the current frame and the previous frame.

2. The multimodal semantic graph construction method based on a master-slave architecture as described in claim 1, characterized in that, In step S1, the formula for calculating the local curvature is as follows: in, The local curvature of the feature point, , , The three eigenvalues ​​are the covariance matrix of the local neighborhood point cloud of the feature point.

3. A multimodal semantic graph construction method based on a master-slave architecture as described in claim 1 or 2, characterized in that, The conversion of the point cloud data of the current frame to the world coordinate system using the optimal pose is achieved by multiplying the point cloud data coordinates of the current frame with the optimal pose.

4. A multimodal semantic graph construction method based on a master-slave architecture as described in claim 1 or 2, characterized in that, After step S3, noise reduction is performed according to the following steps: Calculate the k-neighborhood distance of each point in the 3D point cloud map projected onto the camera coordinate system and evaluate the local point cloud density; The clustering radius is set according to the assessed point cloud density, and the point cloud is clustered according to the clustering radius; Points that are not clustered are deleted using Euclidean point cloud clustering.

5. The multimodal semantic graph construction method based on a master-slave architecture as described in claim 4, characterized in that, The formula for setting the cluster radius is as follows: in, For point clouds that are being considered for clustering, d For that point cloud to its The average distance between the nearest neighbors, To calculate the number of points considered when calculating nearest neighbors, k Scaling factor This is the minimum cluster radius offset.

6. The multimodal semantic graph construction method based on a master-slave architecture as described in claim 1, characterized in that, Before step S3, remove dynamic points from the point cloud data according to the following steps: Obtain the echo intensity of each point in the point cloud data and convert the echo intensity into a grayscale value; Calculate the grayscale variance of each point at different times; Points with a grayscale variance greater than a preset dynamic threshold are considered dynamic points and are deleted.

7. The multimodal semantic graph construction method based on a master-slave architecture as described in claim 6, characterized in that, The dynamic threshold is set according to the following formula: Where P is the point cloud for dynamic detection; The dynamic intensity difference threshold calculated for point P; This is the distance influence coefficient; The distance from point P to the sensor; The standard deviation influence coefficient; The standard deviation of the echo intensity at point P.

8. The multimodal semantic graph construction method based on a master-slave architecture as described in claim 1, characterized in that, After step S3, the semantic tags are further optimized according to the following steps: Calculate the confidence entropy of each semantic label. Points with confidence entropy greater than a preset threshold are designated as hard example features. Use these hard example features to train the semantic segmentation neural network, thereby updating the semantic segmentation neural network. The formula for calculating confidence entropy is as follows: in, Let be the confidence entropy of the i-th detected target; This represents the Softmax probability that the detected target belongs to category c. This represents the total number of semantic categories.

9. A multimodal semantic mapping system based on a master-slave architecture, characterized in that, The system includes an executor for executing a multimodal semantic graphing method based on a master-slave architecture as described in any one of claims 1-8.

Images