Incremental semantic segmentation system based on visual SLAM

Abstract

The invention relates to the technical field of simultaneous localization and mapping, and discloses a visual SLAM-based incremental semantic segmentation system, which comprises a data preparation module used for storing high-precision depth information generated by a sensor collecting an image, RGB information and camera internal reference information; the dense mapping module is used for carrying out key frame pose estimation on high-precision depth information, RGB information and camera internal reference information generated by the image, generating key frame pose information and three-dimensional sparse point cloud coordinate information, and carrying out dense mapping thread calculation to generate a dense three-dimensional point cloud map; the incremental semantic segmentation module is used for calculating the object category and probability of the image by utilizing a trained incremental semantic segmentation model according to the pose information of the key frame, and generating an RGB-D segmented image; the fusion module is used for performing point cloud reprojection, probability fusion and point cloud updating according to dense three-dimensional point cloud mapping to generate a global semantic map; according to the method, the flexibility, the precision and the real-time performance of the semantic SLAM system are improved.

Description

Technical Field

[0001] The present invention relates to the field of simultaneous positioning and mapping technology, and in particular to an incremental semantic segmentation system based on visual SLAM. Background Art

[0002] In an unknown environment, how a robot can move while building an incremental map based on its own position and achieve accurate autonomous positioning according to the map is the core issue in the field of autonomous exploration of intelligent agents, namely Simultaneous Localization and Mapping (SLAM). Positioning refers to the ability of a robot to obtain the internal state of the system, including position, direction, speed, etc. Mapping refers to the ability of the system to simultaneously perceive the external environment and extract and reconstruct it. If you want the intelligent agent to interact with the environment, such as to achieve tasks such as object search and human-computer interaction, you need both geometric information and semantic information of the environment to perform high-level reasoning on the functions of the object environment.

[0003] The ORB-SLAM framework has played an important role in the development of visual SLAM in the past and promoted the maturity of this technology. The latest system of this framework is ORB-SLAM3.0. It is suitable for various camera types, such as monocular, RGB-D and stereo vision, and provides improved pose estimation output, but its main purpose is to achieve higher precision positioning, and the established map is still a sparse landmark map. In order to interact with the intelligent agent and the environment, it is necessary to improve the algorithm and generate a dense point cloud map to prepare for the next step of fusing semantic information. At present, the indirect semantic segmentation method is one of the mainstream methods based on deep learning point cloud segmentation. The indirect semantic segmentation method converts the original point cloud data into a conventional 3D voxel grid or multi-view, and indirectly extracts features from the three-dimensional point cloud data by data transformation. For example, a study is based on two-dimensional multi-view, first projecting the point cloud into a two-dimensional image of multiple views, using convolution dense pixel segmentation technology on the projected two-dimensional image, and then back-projecting, thereby realizing the semantic segmentation of point cloud data.

[0004] Although current deep learning methods are very effective in achieving point cloud semantic segmentation, they are also heavily dependent on the full amount of training data in the dataset. When encountering new categories that need to be segmented in an open environment, it is necessary to jointly construct the datasets of new objects and old objects, and then retrain the network. This process is time-consuming and requires a lot of computing resources. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an incremental semantic segmentation system based on visual SLAM, which is used to solve the problem that the existing semantic SLAM system is overly dependent on the training of the full data set and needs to be retrained when encountering new objects in an open environment.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] An incremental semantic segmentation system based on visual SLAM, comprising:

[0008] The data preparation module is used to store high-precision depth information, RGB information and camera intrinsic parameter information generated by the sensor's image acquisition;

[0009] The dense mapping module is used to perform key frame pose estimation on the high-precision depth information, RGB information and camera intrinsic parameter information generated by the image, generate key frame pose information and three-dimensional sparse point cloud coordinate information, and perform dense mapping thread calculation to generate dense three-dimensional point cloud mapping;

[0010] The incremental semantic segmentation module is used to calculate the object category and probability of the image based on the key frame pose information and use the trained incremental semantic segmentation model to generate an RGB-D segmentation map;

[0011] The fusion module is used to build maps based on dense 3D point clouds, perform point cloud reprojection, probabilistic fusion and point cloud update to generate a global semantic map.

[0012] Furthermore, when the sensor collects images, a binocular camera and an inertial measurement unit are jointly calibrated to generate high-precision depth information, RGB information, and camera intrinsic parameter information.

[0013] Furthermore, the high-precision depth information, RGB information and camera intrinsic parameter information generated by the image are used to estimate the key frame pose, generate the key frame pose information and three-dimensional sparse point cloud coordinate information, and perform dense mapping thread calculation. The specific process of generating dense three-dimensional point cloud mapping is as follows:

[0014] Step 1: Create a new dense mapping thread;

[0015] Step 2: Insert the key frames between the tracking thread and the local mapping thread into the new dense mapping thread;

[0016] Step 3: According to the high-precision depth information, RGB information and phase intrinsic parameter information, the pixel coordinates of the key frame are converted into binocular camera coordinates, that is:

[0017]

[0018] Among them, X c , Y c , Z c represents the three-dimensional coordinates of the key frame in the binocular camera coordinate system, u and v represent the pixel coordinates of the key frame, that is, the horizontal and vertical coordinates of the key frame in the image, u 0、v 0 Represents the optical center coordinates of the binocular camera, that is, the horizontal and vertical coordinates of the center point of the image, Z c represents the distance between the object and the binocular camera, f x 、f y Respectively represent the focal length of the binocular camera in the x direction and the y direction;

[0019] Step 4: According to the key frame pose information of the tracking thread, the binocular camera coordinates of the key frame are converted into world coordinates, that is:

[0020]

[0021] Among them, 1 represents the translation operation, that is, it is expanded to the fourth component of the homogeneous coordinates, R represents the 3×3 rotation matrix, that is, the rotation of the binocular camera relative to the world coordinate system, T represents the transpose, t represents the 3×1 translation vector, that is, the translation position of the binocular camera relative to the world coordinate system, X, Y, Z represent the three-dimensional coordinates of the key frame in the world coordinate system, that is, the position of the object in the image in the world coordinate system, L w Represents the conversion from the world coordinate system to the camera coordinate system;

[0022] Step 5: Use the display function to perform point cloud voxel filtering on all key frames in the world coordinate system to generate dense 3D point cloud mapping.

[0023] Furthermore, the specific process of performing point cloud voxel filtering on all key frames in the world coordinate system to generate dense 3D point cloud mapping is as follows:

[0024] All key frames in the world coordinate system are merged into a global point cloud, and the global point cloud is filtered and displayed to generate a dense 3D point cloud map.

[0025] Furthermore, the incremental semantic segmentation model is a SATS model, and the SATS model includes several encoding blocks and 1 decoding block, wherein the encoding block includes several attention heads, and the decoding block is a multi-layer perceptron.

[0026] Furthermore, the training process of the trained incremental semantic segmentation model is:

[0027] First, the training data set is input into the SATS model, the encoding block is used to extract multidimensional features, and the decoding block is used to fuse the multidimensional features;

[0028] At the same time, during the encoding and decoding process, the category region pooling method is introduced to pool the elements of the same category in the trained self-attention map to generate a pooled class-specific self-attention vector, and the self-attention distillation loss function is used to distill the pooled class-specific self-attention vector. Finally, the cross loss function is introduced as the classification loss function. By constructing a total loss function to supervise the SATS model, the trained SATS model, that is, the trained incremental semantic segmentation model, is obtained.

[0029] Furthermore, the pooled class-specific self-attention vector is:

[0030]

[0031] Among them, f i，j，b，h represents the pooling vector of the b-th element of the j-th encoding block in the i-th self-attention map in the h-th attention head, M i，j，b The position set of the b-th element in the j-th encoding block in the i-th self-attention map, A i，j，h It represents the 3D self-attention map of the h-th attention head in the last layer of the j-th coding block, and u′ and v′ represent the horizontal and vertical coordinate positions of the b-th element, respectively.

[0032] Furthermore, the self-attention distillation loss function is:

[0033]

[0034] Among them, L a (θ) represents the self-attention distillation loss, N represents the number of images in the training dataset, H represents the number of self-attention heads, and B i represents the set of categories appearing in the i-th image, represents the pooling vector of the b-th element in the i-th self-attention map at the h-th attention head at the t-th time, Represents the pooling vector of the b-th element in the i-th self-attention map at the h-th attention head at the t-1-th time.

[0035] Furthermore, the total loss function is:

[0036] L(θ)=L c (θ)+λ a L a (θ)+λ d L d (θ)

[0037] Among them, L(θ) represents the total loss, L c (θ) represents the cross entropy loss, λ a , d They represent the weight hyperparameters for adjusting self-attention distillation and knowledge distillation, L d(θ) denotes the unbiased knowledge distillation loss.

[0038] Further, the fusion module includes a service submodule and a client submodule;

[0039] The service submodule is used to receive the RGB-D segmentation map and the key frames transmitted by the client submodule, segment the key frames using the incremental semantic segmentation model, and transmit the segmented key frames to the client submodule;

[0040] The client submodule is used to receive dense 3D point cloud mapping and transmit key frames to the server through the TCP / IP protocol. At the same time, according to the segmentation key frames returned by the server, the segmentation key frames and the dense 3D point cloud mapping are reprojected and probabilistically fused. Finally, the Bayesian algorithm is used to update the point cloud labels to generate a global semantic map.

[0041] The present invention has the following beneficial effects:

[0042] 1. The present invention proposes an incremental semantic segmentation system based on visual SLAM, designs an incremental semantic segmentation module for incremental learning, and improves the training speed of the trained incremental semantic segmentation model by reducing the amount of new category data at each stage, thereby avoiding retraining of a completely new data set;

[0043] 2. Design a dense mapping module to upgrade sparse point clouds to dense point cloud maps, and achieve more accurate 3D semantic map construction by fusing geometric and semantic information;

[0044] 3. Design a fusion module to achieve semantic segmentation in three-dimensional space by mutual projection of three-dimensional and two-dimensional images, breaking through the limitation of semantic segmentation only on two-dimensional images;

[0045] 4. The system can also dynamically adapt to new object categories without retraining the entire incremental semantic segmentation model, which improves the interaction ability of the intelligent agent with the environment, such as the ability to perform tasks such as object search and human-computer interaction; it also increases the flexibility, accuracy and real-time performance of the semantic SLAM system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the structure of an incremental semantic segmentation system based on visual SLAM proposed in the present invention;

[0047] Figure 2 Schematic diagram of sensor jitter comparison before and after calibration in the embodiment;

[0048] Figure 3 This is a schematic diagram of the implementation principle of the dense mapping module;

[0049] Figure 4Schematic diagram of mapping the generated dense 3D point cloud;

[0050] Figure 5 A simplified structural diagram of the SATS model

[0051] Figure 6 A structural diagram of the fusion module. DETAILED DESCRIPTION

[0052] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0053] like Figure 1 As shown, an incremental semantic segmentation system based on visual SLAM includes:

[0054] The data preparation module is used to store high-precision depth information, RGB information, and camera intrinsic parameter information generated by the sensor's image acquisition.

[0055] Specifically, when the sensor collects images, a binocular camera and an inertial measurement unit are jointly calibrated to generate high-precision depth information, RGB information, and camera intrinsic parameter information.

[0056] In this embodiment, when the sensor collects images, it is necessary to calibrate the sensor to obtain the internal parameters of each sensor and the coordinate conversion relationship between them, so as to fix the calibration parameters. That is, the ultimate purpose of calibration is to obtain high-precision depth information, RGB information and camera internal parameter information of the image; the sensor jitter comparison before and after calibration is as follows: Figure 2 As shown, from Figure 2 The left side is the uncalibrated sensor, and the right side is the calibrated sensor. Figure 2 It can be seen that the calibrated sensor information is more accurate.

[0057] The dense mapping module is used to perform key frame pose estimation on the high-precision depth information, RGB information and camera intrinsic parameter information generated by the image, generate key frame pose information and three-dimensional sparse point cloud coordinate information, and perform dense mapping thread calculation to generate dense three-dimensional point cloud mapping.

[0058] In this embodiment, the dense mapping module implements dense 3D point cloud mapping based on the ORB-SLAM3 system, which uses the visual feature point method to generate key frame poses and sparse point cloud maps; in addition, in order to achieve dense semantic segmentation, it is necessary to use the depth information and camera model of the data set or sensor to pre-build a dense point cloud map. The implementation principle of the dense mapping module is as follows Figure 3 As shown, the specific operations for generating dense 3D point cloud mapping are as follows:

[0059] Step 1: Create a new dense mapping thread.

[0060] Step 2: Insert the key frames between the tracking thread and the local mapping thread into the new dense mapping thread.

[0061] Step 3: According to the high-precision depth information, RGB information and phase intrinsic parameter information, the pixel coordinates of the key frame are converted into binocular camera coordinates, that is:

[0062]

[0063] Among them, X c , Y c , Z c represents the three-dimensional coordinates of the key frame in the binocular camera coordinate system, u and v represent the pixel coordinates of the key frame, that is, the horizontal and vertical coordinates of the key frame in the image, u 0 、v 0 Represents the optical center coordinates of the binocular camera, that is, the horizontal and vertical coordinates of the center point of the image, Z c represents the distance between the object and the binocular camera, f x 、f y Represent the focal length of the binocular camera in the x direction and the y direction respectively.

[0064] In this embodiment, the distance Z between the object and the binocular camera c It also refers to the depth value, that is, the depth information; at the same time, the focal length f of the binocular camera in the x direction x The focal length f in the y direction y It is obtained by calibrating the camera's intrinsic parameter matrix.

[0065] Step 4: According to the key frame pose information of the tracking thread, the binocular camera coordinates of the key frame are converted into world coordinates, that is:

[0066]

[0067] Among them, 1 represents the translation operation, that is, it is expanded to the fourth component of the homogeneous coordinates, R represents the 3×3 rotation matrix, that is, the rotation of the binocular camera relative to the world coordinate system, T represents the transpose, t represents the 3×1 translation vector, that is, the translation position of the binocular camera relative to the world coordinate system, X, Y, Z represent the three-dimensional coordinates of the key frame in the world coordinate system, that is, the position of the object in the image in the world coordinate system, L w Represents the transformation from the world coordinate system to the camera coordinate system.

[0068] Step 5: Use the display function to perform point cloud voxel filtering on all key frames in the world coordinate system to generate dense 3D point cloud mapping.

[0069] Specifically, the point cloud voxel filtering is performed on the key frames in all world coordinate systems to generate dense three-dimensional point cloud mapping. The specific process is: all key frames in the world coordinate system are merged into a global point cloud, and the global point cloud is filtered and displayed to generate a dense three-dimensional point cloud mapping.

[0070] In this embodiment, the synthesis operation process of dense 3D point cloud mapping is as follows: first, install the pcl1.10 library, call the CloudViewer class under the visualization namespace under pcl, and create a viewer object at the same time; call the VoxelGrid class under pcl, and create a voxel object at the same time; merge the multi-frame point clouds in the world coordinate system into a global point cloud; input the point cloud using the setInputCloud method of the voxel object, and filter it using the filter method; finally, display the point cloud using the showCloud method of the viewer object, thereby generating a dense 3D point cloud mapping; wherein the generated dense 3D point cloud mapping is as follows Figure 4 As shown, from Figure 4 As can be seen in the figure, using the dense mapping module for dense mapping can improve image quality and clarity; sparse mapping cannot provide enough pixels for semantic segmentation, so dense mapping is for semantic segmentation, which is also one of the results expected by the system.

[0071] The incremental semantic segmentation module is used to calculate the object category and probability of the image based on the key frame pose information and generate an RGB-D segmentation map using the trained incremental semantic segmentation model.

[0072] Specifically, the incremental semantic segmentation model is a SATS model, and the SATS model includes several encoding blocks and 1 decoding block, wherein the encoding block includes several attention heads, and the decoding block is a multi-layer perceptron.

[0073] In this embodiment, the SATS model is selected as the baseline model for incremental semantic segmentation. The model consists of 4 encoding blocks (TransformerBlock) as encoders to extract multi-dimensional features, and a multi-layer perceptron MLP as a decoder to fuse multi-level features. In addition, in the training process of the model, that is, the incremental learning stage, the category region pooling method and the unbiased knowledge distillation method are used to avoid catastrophic forgetting of old categories, and the cross entropy loss is used as the classification loss to supervise the model training. Among them, the category-specific region pooling method generates a self-attention vector representing the category by pooling the elements belonging to the same category in the self-attention map, thereby reducing the impact of noise and small changes; at the same time, in order to transfer the intra-class and inter-class relationships in the old model to the new model, SATS realizes the distillation of self-attention information by minimizing the self-attention distillation loss function. Among them, the simplified structure of the SATS model is as follows: Figure 5 shown.

[0074] Specifically, the training process of the trained incremental semantic segmentation model is:

[0075] Firstly, the training dataset is input into the SATS model, and the encoding block is used to extract multi-dimensional features, and the decoding block is used to fuse the multi-dimensional features. Meanwhile, in the encoding and decoding process, the category region pooling method is introduced to pool the elements of the same category in the trained self-attention map to generate the pooled class-specific self-attention vector, and the self-attention distillation loss function is used to distill the pooled class-specific self-attention vector. Finally, the cross loss function is introduced as the classification loss function. By constructing the total loss function to supervise the SATS model, the trained SATS model, that is, the trained incremental semantic segmentation model, is obtained.

[0076] In this embodiment, the training data set is the PASCAL VOC 2012 data set; wherein, the data set contains detailed annotations of 20 common types of objects, and the best model trained for it can achieve an intersection-over-union ratio of 90.5%; the present invention considers using it as a training data set for the incremental semantic segmentation model based on moderate data volume and high precision. At the same time, in the training process, that is, the incremental learning stage, only a small amount of new category data is required at each stage. Compared with the basic stage, the incremental stage can quickly adjust the model at a training speed 10 times faster, avoiding retraining of the full data set.

[0077] Specifically, the class-specific self-attention vector after pooling is:

[0078]

[0079] Among them, f i，j，b，h represents the pooling vector of the b-th element of the j-th encoding block in the i-th self-attention map in the h-th attention head, Mi，j，b The position set of the b-th element in the j-th encoding block in the i-th self-attention map, A i，j，h It represents the 3D self-attention map of the h-th attention head in the last layer of the j-th coding block, and u′ and v′ represent the horizontal and vertical coordinate positions of the b-th element, respectively.

[0080] Specifically, the self-attention distillation loss function is:

[0081]

[0082] Among them, L a (θ) represents the self-attention distillation loss, N represents the number of images in the training dataset, H represents the number of self-attention heads, and B i represents the set of categories appearing in the i-th image, represents the pooling vector of the b-th element in the i-th self-attention map at the h-th attention head at the t-th time, Represents the pooling vector of the b-th element in the i-th self-attention map at the h-th attention head at the t-1-th time.

[0083] In this embodiment, the self-attention distillation loss L a (θ) is used to transfer the intra-class and inter-class relations in the old model to the new model.

[0084] Specifically, the total loss function is:

[0085] L(θ)=L c (θ)+λ a L a (θ)+λ d L d (θ)

[0086] Among them, L(θ) represents the total loss, L c (θ) represents the cross entropy loss, λ a , d They represent the weight hyperparameters for adjusting self-attention distillation and knowledge distillation, L d (θ) denotes the unbiased knowledge distillation loss.

[0087] In this example, the unbiased knowledge distillation loss L d (θ) is used to transfer the knowledge output by the old model to the new model.

[0088] The fusion module is used to build maps based on dense 3D point clouds, perform point cloud reprojection, probabilistic fusion and point cloud update to generate a global semantic map.

[0089] In this embodiment, the structure of the fusion module is as follows: Figure 6 As shown, the implementation principle is as follows:

[0090] Specifically, the fusion module includes a service terminal module and a client terminal module;

[0091] The service submodule is used to receive the RGB-D segmentation map and the key frames transmitted by the client submodule, segment the key frames using the incremental semantic segmentation model, and transmit the segmented key frames to the client submodule.

[0092] The client submodule is used to receive dense 3D point cloud mapping and transmit key frames to the server through the TCP / IP protocol. At the same time, according to the segmentation key frames returned by the server, the segmentation key frames and dense 3D point cloud mapping are reprojected and probabilistically fused. Finally, the Bayesian algorithm is used to update the point cloud labels to generate a global semantic map.

[0093] In summary, the incremental semantic segmentation system based on visual SLAM proposed in the present invention designs an incremental semantic segmentation module for incremental learning, and improves the training speed of the trained incremental semantic segmentation model by reducing the amount of new category data at each stage, thereby avoiding retraining of a completely new data set; at the same time, a dense mapping module is designed to upgrade the sparse point cloud to a dense point cloud map, and to achieve more accurate three-dimensional semantic map construction by fusing geometric and semantic information; in addition, a fusion module is designed to break through the limitation of semantic segmentation only on two-dimensional images through mutual projection of three-dimensional and two-dimensional space, and to achieve semantic segmentation in three-dimensional space; finally, the system can also dynamically adapt to new object categories without retraining the entire incremental semantic segmentation model, thereby improving the interaction ability of the intelligent agent with the environment, such as the ability to achieve tasks such as object search and human-computer interaction.

[0094] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0095] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An incremental semantic segmentation system based on visual SLAM, characterized in that: include: The data preparation module is used to store high-precision depth information, RGB information and camera intrinsic parameter information generated by the sensor's image acquisition; The dense mapping module is used to perform key frame pose estimation on the high-precision depth information, RGB information and camera intrinsic parameter information generated by the image, generate key frame pose information and three-dimensional sparse point cloud coordinate information, and perform dense mapping thread calculation to generate dense three-dimensional point cloud mapping; The incremental semantic segmentation module is used to calculate the object category and probability of the image based on the key frame pose information and use the trained incremental semantic segmentation model to generate an RGB-D segmentation map; The fusion module is used to build maps based on dense 3D point clouds, perform point cloud reprojection, probabilistic fusion and point cloud update to generate a global semantic map.

2. The incremental semantic segmentation system based on visual SLAM according to claim 1, characterized in that: When the sensor collects images, the binocular camera and inertial measurement unit are jointly calibrated to generate high-precision depth information, RGB information and camera intrinsic parameter information.

3. The incremental semantic segmentation system based on visual SLAM according to claim 1, characterized in that: The key frame pose estimation is performed on the high-precision depth information, RGB information and camera intrinsic parameter information generated by the image, key frame pose information and three-dimensional sparse point cloud coordinate information are generated, and dense mapping thread calculation is performed. The specific process of generating dense three-dimensional point cloud mapping is as follows: Step 1: Create a new dense mapping thread; Step 2: Insert the key frames between the tracking thread and the local mapping thread into the new dense mapping thread; Step 3: According to the high-precision depth information, RGB information and phase intrinsic parameter information, the pixel coordinates of the key frame are converted into binocular camera coordinates, that is: Among them, X c , Y c , Z c represents the three-dimensional coordinates of the key frame in the binocular camera coordinate system, u and v represent the pixel coordinates of the key frame, that is, the horizontal and vertical coordinates of the key frame in the image, u0 and v0 represent the optical center coordinates of the binocular camera, that is, the horizontal and vertical coordinates of the center point of the image, Z c represents the distance between the object and the binocular camera, f x 、f y Respectively represent the focal length of the binocular camera in the x direction and the y direction; Step 4: According to the key frame pose information of the tracking thread, the binocular camera coordinates of the key frame are converted into world coordinates, that is: Among them, 1 represents the translation operation, that is, it is expanded to the fourth component of the homogeneous coordinates, R represents the 3×3 rotation matrix, that is, the rotation of the binocular camera relative to the world coordinate system, T represents the transpose, t represents the 3×1 translation vector, that is, the translation position of the binocular camera relative to the world coordinate system, X, Y, Z represent the three-dimensional coordinates of the key frame in the world coordinate system, that is, the position of the object in the image in the world coordinate system, L w Represents the conversion from the world coordinate system to the camera coordinate system; Step 5: Use the display function to perform point cloud voxel filtering on all key frames in the world coordinate system to generate dense 3D point cloud mapping.

4. The incremental semantic segmentation system based on visual SLAM according to claim 3, characterized in that: The specific process of performing point cloud voxel filtering on all key frames in the world coordinate system to generate dense 3D point cloud mapping is as follows: All key frames in the world coordinate system are merged into a global point cloud, and the global point cloud is filtered and displayed to generate a dense 3D point cloud map.

5. The incremental semantic segmentation system based on visual SLAM according to claim 4, characterized in that: The incremental semantic segmentation model is a SATS model, and the SATS model includes several encoding blocks and 1 decoding block, wherein the encoding block includes several attention heads, and the decoding block is a multi-layer perceptron.

6. The incremental semantic segmentation system based on visual SLAM according to claim 5, characterized in that: The training process of the trained incremental semantic segmentation model is: First, the training data set is input into the SATS model, the encoding block is used to extract multidimensional features, and the decoding block is used to fuse the multidimensional features; At the same time, during the encoding and decoding process, the category region pooling method is introduced to pool the elements of the same category in the trained self-attention map to generate a pooled class-specific self-attention vector, and the self-attention distillation loss function is used to distill the pooled class-specific self-attention vector. Finally, the cross loss function is introduced as the classification loss function. By constructing a total loss function to supervise the SATS model, the trained SATS model, that is, the trained incremental semantic segmentation model, is obtained.

7. The incremental semantic segmentation system based on visual SLAM according to claim 6, characterized in that: The class-specific self-attention vector after pooling is: Among them, f i,j,b,h represents the pooling vector of the b-th element of the j-th encoding block in the i-th self-attention map in the h-th attention head, M i,j,b The position set of the b-th element in the j-th encoding block in the i-th self-attention map, A i,j,h Represents the three-dimensional self-attention map of the h-th attention head in the last layer of the j-th coding block, and u' and v' represent the horizontal and vertical coordinate positions of the b-th element respectively.

8. The incremental semantic segmentation system based on visual SLAM according to claim 7, characterized in that: The self-attention distillation loss function is: Among them, L a (θ) represents the self-attention distillation loss, N represents the number of images in the training dataset, H represents the number of self-attention heads, and B i represents the set of categories appearing in the i-th image, represents the pooling vector of the b-th element in the i-th self-attention map at the h-th attention head at the t-th time, Represents the pooling vector of the b-th element in the i-th self-attention map at the h-th attention head at the t-1-th time.

9. The incremental semantic segmentation system based on visual SLAM according to claim 8, characterized in that: The total loss function is: L(θ)=L c (θ)+λ a L a (θ)+λ d L d (i) Among them, L(θ) represents the total loss, L c (θ) represents the cross entropy loss, λ a , d They represent the weight hyperparameters for adjusting self-attention distillation and knowledge distillation, L d (θ) denotes the unbiased knowledge distillation loss.

10. The incremental semantic segmentation system based on visual SLAM according to claim 9, characterized in that: The fusion module includes a service terminal module and a client terminal module; The service submodule is used to receive the RGB-D segmentation map and the key frames transmitted by the client submodule, segment the key frames using the incremental semantic segmentation model, and transmit the segmented key frames to the client submodule; The client submodule is used to receive dense 3D point cloud mapping and transmit key frames to the server through the TCP / IP protocol. At the same time, according to the segmentation key frames returned by the server, the segmentation key frames and the dense 3D point cloud mapping are reprojected and probabilistically fused. Finally, the Bayesian algorithm is used to update the point cloud labels to generate a global semantic map.

Images