A semantic mapping method and system based on visual SLAM and YOLOv5

Abstract

This invention provides a semantic mapping method and system based on visual SLAM and YOLOv5, achieving positioning optimization and globally consistent semantic mapping. The design first uses line feature optimization and weighted optimization strategies to obtain more accurate camera poses to improve mapping quality. It then integrates the YOLOv5 semantic segmentation network to obtain semantic information. A semantic map construction method combining facets and supersets is used to address semantic inconsistencies during the mapping process, ultimately constructing a globally consistent semantic map. This invention enables real-time tracking, mapping, and back-end optimization, resulting in high-precision pose estimation and globally consistent semantic map construction.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction and semantic map construction, and in particular to a semantic mapping method and system based on visual SLAM and YOLOv5. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) uses sensors onboard robots to achieve positioning, tracking, and mapping in unknown environments. Camera-based SLAM is known as visual SLAM. Compared to other sensors, cameras are easy to install, inexpensive, and offer rich data.

[0003] With the rapid development of science and technology, mobile robots have begun to be widely used in various industries. Simultaneous Localization and Mapping (SLAM) technology, in particular, provides solutions for robot navigation and obstacle avoidance in unknown environments, becoming a key technology in this field. However, traditional SLAM algorithms are unable to construct semantically informed environmental maps. This hinders robots from achieving a high-level semantic understanding of their environment when performing advanced environmental interaction tasks, thus limiting their development. Therefore, developing SLAM algorithms capable of constructing semantic environmental maps is a key step in improving robot intelligence.

[0004] Commonly used visual SLAM algorithms typically perform positioning by tracking features in images. For example, the state-of-the-art ManhattanSLAM achieves precise positioning by tracking point, line, and surface features and Manhattan constraints, using a surface model to construct a dense map of the environment. In ManhattanSLAM, point features provide the most stable constraints for pose estimation due to their stability and matching accuracy, while line and surface features not only provide additional constraints for pose optimization in complex environments, but also provide sufficient constraints for positioning in weakly textured environments where the number of point features is insufficient. However, the stability of line features is affected by feature extraction algorithms and depth map holes, which may introduce incorrect constraints and reduce system accuracy. Moreover, ManhattanSLAM cannot construct semantic maps containing semantic information and cannot provide information for the system's high-level environmental interaction tasks such as object recognition and positioning, and environmental identification. Existing visual SLAM algorithms that can construct environmental semantic information often fail to consider the semantic inconsistency problem during the mapping process, resulting in poor quality of constructed semantic maps. Summary of the Invention

[0005] The purpose of the present invention is to provide a semantic mapping method and system based on visual SLAM and YOLOv5 to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A semantic mapping method based on visual SLAM and YOLOv5, comprising:

[0008] In the positioning process, the line features are merged to improve the stability of the line features. The depth of the end points of the line features is optimized through clustering and straight line fitting to solve the missing depth of the end points of the line features caused by the depth map hole problem. The line feature matching method is improved to make the line feature matching more efficient and fast.

[0009] After completing stable matching, the algorithm jointly optimizes the camera pose by combining point-line-surface errors, adds weights to the point-line-surface error model, and obtains a more accurate pose.

[0010] By judging the feature points of the current frame and the inter-frame pose, a key frame is selected. After confirming as a key frame, the frame is transmitted to the back-end optimization and mapping thread;

[0011] In the mapping thread, first, the new key frame is subjected to semantic segmentation to obtain semantic information and construct a semantic graph, preparing for subsequent semantic mapping;

[0012] After obtaining the semantic graph, superpixel segmentation is performed according to the color graph, and then semantic information is obtained. Then, data is extracted from the face element database and the super set database to fuse the semantic superpixels. According to the remaining superpixels, a semantic face element is created, and after creation, super set fusion is performed on the newly created semantic face element. The semantic face element that fails to fuse is further subjected to super set creation. Finally, the super set is updated according to the face element data under its jurisdiction, and a globally consistent semantic map is constructed according to the super set information;

[0013] Preferably, after detecting the line features, unstable short line features are merged into more stable long line features according to the following constraints: 1) the angle between line features l1 and l2 is within a threshold. 2) the distance between line features l1 and l2 is within a threshold, and the distance between line features is judged by the distance between the midpoints of the line segments. 3) to prevent line features from being in a containing relationship, set the projections of the two line features on the XY axis to be non-repetitive.

[0014] Preferably, the depth missing problem in the process of three-dimensionalization of line features is optimized to obtain more accurate depth values. Specifically, for the detected line features, first, all depth-bearing pixel points are obtained in combination with the depth map and projected into the camera coordinate system to obtain three-dimensional points. After obtaining the three-dimensional points, distance clustering is performed, and then straight line fitting is performed according to the clustering results. Finally, the depth values of the end points of the line features are obtained according to the straight line fitting results.

[0015] Preferably, a matching method based on point-to-line distance constraints is provided. The specific steps are as follows: 1) Line feature sampling, i.e., uniformly sampling nine points on the line feature to be matched. These sampling points can be selected at equal intervals on the line feature to ensure that different parts of the line feature are covered. 2) Back-projection to the image, i.e., back-projecting the sampling points onto the image plane to obtain the corresponding pixel coordinates. This can be calculated using the camera's intrinsic parameter matrix and depth information. 3) Distance calculation, i.e., for each sampling point, calculating its average distance to the matching line segment. This can be obtained by accumulating the distance between the sampling point and the matching line segment and dividing it by the number of sampling points to obtain the average distance. 4) Matching judgment, i.e., comparing the average distance with a pre-set threshold. If the average distance is less than the threshold, the line feature is considered a match. Through these four steps, the algorithm can achieve fast and stable line feature matching while ensuring matching accuracy.

[0016] Preferably, since point feature matching is more accurate and stable, its error term is given a higher weight. To ensure the real-time performance of the system, avoid complex weight setting methods. Point, line and surface weight w p 、w l and w φ Number of matches N based on point features p The allocation is set up as shown in the formula:

[0017]

[0018] The threshold N is set to 40. The Levenberg-Marquardt algorithm is used to minimize the following energy function to optimize the camera pose:

[0019]

[0020] Where {R* cw,t* cw} is the current camera pose, which is the optimal solution; N p 、N l and N φ are the number of point feature matches, line feature matches, and surface feature matches of the current frame respectively; w p 、w l and w φ is the weight of the error term; H p 、H l and H φ is the Huber robust loss function; e pi 、e lj and e φk are the errors of points, lines and surfaces respectively; Σ p ,Σ l and Σ φ Represents the covariance matrix of points, lines, and areas.

[0021] Preferably, the key frame is input into the trained YOLOv5s-seg model, and the model outputs a set of semantic information. Each semantic information contains the category to which the object belongs, the confidence, the prediction box, and the segmentation mask. According to the output information, a semantic graph is constructed to save semantics. The semantic graph is set as a double-channel image, the first channel stores the semantic category C, and the second channel stores the semantic confidence P.

[0022] Preferably, based on the face element-based dense map construction method, a dense semantic map is constructed in combination with semantic information. The mapping method reconstructs a globally consistent semantic map by inputting the color image, the depth image, and the frame pose of the system.

[0023] Preferably, instead of creating a face element for each pixel, the color image is first segmented into superpixels, and then the superpixels are constructed into face elements. This not only greatly reduces the computational load of the system, but also reduces the influence of abnormal values and depth holes in the low-quality depth image on the mapping. An improved SLIC method is used to obtain superpixels, and semantic information is added to each superpixel. For each superpixel, the initial category C sp is defined as a background object, and the confidence P sp is set to 0.2. During the superpixel segmentation process, the semantic category of the superpixel is updated to the semantic category with the most number of pixels within the superpixel range, and the confidence is set to the confidence of that category. The confidence update is shown in the formula:

[0024]

[0025] where C TH represents that the semantic category is an object; C BG represents that the semantic category is a background; P c (*) is the confidence of the corresponding category; and MaxNum(*) is the semantic category with the most number of pixels in the range.

[0026] Preferably, semantic information is also added to the face element, and the initial category C sl is set to a background object, and the confidence P sl is set to 0.2. In addition, the ordinal number of the super set to which each face element belongs is also added to establish the relationship between the face element and the super set. After completing the superpixel segmentation with semantics, the face elements in the system are used to fuse these superpixels, and the face elements and the corresponding super sets are updated.

[0027] Further, whether to fuse the face element and the superpixel is determined by the physical distance and the normal vector angle between them. If the condition is met, the attributes of the face element are updated jointly with the superpixel. Among them, the semantic-related attributes are determined by comparing the semantic confidence P sl of the semantic face element and the semantic confidence P sp of the superpixel. When P sp > Psl When C sl =C sp 、P sp =P sl When the semantic information of a facet changes, the superset that controls the facet is updated. First, determine whether the facet is fused into the superset. If not, create a new superset based on the facet metadata. The category of the newly created superset is defined as the semantic category of the facet, and the corresponding confidence PC st in the confidence list is updated as: PC st = PC st + P sl , radius R st Set to the surface radius R sl The superset position is set to the position of the facet. If the facet has been fused, only the semantic confidence attribute of the corresponding superset is updated.

[0028] Optionally, new surfels are created based on unfused superpixels. The attributes of the new surfels are set to the same attributes as the superpixels. To control the semantic information of these surfels, the new surfels are fused using the superset in the system. Simultaneously, the superset needs to be updated to reflect the presence of these new surfels.

[0029] Furthermore, after creating a new facet, it is fused with the facets in the system. When the following conditions are met at the same time, the facet is fused with the corresponding superset:

[0030] Preferably, the bin class C sl For superset category C st or background C BG , that is: C sl ∈{C st ,C BG};

[0031] Preferably, the distance D between the facet and the superset is smaller than the radius R of the superset. st Add the radius R of the bin sl , that is: D <R st +R sl ;

[0032] Alternatively, if the new facet does not simultaneously meet the above conditions with any of the supersets, and the new facet's category is not background, a new superset is created based on the new facet. The steps for creating a new superset are the same as those for facet fusion and are not detailed here. If the above conditions are met, the semantic confidence attribute of the superset is updated based on the facet.

[0033] Furthermore, after the facet update is completed, all supersets are updated. Specifically, the algorithm updates the center position, radius R of all supersets. st and category C stAttribute. For a superset, traverse all the patches under its jurisdiction, and record the maximum radius, the maximum and minimum values in XYZ directions respectively. Update the superset using the collected data, specifically, update the center position of the superset as:

[0034]

[0035] Update the radius of the superset as:

[0036]

[0037] Update the semantic category of the superset as:

[0038]

[0039] PC st is the confidence of a certain category among all categories.

[0040] In the mapping process, when the semantic segmentation results of the same object are inconsistent, the semantic of the object patch constructed by the application may change, but the semantic information of the superset to which the object patch belongs will not change suddenly. Moreover, the semantic information of the patch in the map is provided by the superset, so the global consistency of the semantic of the object is maintained.

[0041] On the other hand, to achieve the above object, the application further provides a semantic mapping system of visual SLAM and YOLOv5, comprising:

[0042] Line feature merging module: merging the detected short line features into long line features.

[0043] Line feature endpoint depth optimization module: fitting a straight line according to the points with depth on the line feature, and obtaining the depth of the line feature endpoint through the fitted straight line.

[0044] Line feature matching module: judging the matching degree of line features by calculating the geometric constraints between line features.

[0045] Weighted optimization module: adding weights to the optimization items participating in pose optimization, and the weights dynamically change.

[0046] Global consistent semantic mapping module: constructing a semantic map through image data and pose data, and maintaining semantic consistency in the mapping process.

[0047] Preferably, the semantic mapping module comprises:

[0048] First point superpixel segmentation: performing superpixel segmentation on the input image data, and constructing semantic superpixels by assigning semantics to the superpixels;

[0049] Second point patch fusion: creating patch fusion through superpixels, and updating patch data and superset data at the same time.

[0050] Third supersets update: update the supersets information through the subordinated face element of the supersets.

[0051] Compared with the prior art, the present application has the following advantages and technical effects:

[0052] (1) The present application provides a semantic mapping method based on visual SLAM and YOLOv5, which first obtains more accurate camera pose through line feature optimization and weighted optimization strategy to improve mapping quality; then combines YOLOv5 semantic segmentation network to obtain semantic information, uses a semantic map construction method combining face element and superset to solve the semantic inconsistency problem in the mapping process, and finally constructs a globally consistent semantic map. Experimental verification shows that the proposed algorithm not only exceeds the ManhattanSLAM algorithm in accuracy, but also constructs a semantic map superior to other advanced mapping algorithms.

[0053] (2) The present application is based on ManhattanSLAM and aims at the instability of line feature extraction and matching, and performs line feature related optimization; weights are added for optimization to obtain more accurate pose; the YOLOv5s-seg semantic segmentation model is combined to perform semantic segmentation on key frames to obtain semantic information of the surrounding environment while maintaining algorithm efficiency; to solve the influence of excessive segmentation or non-segmentation of semantics on mapping, a new incremental semantic map construction method is proposed, which uses superset as the semantic controller of face element after semantic regularization of superset. Experimental verification shows that the present application not only obtains high-precision pose, but also constructs a globally consistent semantic map of the environment.

[0054] (3) The present application improves line feature extraction and matching, adds weights for optimization, and obtains better positioning accuracy and robustness than other advanced algorithms. The present application obtains higher positioning accuracy than other advanced algorithms; in terms of mapping, the present application not only reconstructs a high-quality semantic map, but also effectively handles the semantic inconsistency problem in the mapping process. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0056] Figure 1 The semantic mapping method based on visual SLAM and YOLOv5 provided by the present application is shown in the flowchart.

[0057] Figure 2A semantic mapping method based on visual SLAM and YOLOv5 according to an embodiment of the present application;

[0058] Figure 3 A line feature endpoint depth optimization flowchart according to an embodiment of the present application;

[0059] Figure 4 A global consistent semantic mapping method architecture schematic diagram according to an embodiment of the present application;

[0060] Figure 5 A patch fusion update flowchart according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] The present embodiment provides a semantic mapping method based on visual SLAM and YOLOv5, the overall flow is as shown in Figure 1 , the overall algorithm framework is as shown in Figure 2 , and the algorithm flow includes:

[0062] In the positioning process, the line features are merged to improve the stability of the line features. The depth of the line feature endpoints is optimized through clustering and straight line fitting to solve the missing of the depth of the line feature endpoints caused by the depth map hole problem. The line feature matching method is improved to make the line feature matching more efficient and fast.

[0063] Specifically, after detecting the line features, the unstable short line features are merged into more stable long line features according to the following constraints: 1) the angle between the two-dimensional line features l1 and l2 is within a threshold. 2) the distance between the two-dimensional line features l1 and l2 is within a threshold, and the distance between the line features is judged by the distance between the midpoints of the line segments. 3) to prevent the line features from being in a containing relationship, set the projections of the two line features on the XY axis to be non-repetitive.

[0064] Further, the depth missing problem in the process of three-dimensionalization of the line features is optimized to obtain more accurate depth values, and the flow is as shown in Figure 2 . Specifically, for the detected line features, first, all depth points are obtained by combining the depth map and projecting them into the camera coordinate system to obtain three-dimensional points. After obtaining the three-dimensional points, distance clustering is performed, and then straight line fitting is performed according to the clustering results. Finally, the depth values of the line feature endpoints are obtained according to the straight line fitting results.

[0065] Furthermore, a matching method based on point-to-line distance constraints is used to improve line feature matching, making it more efficient and faster. The specific steps are as follows: 1) Line feature sampling: uniformly sampling nine points on the line feature to be matched. These sampling points can be selected at equal intervals on the line feature to ensure coverage of different parts of the line feature. 2) Back-projection to the image: back-projecting the sampling points onto the image plane to obtain the corresponding pixel coordinates. This can be calculated using the camera's intrinsic parameter matrix and depth information. 3) Distance calculation: for each sampling point, calculating the average distance to the matching line segment. This is achieved by summing the distances between the sampling points and the matching line segment and dividing the distance by the number of sampling points. 4) Match determination: comparing the average distance with a pre-set threshold. If the average distance is less than the threshold, the line feature is considered a match. Through these four steps, the algorithm achieves fast and stable line feature matching while ensuring matching accuracy.

[0066] After completing a stable match, the algorithm will combine point, line, and surface errors to jointly optimize the camera pose, adding weights to the point, line, and surface error model to obtain a more accurate pose.

[0067] Specifically, after completing the stable matching, the error term is given a higher weight because the point feature matching is more accurate and stable. To ensure the real-time performance of the system, avoid complex weight setting methods. p 、w l and w φ Number of matches N based on point features p The allocation is set up as shown in the formula:

[0068]

[0069] The threshold N is set to 40.

[0070] Furthermore, the Levenberg-Marquardt algorithm is used to minimize the following energy function to optimize the camera pose:

[0071]

[0072] where {R * cw ,t * cw} is the current camera pose, which is the optimal solution; N p 、N l and N φ are the number of point feature matches, line feature matches, and surface feature matches of the current frame respectively; w p 、w l and w φ is the weight of the error term; H p 、Hl and H φ are Huber robust loss functions for point, line and plane respectively; e pi , e lj and e φk are errors for point, line and plane respectively;∑ p ,∑ l and∑ φ represent covariance matrices for point, line and plane.

[0073] Key frames are selected by judging the feature points of the current frame and the inter-frame pose. After being confirmed as a key frame, the frame is transmitted to the backend optimization and mapping thread, and the mapping method architecture is shown in Figure 4 ;

[0074] Specifically, in this example, the key frame is transmitted to the trained YOLOv5s-seg model, and the model outputs a set of semantic information. Each semantic information contains the object class to which the object belongs, the confidence, the prediction box and the segmentation mask. According to the output information, a semantic graph is constructed to save semantics. The semantic graph is set as a double-channel image, the first channel stores the semantic class C, and the second channel stores the semantic confidence P.

[0075] In the mapping thread, first, the semantic information is obtained by performing semantic segmentation on the new key frame, and a semantic graph is constructed to prepare for the subsequent semantic mapping;

[0076] Specifically, on the basis of the face element-based dense map construction method, a dense semantic map is constructed in combination with the semantic information. The mapping method reconstructs a globally consistent semantic map by inputting the color image, the depth image and the frame pose of the system. Unlike creating a face element for each pixel, the color image is first segmented into superpixels, and then the superpixels are constructed into face elements. In this way, not only the computational load of the system can be greatly reduced, but also the influence of abnormal values and depth holes in the low-quality depth image on the mapping can be reduced. An improved SLIC method is used to obtain superpixels, and semantic information is added to each superpixel. For each superpixel, the initial definition of its class Csp is a background object, and the confidence Psp is set to 0.2. During the superpixel segmentation process, the semantic class of the superpixel is updated to the semantic class with the most number of pixels within the range of the superpixel, and the confidence is set to the confidence of the class. The confidence update is shown in the formula:

[0077]

[0078] where C is the semantic class, C TH represents that the semantic class is an object; C BG represents that the semantic class is background; P c (*) is the confidence of the corresponding class; MaxNum(*) is the semantic class with the most number of pixels in the range.

[0079] Furthermore, semantic information is added to the facets, and their initial categories are set to C sl is the background object, and the confidence P sl The value of α is set to 0.2. Furthermore, the sequence number of the superset to which each surface element belongs is added to establish a connection between the surface element and the superset. After completing the semantic superpixel segmentation, the superpixels are fused using the surface elements in the system, and the surface element and the corresponding superset are updated simultaneously.

[0080] The update process of the facet is as follows: Figure 5 As shown in Figure 1, first, the physical distance and normal vector angle between the surface element and the superpixel are used to determine whether to fuse the two. If the conditions are met, the superpixel is combined to update the various attributes of the surface element. Among them, the semantically related attributes are updated by comparing the semantic confidence P of the semantic surface element. sl and the semantic confidence P of the superpixel sp To determine. When P sp >P sl When C sl =C sp 、P sp =P sl When the semantic information of a facet changes, the superset that controls the facet is updated. First, determine whether the facet is fused into the superset. If not, create a new superset based on the facet metadata. The category of the newly created superset is defined as the semantic category of the facet, and the corresponding confidence PC st in the confidence list is updated as: PC st = PC st + P sl , radius R st Set to the surface radius R sl The superset position is set to the position of the facet. If the facet has been fused, only the semantic confidence attribute of the corresponding superset is updated.

[0081] Furthermore, for unfused superpixels, new surfels are created based on them. The attributes of these new surfels are set to the corresponding attributes of the superpixels. To control the semantic information of these surfels, the new surfels are fused using the superset in the system. Simultaneously, the superset needs to be updated to reflect the presence of these new surfels.

[0082] After creating a new facet, it is fused with the facets in the system. When the following conditions are met at the same time, the facets are fused with the corresponding superset:

[0083] (1) Facet category C sl For superset category C st or background C BG , that is: C sl ∈{C st ,C BG};

[0084] (2) The distance D between the face and the superset is less than the radius R of the supersetst Radius of the facet R sl That is: D < R st + R sl ;

[0085] If the new facet does not satisfy the above conditions with all the supersets, and the category of the new facet is not background, a new superset is created according to the new facet. The creation step here is the same as creating a new superset when the facets are fused, and thus will not be described again. When the above conditions are satisfied, the semantic confidence attribute of the superset is updated according to the facet.

[0086] Further, after the facet update is completed, all supersets are updated. Specifically, the algorithm updates the center position, radius R st and the category C st of the superset. For a superset, all the facets under its jurisdiction are traversed, and the maximum radius, the maximum and minimum values in the XYZ directions are recorded respectively. The collected data is used to update the superset, specifically, the center position of the superset is updated as follows:

[0087]

[0088] The radius of the superset is updated as follows:

[0089]

[0090] PC st is the confidence of a category in all categories; MaxX is the maximum value in the X direction, MinX is the minimum value in the X direction, MaxY, MinY, MaxZ and MinZ are the same.

[0091] In the mapping process, when the semantic segmentation results of the same object are inconsistent, the semantic of the object facet constructed by the present application will change, but the superset semantic information will not change suddenly. Moreover, the semantic information of the facet in the map is provided by the superset, so the global consistency of the object semantic is maintained.

[0092] After the semantic map is obtained, superpixel segmentation is performed according to the color map, and then the superpixels with semantic information are obtained in combination with the semantic map. Then, the data of the facets and the supersets are extracted from the facet database and the superset database to fuse the semantic superpixels. The semantic facets are created according to the remaining superpixels, the newly created semantic facets are fused by the superset, and the semantic facets that fail to be fused are fused according to the superset. Finally, the supersets are updated according to the facet data under the jurisdiction of the superset, and the globally consistent semantic map is constructed according to the superset information. The complete process of the algorithm is shown in Figure 1 .

[0093] The above merely provides the preferred embodiments of the present application, and the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A semantic mapping method based on visual SLAM and YOLOv5, characterized in that: The semantic mapping method comprises the following steps: Step 1: During the positioning process, line features are merged to improve the stability of line features: first, all pixel points with depth on the line feature are obtained in combination with the depth map and projected into the camera coordinate system to obtain three-dimensional points. Then, the three-dimensional points are clustered to exclude noise points. Finally, Ransac line fitting is performed based on the clustering results, and the depth of the line feature endpoint is obtained based on the fitted line. A matching method based on point-to-line distance constraints is used, and the LBD line feature descriptor that is easily affected by illumination is not used. The specific steps are to uniformly sample nine points on the line feature to be matched, back-project the sampling points onto the image plane, and obtain the corresponding pixel coordinates. For each sampling point, the distance between the sampling point and the matching line segment is accumulated and divided by the number of sampling points to obtain the average distance. The average distance is compared with a pre-set threshold. If the average distance is less than the threshold, the line feature is considered to be matched. Step 2: After completing a stable match, the algorithm will combine point-line-plane errors to jointly optimize the camera pose, adding weights to the point-line-plane error model to obtain a more accurate pose. Step 3: Select key frames by judging the feature points of the current frame and the inter-frame pose. After confirming them as key frames, pass the frames to the backend optimization and mapping thread. Step 4: Pass the keyframe into the trained YOLOv5s-seg model. The model outputs a set of semantic information. Each semantic information contains the object category, confidence, prediction box and segmentation mask. Based on the output information, a semantic map is constructed to preserve the semantics. The semantic map is set as a two-channel image. The first channel stores the semantic category C and the second channel stores the semantic confidence P. Step 5: After obtaining the semantic map, perform superpixel segmentation based on the color map, and then combine it with the semantic map to obtain superpixels with semantic information; Step 6: Extract data from the facet database and the superset database to fuse semantic superpixels: Create semantic facets based on the remaining superpixels. After creation, perform superset fusion on the newly created semantic facets, and create supersets based on the semantic facets that failed to be fused. Step 7: Update the superset based on the facet metadata under the superset, and build a globally consistent semantic map based on the superset information.

2. A semantic mapping method based on visual SLAM and YOLOv5 according to claim 1, characterized in that, The process of merging line features described in step 1 is as follows: The split short line features are merged and restored into long line features according to the direction constraint and the distance constraint, and after the line features are merged, the short line features with smaller length are filtered out.

3. A semantic mapping method based on visual SLAM and YOLOv5 according to claim 1, characterized in that, The method of combining point, line and surface errors described in step 2 includes: The camera pose is optimized by combining the reprojection error constraints of feature points, the reprojection distance constraints of line feature endpoints, and the matching constraints of planes to obtain a more accurate pose, and different weights are assigned to the constraints constructed by point, line, and surface features.

4. A semantic mapping method based on visual SLAM and YOLOv5 according to claim 1, characterized in that, Constructing the semantic graph described in step 4 includes: The facet-based dense map construction method uses facets of different sizes to reconstruct the surface of real objects: in the constructed map, real objects can be represented as a set of single or multiple adjacent facets, and these facets should have consistent semantic labels; based on this, a superset is defined as a set consisting of one or more facets. Objects in the real world are represented in the map by one or more adjacent supersets, and the facets in the superset have consistent semantic labels.

5. A semantic mapping method based on visual SLAM and YOLOv5 according to claim 1, characterized in that, The superpixel segmentation based on the color image described in step 5 includes: First, the color image is segmented into superpixels, and then the surface elements are constructed from the superpixels.

6. A semantic mapping method based on visual SLAM and YOLOv5 according to claim 1, characterized in that, The superset fusion described in step 6 includes: The semantic facet update process is an incremental map creation and update process, the purpose of which is to integrate the local map created in this frame into the global map: first, the facets and superpixels are fused. If the conditions are met, the various attributes of the facets are updated in conjunction with the superpixels. If not, new facets are created based on the superpixels. After the facets are created, super-fusion is performed based on the relationship between the facets and the superset. Finally, the superset fusion result is used to determine whether to update the superset with reverse mapping or to create a new superset. In the fusion of facets and superpixels, the physical distance and normal vector angle between the facets and superpixels are used to determine whether to fuse the two.

7. A semantic mapping method based on visual SLAM and YOLOv5 according to claim 1, characterized in that, The superset update described in step 7 includes: According to the attributes of the facets under the superset, the average geometric attributes are calculated and the position, size and semantic attributes of the superset are updated respectively.

8. A semantic mapping system based on visual SLAM and YOLOv5, applied to the method according to any one of claims 1 to 7, characterized in that: include: Line feature merging module: merges the detected short line features into long line features; Line feature endpoint depth optimization module: Fit a straight line based on the points with depth on the line feature, and obtain the line feature endpoint depth through the fitted straight line; Line feature matching module: determines the matching degree of line features by calculating the geometric constraints between line features; Weighted optimization module: adds weights to the optimization items involved in pose optimization, and the weights change dynamically; Globally consistent semantic mapping module: Constructs semantic maps through image data and pose data, maintaining semantic consistency during the mapping process.

9. A semantic mapping system based on visual SLAM and YOLOv5 according to claim 8, characterized in that, The globally consistent semantic mapping module includes: The first superpixel segmentation unit: performs superpixel segmentation on the input image data, and then assigns semantic meaning to the superpixels to construct semantic superpixels; Second point surface element fusion unit: Create surface elements through superpixels and update the original surface data and superset data at the same time; The third point is the superset update unit: it updates the superset information through the facets under the superset.

Images