A lightweight visual semantic odometry method for dynamic scenes

Abstract

A lightweight visual semantic odometry method for dynamic environments is proposed. First, RGB and depth images are acquired and sent to the ORB feature extraction thread, semantic segmentation thread, and depth contour detection thread, respectively. Second, static and dynamic prior targets are used, and the boundary expansion result of the depth contour mask is awaited. Simultaneously, based on the depth gradient of the depth image, the approximate shape of the target is determined, and a depth mask is drawn. Then, the depth mask is dilated using morphological processing to expand the mask boundary blur caused by camera precision, and pixel-level image edge fusion is performed with the semantic segmentation result of the dynamic target to obtain a corrected mask. Finally, feature points within the dynamic mask are removed from the grayscale region of the dynamic target based on the corrected mask, and the remaining static feature points are combined with semantic labels to construct a semantic point cloud map. This invention improves the real-time performance and semantic feature extraction efficiency of semantic SLAM.

Description

Technical Field

[0001] This invention relates to the field of robot localization technology, and more specifically, to a lightweight visual semantic odometry method for dynamic environments. Background Technology

[0002] Path planning, localization and navigation, and obstacle avoidance are the three key technologies in mobile robot technology. Among them, Simultaneous Localization and Mapping (SLAM) is one of the research directions in localization and navigation technology. It is divided into several categories, such as laser SLAM, visual SLAM, multi-sensor fusion SLAM, and semantic SLAM. Semantic SLAM technology, with its advantage of combining deep learning technology for scene understanding, constructs semantic maps, which greatly improves the robot's localization accuracy.

[0003] Most common semantic SLAM algorithms rely on computer vision combined with deep learning algorithms, such as semantic segmentation, instance segmentation, and object detection, to extract scene semantic information. These algorithms often suffer from drawbacks such as large parameter counts and low real-time performance, requiring higher computational power for real-time operation. Therefore, based on these issues, more and more researchers are investing in lightweight semantic segmentation networks, striving to improve their accuracy and real-time performance in scene understanding. This provides a good foundation for their application in semantic SLAM, facilitating improved real-time performance. However, compared to traditional semantic segmentation networks, lightweight semantic segmentation networks inevitably reduce segmentation accuracy, leading to decreased segmentation accuracy for specific targets in dynamic scenes. This results in some noise points in the map that belong to dynamic targets. Some researchers have adopted brute-force removal methods, achieving some beneficial results, but some noise points still leak into the map. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a lightweight visual semantic odometry method for dynamic environments, which improves the real-time performance and semantic feature extraction efficiency of semantic SLAM, utilizes a lightweight semantic segmentation network to improve the efficiency of semantic information acquisition, and solves the problem of dynamic noise point leakage caused by semantic segmentation networks, thereby improving the real-time performance and positioning accuracy of semantic SLAM.

[0005] The technical solution proposed to solve the above-mentioned technical problems is as follows:

[0006] A lightweight visual semantic odometry method for dynamic environments, the method comprising the following steps:

[0007] Step (1): Acquire RGB and Depth images using an RGB-D camera, and send the RGB images to the ORB feature extraction thread and semantic segmentation thread respectively, and send the Depth image frames to the depth contour detection thread.

[0008] Step (2): The ORB feature extraction thread extracts the ORB features of each frame of the RGB image sent in, detects dynamic feature points according to the epipolar constraint, and then waits for the dynamic target segmentation result of the semantic segmentation thread.

[0009] Step (3): The semantic segmentation thread first uses a lightweight semantic segmentation network to segment each frame of the transmitted RGB image, obtains the segmented image mask, and waits for the depth mask result from the depth contour detection thread.

[0010] Step (4): The depth contour detection thread obtains the depth gradient by traversing the depth value of each pixel, thereby determining targets with similar depths and generating a depth mask. Then, based on morphological processing, the depth mask is dilated to expand the mask boundary blur caused by camera precision and sent to the image boundary fusion module.

[0011] Step (5): The image boundary fusion module receives the scene segmentation result and the depth mask after the depth contour expansion from the lightweight semantic segmentation network, and fuses the mask based on the weighted average method to achieve dynamic mask correction.

[0012] Step (6): Using the mask corrected in step (5) and the dynamic feature points initially selected in step (2), determine the final dynamic feature points and remove them. Input the remaining static feature points into the visual odometry module to construct a semantic point cloud map.

[0013] Furthermore, the lightweight visual semantic odometry for dynamic environments implemented by the method includes a tracking module, a visual odometry module, and a semantic mapping module. The process of step (1) is as follows: The tracking module is specifically divided into three sub-threads: ORB (Oriented FAST and Rotated BRIEF) feature extraction thread, semantic segmentation thread, and depth contour detection thread, and the three threads run in parallel; wherein, RGB image frames are sent to the ORB feature extraction thread and the semantic segmentation thread respectively, and the depth image frames are sent to the depth contour detection thread.

[0014] Preferably, the process of step (2) is as follows: Detect dynamic feature points based on RGB image frames, match the feature points extracted from the current image frame Ic and the previous frame Ip, and solve the homography matrix H between the two frames using the Random Sample Consensus (RANSAC) algorithm and the Levenberg-Marquarelt (LM) algorithm, assuming the pixel set Ppn It is a point on IP, and its expression is:

[0015] P pn =[u pn v pn 1] T n = 1, 2, 3, 4, 5, ..., k (1)

[0016] Among them, P pn u represents the coordinates of the set of points on the previous frame image Ip. pn and v pn Let be the x and y coordinates of a point, n be the index of the point, k be a positive integer, and the coordinates of the corresponding set of points on Ic are...

[0017] P cn =[u cn v cn 1] T n = 1, 2, 3, 4, 5, ..., k (2)

[0018] Among them, P cn u represents the coordinates of the set of points on the current frame image Ic. cn and v cn Let x and y be the x and y coordinates of the point.

[0019] P is obtained from the homography matrix H. cn for

[0020] P cn =HP Pn ,n=1,2,3,4,5,...,k (3)

[0021] Thus, the camera pose between the previous frame image Ip and the current frame image Ic is obtained by solving the least squares problem:

[0022]

[0023] Among them, P pi and P ci Corresponding to P respectively pn and P cn The point with index i is given above. R and t represent the rotation matrix and translation vector between the two frames, respectively, and N is a positive integer.

[0024] To obtain dynamic feature points, firstly, matching points are extracted from Ip and Ic, and the fundamental matrix F between the two frames is calculated using the RANSAC algorithm; then, the epipolar line in the current frame is calculated using the fundamental matrix F. Therefore, point sets P are selected respectively. pn and P cn A pixel P on p1 =[u p1 vp1 1] T and P c1 =[u c1 v c1 1] T And P p1 ∈P pn P c1 ∈P cn Calculate the polar line l p1 :

[0025]

[0026] Where X, Y, and Z represent the inline matrix, i.e., the x, y, and y coordinates of point P, and F is the base matrix. Then, we calculate the distance from the feature point to the corresponding epipolar line:

[0027]

[0028] Where d represents the distance from the feature point to the corresponding epipolar line.

[0029] The point sets in the current frame and the previous frame are traversed sequentially. When d is greater than the threshold ε we set, it is a dynamic feature point and is stored in the dynamic point set, waiting for the semantic segmentation mask.

[0030] Preferably, the process of step (3) is as follows: the semantic segmentation thread receives the sent RGB image frame and inputs it into the semantic segmentation network for semantic feature extraction. In order to improve the efficiency of semantic information acquisition, this method introduces the lightweight semantic segmentation network TopFormer to extract semantic features, and converts the trained model into an ONNX model, using CUDA to accelerate inference based on the ONNXRuntime-GPU version;

[0031] This method introduces TopFormer, a lightweight semantic segmentation network based on TransFormer. Its miniaturized version can produce competitive real-time inference results on ARM-based mobile devices. The model is trained using the ADE20K dataset, which provides 150 classification categories, adequately handling indoor scene segmentation tasks. For dynamic indoor scenes, this method categorizes common indoor objects into static, dynamic, and movable objects. Static objects include beds, tables, and televisions; dynamic objects include people; and movable objects include chairs, books, and vases. The method transforms the input RGB image to 512×512×3 before inputting it into the semantic segmentation network, outputting a color mask and grayscale image of the same size. It also detects the grayscale values ​​of dynamic prior objects, determining whether a dynamic object exists in the current frame based on these values. If a dynamic object is found, the flag indicating the presence of a dynamic object is incremented to 1.

[0032] Preferably, step (4) is performed as follows: the depth contour detection thread receives the sent Depth image frame and creates a depth mask. Due to the limitations of the depth camera's accuracy, to prevent boundary noise point leakage and unclear ranging of small objects, this method is divided into two parts: coarse traversal and fine traversal. First, a coarse traversal is performed using a 30×30 slider with a step size of 30 to traverse the Depth image and record the depth values ​​in the slider:

[0033] D=d(x:x+29,y:y+29) (7)

[0034] Where D represents the depth value of the corresponding pixel, and x and y represent the horizontal and vertical coordinates of the corresponding pixel, respectively.

[0035] For boundary pixels, the slider can change according to the size of the boundary pixel block, such as... Figure 4 As shown, that is

[0036]

[0037] Where n is the length of the slider, m is the width of the slider, and Edge is the set of boundary pixels.

[0038] After traversing a depth image, depth images within consecutive and similar depth gradient ranges are grouped into one category:

[0039] Obj(n),n=1,2,3,4,……,k (9)

[0040] Here, Obj(n) represents the category.

[0041] Next, the image is denoised to remove isolated image patches smaller than 1000 pixels. Then, a fine traversal is performed. Based on the depth values ​​obtained from the coarse traversal, the dynamic prior mask obtained from semantic segmentation is used to further label the depth pixels, assigning labels to depth image patches (including dynamic and movable objects). Furthermore, the depth images obtained from the coarse traversal are classified, and sub-pixel boundaries of depth image patches with prior dynamic target labels are extracted based on the gray-level centroid method to further clarify the target mask boundaries, as follows:

[0042]

[0043] Where I(x, y) represents the grayscale image, (x, y) represents the pixel coordinates of the image, S represents the target to be extracted, and (x0, y0) represents the grayscale centroid coordinates; thus, the dynamic target depth mask after refining the boundaries is obtained.

[0044] Finally, the depth mask is further extended using morphological dilation to prevent the leakage of dynamic noise points. The depth image after refining the boundaries is converted to grayscale, and the image patches of dynamic targets are traversed, with outward expansion of a certain distance until a threshold τ is reached.

[0045] Preferably, step (5) is performed as follows: After the dynamic target depth mask from step (4) is obtained, the semantic segmentation image from step (3) and the depth image from step (4) are converted to the same size. The semantic segmentation image and the depth image are traversed, and the mask coordinates of the dynamic target located in the image patch are recorded respectively. The two images are then fused using the group averaging method.

[0046] F(x,y)=ω a ·A(x,y)+ω b ·B(x,y) (11)

[0047] Where, ω a ω b The weighting coefficients are A(x,y) and B(x,y), which are the coordinates of the depth mask and the original mask, respectively.

[0048] Because the TopFormer semantic segmentation network does not clearly segment object boundaries, and it suffers from mask dilation when segmenting people,

[0049] Therefore, we take the intersection of the two, using the depth mask boundary as the reference:

[0050]

[0051] Among them, M f For the final corrected mask, and It is divided into the original mask and the depth mask for dynamic targets.

[0052] Preferably, step (6) is performed as follows: This invention combines geometric feature detection and semantic information to remove dynamic feature points, and uses the remaining static feature points to construct a three-dimensional point cloud semantic map. If the dynamic feature points detected by epipolar constraints fall on the dynamic mask segmented by the semantic segmentation thread, the object is determined to be moving. If the object is determined to be moving, all feature points within the dynamic mask are deleted. The remaining static feature points are used in conjunction with the depth image to generate a point cloud, which is then mapped onto the map. The point cloud is assigned a corresponding color based on the label color of the scene segmentation, thus generating a map with semantic information.

[0053] In this invention, firstly, the tracking module is divided into three sub-threads: an ORB feature extraction thread, a semantic segmentation thread, and a depth contour detection thread. RGB image frames are sent to the ORB feature extraction thread and the semantic segmentation thread, respectively, while depth image frames are sent to the depth contour detection thread. The ORB feature extraction thread uses RGB image frames acquired by an RGB-D camera and an epipolar constraint algorithm to detect dynamic and static feature points in the actual scene, while waiting for the mask of the dynamic prior target currently being segmented. Secondly, a lightweight semantic segmentation network segments the static and dynamic prior targets in the actual environment, while waiting for the boundary expansion result of the depth contour mask. Simultaneously, based on the depth gradient of the depth image, the approximate shape of the target is determined, and a depth mask is drawn. Then, based on morphological processing, the depth mask is dilated to expand the mask boundary blur caused by camera precision, and pixel-level image edge fusion is performed with the semantic segmentation result of the dynamic target to obtain a corrected mask. Finally, based on the corrected mask, feature points within the dynamic mask are removed from the grayscale region of the dynamic target, and the remaining static feature points are combined with semantic labels to construct a semantic point cloud map.

[0054] The beneficial effects of this invention are: improving the real-time performance of semantic SLAM and the efficiency of semantic feature extraction. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the lightweight visual semantic odometry framework of the present invention;

[0056] Figure 2 This is a schematic diagram illustrating the principle of the epipolar constraint algorithm for dynamic feature point detection in this invention.

[0057] Figure 3 This is a schematic diagram of the structure of TopFormer, the lightweight semantic segmentation network used in this invention;

[0058] Figure 4 This is a schematic diagram of the slider for traversing the depth image according to the present invention; Detailed Implementation

[0059] The invention will now be further described with reference to the accompanying drawings.

[0060] Reference Figures 1-4 A lightweight visual semantic odometry method for dynamic scenes includes the following steps:

[0061] Step (1): Acquire RGB and Depth images using an RGB-D camera, and send the RGB images to the ORB feature extraction thread and semantic segmentation thread respectively, and send the Depth image frames to the depth contour detection thread.

[0062] The process is as follows:

[0063] The tracking module is divided into three sub-threads: ORB feature extraction thread, semantic segmentation thread, and depth contour detection thread, and the three threads run in parallel. Among them, RGB image frames are sent to the ORB feature extraction thread and semantic segmentation thread respectively, and depth image frames are sent to the depth contour detection thread.

[0064] Step (2): The ORB feature extraction thread extracts the ORB features of each frame of the RGB image sent in, detects dynamic feature points according to the epipolar constraint, and then waits for the dynamic target segmentation result of the semantic segmentation thread.

[0065] The process is as follows:

[0066] Based on RGB image frame detection of dynamic feature points, feature points extracted from the current image frame Ic and the previous frame Ip are matched, and the homography matrix H between the two frames is solved using the Random Sample Consensus (RANSAC) algorithm and the Levenberg-Marquarelt (LM) algorithm. Assuming the pixel set P... pn It is a point on IP, and its expression is:

[0067] P pn =[u pn v pn 1] T n = 1, 2, 3, 4, 5, ..., k (1)

[0068] Among them, P pn u represents the coordinates of the set of points on the previous frame image Ip. pn and v pn Let be the x and y coordinates of a point, n be the index of the point, k be a positive integer, and the coordinates of the corresponding set of points on Ic are...

[0069] P cn =[u cn v cn 1] T ,n=1,2,3,4,5,...,k (2)

[0070] Among them, P cn u represents the coordinates of the set of points on the current frame image Ic. cn and v cn Let x and y be the x and y coordinates of the point.

[0071] P cn It can be obtained from the homography matrix H.

[0072] P cn =HP Pn , n=1,2,3,4,5,……,k (3) The camera pose between the previous frame image Ip and the current frame image Ic is obtained by solving the least squares problem.

[0073]

[0074] Among them, P pi and P ci Corresponding to P respectively pn and P cn The point with index i is given above. R and t represent the rotation matrix and translation vector between the two frames, respectively, and N is a positive integer.

[0075] To obtain dynamic feature points, firstly, matching points are extracted from Ip and Ic, and the fundamental matrix F between the two frames is calculated using the RANSAC algorithm; then, the epipolar line in the current frame is calculated using the fundamental matrix F. Therefore, point sets P are selected respectively. pn and P cn A pixel P on p1 =[u p1 v p1 1] T and P c1 =[u c1 v c1 1] T And P p1 ∈P pn P c1 ∈P cn Calculate the polar line l p1 ,like Figure 2 :

[0076]

[0077] Where X, Y, and Z represent the inline matrix, i.e., the x, y, and y coordinates of point P, and F is the base matrix. Then, the distance from the feature point to the corresponding epipolar line is calculated as follows:

[0078]

[0079] Where d represents the distance from the feature point to the corresponding epipolar line.

[0080] The point sets in the current frame and the previous frame are traversed sequentially. When d is greater than the threshold ε we set, it is a dynamic feature point and is stored in the dynamic point set, waiting for the semantic segmentation mask.

[0081] Step (3): The semantic segmentation thread first uses a lightweight semantic segmentation network to segment each frame of the transmitted RGB image, obtains the segmented image mask, and waits for the depth mask result from the depth contour detection thread.

[0082] The process is as follows:

[0083] The semantic segmentation thread receives the sent RGB image frames and feeds them into the semantic segmentation network for semantic feature extraction. To improve the efficiency of semantic information acquisition, this method introduces the lightweight semantic segmentation network TopFormer to extract semantic features, and converts the trained model into an ONNX model, using CUDA to accelerate inference based on the ONNXRuntime-GPU version.

[0084] This method introduces TopFormer, a lightweight semantic segmentation network based on TransFormer. Its miniaturized version can produce competitive real-time inference results on ARM-based mobile devices. The model is trained using the ADE20K dataset, which provides 150 classification categories, adequately handling indoor scene segmentation tasks. For dynamic indoor scenes, this method categorizes common indoor objects into static, dynamic, and movable objects. Static objects include beds, tables, and televisions; dynamic objects include people; and movable objects include chairs, books, and vases. The method transforms the input RGB image to 512×512×3 before inputting it into the semantic segmentation network, outputting a color mask and grayscale image of the same size. It detects the grayscale values ​​of dynamic prior objects and determines whether a dynamic object exists in the current frame based on these grayscale values. If a dynamic object is found, the flag indicating the presence of a dynamic object is incremented by 1.

[0085] Step (4): The depth contour detection thread obtains the depth gradient by traversing the depth value of each pixel, thereby determining targets with similar depths and generating a depth mask. Then, based on morphological processing, the depth mask is dilated to expand the mask boundary blur caused by camera precision and sent to the image boundary fusion module.

[0086] The process is as follows:

[0087] The depth contour detection thread receives the sent Depth image frame and performs depth mask creation. Due to the accuracy limitations of the depth camera, and to prevent boundary noise point leakage and unclear ranging of small objects, this method is divided into two parts: coarse traversal and fine traversal. First, a coarse traversal is performed using a 30×30 slider with a stride of 30 to traverse the Depth image, recording the depth values ​​in the slider:

[0088] D=d(x:x+29,y:y+29) (7)

[0089] Where D represents the depth value of the corresponding pixel, and x and y represent the horizontal and vertical coordinates of the corresponding pixel, respectively;

[0090] For boundary pixels, the slider can change according to the size of the boundary pixel block, such as... Figure 4 As shown, that is

[0091]

[0092] Where n is the length of the slider, m is the width of the slider, and Edge is the set of boundary pixels.

[0093] After traversing a depth image, depth images within consecutive and similar depth gradient ranges are grouped into one category:

[0094] Obj(n),n=1,2,3,4,……,k (9)

[0095] Here, Obj(n) represents the category.

[0096] Next, the image is denoised to remove isolated image patches smaller than 1000 pixels. Then, a fine traversal is performed. Based on the depth values ​​obtained from the coarse traversal, the dynamic prior mask obtained from semantic segmentation is used to further label the depth pixels, assigning labels to depth image patches (including dynamic and movable objects). Furthermore, the depth images obtained from the coarse traversal are classified, and sub-pixel boundaries of depth image patches with prior dynamic target labels are extracted based on the gray-level centroid method to further clarify the target mask boundaries, as follows:

[0097]

[0098] Where I(x,y) represents the grayscale image, (x,y) represents the pixel coordinates of the image, S represents the target to be extracted, and (x0,y0) represents the grayscale centroid coordinates; thus, the dynamic target depth mask after refining the boundary is obtained.

[0099] Finally, the depth mask is further extended based on the morphological dilation method to prevent the leakage of dynamic noise points. The depth image after the boundary is thinned is then grayscaled. The image blocks of dynamic targets are traversed and the image is expanded outward by a certain distance until the threshold τ is reached.

[0100] Step (5): The image boundary fusion module receives the scene segmentation results and the depth mask after the depth contour expansion from the lightweight semantic segmentation network, and fuses the mask based on the weighted average method to achieve dynamic mask correction.

[0101] The process is as follows:

[0102] After the dynamic target depth mask from step (4) is obtained, the semantic segmentation image from step (3) and the depth image from step (4) are converted to the same size. The semantic segmentation image and the depth image are traversed, and the mask coordinates of the dynamic target located in the image patch are recorded respectively. The two images are then fused using the group averaging method.

[0103] F(x,y)=ωa ·A(x,y)+ω b ·B(x,y) (11)

[0104] Where, ω a ω b The weighting coefficients are A(x,y) and B(x,y), which are the coordinates of the depth mask and the original mask, respectively.

[0105] Because the TopFormer semantic segmentation network does not clearly segment object boundaries and suffers from mask dilation when segmenting people, the intersection of the two is taken, with the depth mask boundary as the reference:

[0106]

[0107] Among them, M f For the final corrected mask, and It is divided into the original mask and the depth mask for dynamic targets.

[0108] Step (6): Use the mask corrected in step (5) and the dynamic feature points initially selected in step (2) to determine the final dynamic feature points and remove them. Input the remaining static feature points into the visual odometry module to construct a semantic point cloud map.

[0109] The process is as follows:

[0110] This invention combines geometric feature detection and semantic information to remove dynamic feature points, and then uses the remaining static feature points to construct a 3D point cloud semantic map. If a dynamic feature point detected by epipolar constraints falls on a dynamic mask segmented by the semantic segmentation thread, the object is determined to be moving. If the object is determined to be moving, all feature points within the dynamic mask are deleted. The remaining static feature points are combined with a depth image to generate a point cloud, which is then mapped onto the map. The point cloud is assigned a color based on the scene segmentation label color, generating a map with semantic information.

[0111] The above describes the effect of one embodiment of the present invention. Obviously, the present invention is not limited to the above embodiment. Various modifications can be made to it without departing from the basic spirit of the present invention and without exceeding the scope of the substantive content of the present invention.

Claims

1. A lightweight visual semantic odometry method for dynamic environments, characterized in that, The method includes the following steps: Step (1): Acquire RGB and Depth images using an RGB-D camera, and send the RGB images to the ORB feature extraction thread and semantic segmentation thread respectively, and send the Depth image frames to the depth contour detection thread. Step (2): The ORB feature extraction thread extracts the ORB features of each frame of the RGB image sent in, detects dynamic feature points according to the epipolar constraint, and then waits for the dynamic target segmentation result of the semantic segmentation thread. Step (3): The semantic segmentation thread first uses a lightweight semantic segmentation network to segment each frame of the transmitted RGB image, obtains the segmented image mask, and waits for the depth mask result from the depth contour detection thread. Step (4): The depth contour detection thread obtains the depth gradient by traversing the depth value of each pixel, thereby determining targets with similar depths and generating a depth mask. Then, based on morphological processing, the depth mask is dilated to expand the mask boundary blur caused by camera precision and sent to the image boundary fusion module. Step (5): The image boundary fusion module receives the scene segmentation result and the depth mask after the depth contour expansion from the lightweight semantic segmentation network, and fuses the mask based on the weighted average method to achieve dynamic mask correction. Step (6): Use the mask corrected in step (5) and the dynamic feature points initially selected in step (2) to determine the final dynamic feature points and remove them. Input the remaining static feature points into the visual odometry module to construct a semantic point cloud map. The process of step (4) is as follows: The depth contour detection thread receives the sent Depth image frame and performs depth mask creation; this is divided into two parts: coarse traversal and fine traversal. First, a coarse traversal is performed, using a 30×30 slider with a stride of 30 to traverse the Depth image and record the depth values ​​in the slider: D=d(x:x+29,y:y+29) (7) where D represents the depth value of the corresponding pixel, and x and y represent the horizontal and vertical coordinates of the corresponding pixel, respectively; For boundary pixels, the slider can change according to the size of the boundary pixel block, that is... Where n is the length of the slider and m is the width of the slider; After traversing a depth image, depth images within consecutive and similar depth gradient ranges are grouped into one category: Obj(n),n=1,2,3,4,……,k (9) Where Obj(n) represents the category; Next, the image is denoised to remove isolated image patches smaller than 1000 pixels. Then, a fine traversal is performed. Based on the depth values ​​obtained from the coarse traversal, the dynamic prior mask obtained from semantic segmentation is used to further label the depth pixels, assigning labels to the depth image patches. Furthermore, the depth images obtained from the coarse traversal are classified, and sub-pixel boundaries of depth image patches with prior dynamic target labels are extracted based on the gray-level centroid method to further clarify the target mask boundaries, as follows: Where I(x, y) represents the grayscale image, (x, y) represents the pixel coordinates of the image, S represents the target to be extracted, and (x0, y0) represents the grayscale centroid coordinates; thus, the dynamic target depth mask after refining the boundaries is obtained; Finally, the depth mask is further extended based on morphological dilation to prevent the leakage of dynamic noise points; the depth image after the boundary is thinned is grayscaled, and the image blocks of dynamic targets are traversed, and the image is expanded outward by a certain distance until the threshold τ is stopped.

2. The lightweight visual semantic odometry method for dynamic environments according to claim 1, characterized in that: The lightweight visual semantic odometry implementation for dynamic environments includes a tracking module, a visual odometry module, and a semantic mapping module. The process of step (1) is as follows: The tracking module is divided into three sub-threads: ORB feature extraction thread, semantic segmentation thread, and depth contour detection thread, and the three threads run in parallel. Among them, RGB image frames are sent to the ORB feature extraction thread and semantic segmentation thread respectively, and depth image frames are sent to the depth contour detection thread.

3. The lightweight visual semantic odometry method for dynamic environments according to claim 1 or 2, characterized in that: The process of step (2) is as follows: Based on dynamic feature point detection in RGB image frames, feature points extracted from the current image frame Ic and the previous frame Ip are matched. The homography matrix H between the two frames is then solved using a random sampling consistency algorithm and a Levenberg-Marquardt algorithm, assuming a pixel set P. pn It is a point on IP, and its expression is: P pn =[u pn v pn 1] T ,n=1,2,3,4,5,……,k (1) Among them, P pn u represents the coordinates of the set of points on the previous frame image Ip. pn and v pn Let be the x and y coordinates of a point, n be the point index, k be a positive integer, and the coordinates of the corresponding set of points on Ic be... P cn =[u cn v cn 1] T ,n=1,2,3,4,5,… …,k (2) Among them, P cn u represents the coordinates of the set of points on the current frame image Ic. cn and v cn Let x and y be the x and y coordinates of the point, and n be the index of the point. Obtained from the homography matrix H P cn =HP Pn ,n=1,2,3,4,5,……,k (3) Thus, the camera pose between the previous frame image Ip and the current frame image Ic is obtained by solving the least squares problem: Among them, P pi and P ci Corresponding to P respectively pn and P cn For the point with index i, R and t represent the rotation matrix and translation vector between two frames, respectively, and N is a positive integer; To obtain dynamic feature points, firstly, matching points need to be extracted from Ip and Ic, and the fundamental matrix F between the two frames is calculated using the RANSAC algorithm; then, the epipolar line in the current frame is calculated using the fundamental matrix F; therefore, point sets P are selected respectively. pn and P cn A pixel P on p1 =[u p1 v p1 1] T and P c1 =[u c1 v c1 1] T And P p1 ∈P pn , P c1 ∈P cn Calculate the polar line l p1 : Where X, Y, and Z represent the inline matrix, i.e., the x, y, and y coordinates of point P, and F is the base matrix; then, we calculate the distance from the feature point to the corresponding epipolar line: Where d represents the distance from the feature point to the corresponding epipolar line. The point sets in the current frame and the previous frame are traversed sequentially. When d is greater than the threshold ε we set, it is a dynamic feature point and is stored in the dynamic point set, waiting for the semantic segmentation mask.

4. The lightweight visual semantic odometry method for dynamic environments according to claim 1 or 2, characterized in that: The process of step (3) is as follows: The semantic segmentation thread receives the sent RGB image frames and feeds them into the semantic segmentation network for semantic feature extraction. To improve the efficiency of semantic information acquisition, a lightweight semantic segmentation network, TopFormer, is introduced to extract semantic features. The trained model is then converted into an ONNX model, and CUDA is used to accelerate inference based on the ONNXRuntime-GPU version. The introduced lightweight semantic segmentation network TopFormer is a lightweight semantic segmentation network based on TransFormer. Its miniature version can produce competitive real-time inference results on ARM-based mobile devices. The model is trained using the ADE20K dataset, which provides 150 classification categories, sufficient for indoor scene segmentation tasks. For dynamic indoor scenes, common indoor objects are divided into static objects, dynamic objects, and movable objects. Static objects include beds, tables, and televisions; dynamic objects include people; and movable objects include chairs, books, and vases. The input RGB image is transformed to 512×512×3 and input into the semantic segmentation network. The output is a color mask and a grayscale image of the same size. The grayscale value of the dynamic prior object is detected, and the presence of a dynamic object in the current frame is determined based on the grayscale value. If a dynamic object is found, the flag indicating the presence of a dynamic object is incremented to 1.

5. The lightweight visual semantic odometry method for dynamic environments according to claim 1 or 2, characterized in that: The process of step (5) is as follows: After the dynamic target depth mask from step (4) is obtained, the semantic segmentation image from step (3) and the depth image from step (4) are converted to the same size; the semantic segmentation image and the depth image are traversed, and the mask coordinates of the dynamic target sitting on the image patch are recorded respectively. The two images are then fused using the cluster averaging method. F(x,y)=ω a ·A(x,y)+ω b ·B(x,y) (11) Where, ω a ω b For weighting coefficients, A(x,y) and B(x,y) are the coordinates of the depth mask and the original mask, respectively; Because the TopFormer semantic segmentation network does not clearly segment object boundaries and suffers from mask dilation when segmenting people, we take the intersection of the two, using the depth mask boundary as the reference: Among them, M f For the final corrected mask, and It is divided into the original mask and the depth mask for dynamic targets.

6. The lightweight visual semantic odometry method for dynamic environments according to claim 1 or 2, characterized in that: The process of step (6) is as follows: Dynamic feature points are removed by combining geometric feature detection and semantic information, and a 3D point cloud semantic map is constructed using the remaining static feature points. If the dynamic feature points detected by epipolar constraints fall on the dynamic mask segmented by the semantic segmentation thread, the object is determined to be moving. If the object is determined to be moving, all feature points within the dynamic mask are deleted. The remaining static feature points are combined with the depth image to generate a point cloud, which is then mapped onto the map. The point cloud color is assigned according to the label color of the scene segmentation to generate a map with semantic information.

Images